Medical document complex index extraction method and system based on large language model

Abstract

The present application relates to a kind of medical documents complex index extraction system based on large language model, including data preprocessing module, model deployment module, knowledge base management module, model calling module, strategy fence rule engine module, accuracy monitoring module, man-machine interaction and data link docking module.The present application also relates to a kind of index extraction method using the medical documents complex index extraction system based on large language model described.The present application solves the defects of poor generalization ability, insufficient understanding depth, weak data integration and poor process adaptability in the prior art;Efficiently, accurately and automatically, from the unstructured medical documents of multiple heterogeneous systems, the complex indicators required for tumor quality control performance evaluation of national three-level public hospitals are extracted, real-time monitoring, dynamic updating and visual display of index data are realized, and the needs of medical quality evaluation and national examination data reporting are met.

Description

Technical Field

[0001] This invention relates to the fields of medical information processing and natural language processing, specifically to a method and system for extracting complex indicators from medical documents based on a large language model. Background Technology

[0002] Currently, the technologies for extracting key indicators from unstructured medical documents can be mainly divided into three categories: 1. A rule-based and dictionary-based approach: This approach constructs matching rules by manually pre-setting keywords and regular expressions, and combines them with a medical terminology dictionary to extract information. For example, it can locate target information using fixed expressions such as "TNM staging:" and "first treatment".

[0003] 2. Traditional machine learning models (such as support vector machines and random forests) require manual feature engineering (such as extracting text keyword frequencies and part-of-speech features) and then training the model with massive amounts of labeled data (such as manually labeled medical records of "whether the first pre-treatment staging assessment has been completed") to finally achieve indicator classification and extraction.

[0004] 3. Early deep learning models (such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) automatically extract local text features (CNNs) or temporal features (RNNs) through neural networks. For example, CNNs can be used to capture staging keywords after "pathological diagnosis:", and RNNs can be used to identify the order of "treatment-assessment".

[0005] The shortcomings of existing technology are: 1. Poor generalization ability The problem stems from the "rule-based and dictionary-based approach": the rules only cover fixed expressions preset by humans (such as only recognizing "first treatment" and not recognizing department-specific terms such as "initial treatment" and "first course of treatment"), and they do not adapt to the differences in document structure between different hospitals (such as the "staging chapter" being at the beginning of the pathology report in some hospitals and at the end in others). When a new scenario is used, the rules become directly invalid.

[0006] The problem stems from "traditional machine learning models": model performance is highly tied to the scene features of the training data (e.g., training data comes from lung cancer medical records). If applied to other types of cancer such as gastric cancer, or if switching from documents from tertiary hospitals to community hospital documents (which are more concise and lack standardized chapters), the change in feature distribution can cause the accuracy rate to drop by more than 40%, requiring the model to be retrained.

[0007] The limitations of early deep learning models mean they can only handle single formats (such as plain text) and single cancer types. When faced with garbled text in PDF scans, tabular pathology reports, or cross-cancer indicators (the staging standards for lung cancer and breast cancer are different), the models are not universal and need to be optimized separately for different scenarios.

[0008] 2. Insufficient depth of understanding The problem stems from "traditional machine learning models": they can only learn surface features of text (such as keyword frequency and sentence length), and cannot capture deep logic and clinical knowledge. For example, they cannot combine cross-textual logic such as "no history of tumor treatment in the admission record" and "the surgical record is the first surgery" to judge the "first treatment status", nor can they match the staging examination standards of "chest CT + bone scan" in the "Lung Cancer Diagnosis and Treatment Guidelines", and cannot complete complex indicator reasoning.

[0009] The problem stems from "early deep learning models": semantic understanding is limited in dimension (CNN can only capture local features, and RNN can only capture temporal features), and it cannot integrate multi-dimensional information. For example, when calculating the "intraoperative lymph node dissection standardization rate", it cannot link the "number of dissections" in the surgical record, the "number of lymph node metastases" in the pathology report, and the "minimum dissection standard" in the guidelines. It can only identify single data in isolation, leading to errors in the calculation of indicators.

[0010] 3. Strong dependence on labeled data Traditional machine learning models require manual completion of two tasks: "feature engineering + data annotation". On the one hand, it is necessary to manually select text features related to the indicators (such as the frequency of occurrence of words related to "staging"). On the other hand, it is necessary for professional medical staff to annotate the indicator results of each medical record (such as annotating "whether postoperative pathological TNM staging has been completed").

[0011] Labeling costs in the medical field are extremely high (labeling a single medical record takes more than 30 minutes, and labeling 2,000 medical records takes one month), and high-quality labeled data is scarce (different medical staff may have different understandings of the "stage assessment completion standard"). Without sufficient labeled data, the model cannot be trained and may even overfit.

[0012] 4. Weak data integration capabilities The "rule-based and dictionary-based approach" results in rules that only apply to single-source text (such as processing only discharge summaries) and lack cross-system data association mechanisms. For example, when extracting the "pre-treatment clinical staging assessment rate," it is impossible to link the "visit time" of the HIS system with the "treatment start time" of the chemotherapy system to determine the "first treatment node." Only information from a single system can be extracted in isolation, forming data silos.

[0013] The "early deep learning model" resulted in the lack of multi-system data access interfaces, and could only process single-modality (plain text) or single-source data (such as electronic medical records only). For example, when extracting "postoperative pathological TNM staging rate", it could not automatically associate the "operation time" of the surgical system with the "report issuance time" of the pathology system, and was prone to missing cases of "postoperative pathology reports not being issued in a timely manner", with a key information omission rate of more than 30%.

[0014] 5. Poor process adaptability When the statistical criteria for national civil service examination indicators are dynamically updated (such as adjustments to TNM staging standards or the addition of "mandatory items for preoperative assessment"), rule-based methods require manual rewriting and debugging of rules (such as adding "PET-CT" as a valid item for staging assessment), which takes as long as 1-2 weeks. Furthermore, there is no real-time response mechanism, and adjustments can only be made after the criteria are updated, making it impossible to complete the monthly indicator statistics simultaneously. Data can only be entered retroactively, which completely fails to meet the timeliness requirement of "monthly upload" for national civil service examinations. Summary of the Invention

[0015] To address the aforementioned problems, this invention provides a method and system for extracting complex indicators from medical documents based on a large language model. Its purpose is to overcome the shortcomings of existing technologies, such as poor generalization ability, insufficient depth of understanding, weak data integration, and poor process adaptability. It efficiently, accurately, and automatically extracts complex tumor quality control indicators required for the performance evaluation of national tertiary public hospitals from unstructured medical documents from multiple heterogeneous systems, enabling real-time monitoring, dynamic updating, and visualization of indicator data, thus meeting the needs of medical quality assessment and national examination data reporting.

[0016] To solve the above problems, the technical solution provided by the present invention is as follows: A system for extracting complex indicators from medical documents based on a large language model includes a data preprocessing module, a model deployment module, a knowledge base management module, a model invocation module, a policy-fence rule engine module, an accuracy monitoring module, and a human-computer interaction and data link integration module, wherein: The data preprocessing module is used to perform standardized collection, data anonymization, data cleaning, and format standardization processing on medical document data.

[0017] The model deployment module is used to deploy a pre-defined large language model on the server and provide concurrent inference services through the inference engine.

[0018] The knowledge base management module is used to construct a vectorized tumor quality control knowledge base that includes a policy document knowledge base, a treatment guideline knowledge base, a technical standard knowledge base, and a clinical knowledge base, and to provide semantic retrieval results to the model calling module.

[0019] The model invocation module is used to concatenate the preprocessed medical document data, the recall results of the vectorized tumor quality control knowledge base, and the prompt word template into prompt words and input them into the large language model, and then generate structured index judgment results and inference chains.

[0020] The policy fence rule engine module is used to perform terminology classification correction, consistency verification, and cancer type detection strategy verification on the input and output of the large language model, and to mark the verification results that do not conform to the preset rules.

[0021] The accuracy monitoring module is used to statistically evaluate the correctness, consistency and relevance of the output of the large language model, and trigger prompt word optimization or model optimization based on the evaluation results.

[0022] The human-computer interaction and data link docking module is used to transmit the indicator judgment results, inference chain and abnormal warning back to the hospital business system through the docking protocol and provide a visual interactive interface.

[0023] Preferably, the data desensitization method of the data preprocessing module is as follows: performing mask replacement processing on the patient name, ID number, contact information, and address fields while retaining some characters; the data cleaning method of the data preprocessing module is as follows: removing or marking the medical document data with missing preset fields, and marking the medical document data that meets the preset outlier judgment conditions; the format standardization method is as follows: segmenting the medical document data after OCR recognition according to a preset character threshold and unifying the text encoding.

[0024] Preferably, the large language model is a Deepseek-R1-Distill-Qwen-32B-AWQ model; the large language model adopts a 4-bit AWQ quantization weight format; the large language model adopts a group weight quantization strategy with a quantization group size of 128; the large language model loads the quantized weights and a cache data structure for storing intermediate results of attention calculation in historical context in the GPU memory, thereby reducing memory usage and maintaining inference accuracy without changing the model structure.

[0025] Preferably, the model deployment module runs the large language model through the vLLM inference engine; the vLLM inference engine sets the corresponding maximum concurrency according to the hardware configuration scheme of the system server and dynamically allocates video memory; the tensor parallelism of the large language model is equal to the number of GPUs; the model deployment module encapsulates the model call interface through the FastAPI framework and supports batch requests.

[0026] Preferably, the knowledge base management module uses the bge-m3 embedding model to convert the indicator definitions and knowledge base text into vectors of a preset dimension, and constructs an HNSW index structure through the Chroma vector database; the knowledge base management module calculates vector similarity using cosine similarity, and recalls the corresponding knowledge entry when the cosine similarity is greater than a preset similarity threshold.

[0027] Preferably, the model invocation module uses the prompt word template and structured output constraints; the prompt word contains no less than four thought chain guidance steps; the structured output constraints require the large language model output to include numerator markers, denominator markers and inference chain arrays, and automatically trigger secondary prompt correction when the output result does not conform to the preset JSON structure.

[0028] Preferably, the strategy fence rule engine module performs the terminology classification correction on the model input stage using terminology classification rules stored in JSON format, performs the consistency check on the model output stage, and marks cases awaiting manual review when the consistency check fails.

[0029] Preferably, the accuracy monitoring module randomly selects no less than 10% of cases as test samples, then calls the FactScore tool to compare the model inference chain with the knowledge base content for factual consistency, and triggers prompt word optimization or model optimization when the accuracy index or the consistency index is lower than a preset monitoring threshold; the accuracy index is expressed by the following formula: Where: a represents the number of medical records whose model judgment results are consistent with manual annotation; n represents the total number of medical records tested.

[0030] The consistency index is expressed by the following formula: Where: c represents the number of times the model results are consistent when the same medical record is called 3 times; m represents the total number of repeated calls.

[0031] The correlation index is expressed as follows: Where: r is used to characterize the number of medical records related to the model output inference chain and indicator requirements.

[0032] Preferably, the human-computer interaction and data link docking module uses the HL7FHIRR4 standard data interface to synchronously transmit the indicator judgment results output by the model and the inference chain back to the hospital business system.

[0033] An indicator extraction method utilizing the aforementioned medical document complex indicator extraction system based on a large language model includes the following steps: S1. The medical document data is processed by the data preprocessing module to perform standardized collection, data desensitization, data cleaning, and format standardization to generate structured text that can be used for reasoning in the large language model.

[0034] S2. Input the structured text into the model calling module, and the knowledge base management module synchronously provides the parameters and evaluation strategies corresponding to the indicator name and cancer type.

[0035] S3. Perform inference on the concatenated prompt words using the large language model, and then output the indicator judgment result and the inference chain.

[0036] S4. The strategy fence rule engine module performs term classification correction, consistency verification, and cancer detection strategy verification on the output of the large language model to obtain the final indicator judgment result.

[0037] S5. The indicator judgment result obtained in step S5 is returned to the hospital business system through the human-computer interaction and data link docking module and displayed visually; at the same time, the accuracy monitoring module evaluates the indicator judgment result and triggers prompt word optimization or model optimization based on the evaluation result.

[0038] Compared with the prior art, the present invention has the following advantages: 1. Statistical efficiency has been improved by orders of magnitude. Results: The time required for statistical analysis of tumor quality control indicators has been reduced from 5-10 days (2000 medical records) for manual operation to 4 hours for automatic calculation by the system, improving efficiency by more than 30 times.

[0039] Corresponding technology and data: This effect stems directly from the automatic integration of data access modules across multiple systems and the batch processing capabilities of the LLM inference engine. The processing speed reaches 3-5 long medical records (approximately 15,000 words) per minute, with a daily processing capacity exceeding 10,000 documents, fully meeting the timeliness requirements of the national civil service examination's "monthly reporting" requirement.

[0040] 2. The accuracy of indicator extraction significantly surpasses that of manual methods. Results: The accuracy of core indicator statistics has been systematically improved. Taking TNM staging identification as an example, the accuracy rate increased from 82.3% with manual statistics to 95.7% with this system, an increase of 13.4 percentage points. The overall indicator statistical accuracy rate has remained stable above 90%.

[0041] Corresponding technologies and data: This achievement benefits from the cross-validation of "AI large language model + SQL" (S200) and the meticulous guidance of the "task parsing and prompting engineering module". As shown in the model test report above, the average accuracy of the six core indicators reached 93.3%, significantly outperforming other existing technologies and methods.

[0042] 3. Fundamental breakthroughs have been achieved in data integration and integrity. Results: Through the multi-source data integration platform (S100), the omission rate of key information was significantly reduced from about 30% when manually integrated to 5.2% when processed by the system, a reduction of more than 80%.

[0043] Corresponding technologies and data: This effect is directly guaranteed by the data integrity verification mechanism and cross-system patient identification association technology. Examples show that the overall integrity of lung cancer patient data improved from less than 85% to 97.1%, ensuring the accuracy of the denominator in the indicator calculation.

[0044] 4. Clinical quality control has shifted from post-hoc statistics to real-time intervention. Results: Through a dynamic rule engine and a real-time early warning module (S300), the system can perform real-time verification and blocking when doctors prescribe medical orders. In pilot hospitals, the interception rate of non-standard treatment orders caused by incomplete data reached 100%, achieving "pre- and in-process" quality control in the diagnosis and treatment process.

[0045] Corresponding technology and data: As shown in the example, the system successfully intercepted the medical order of "prescribing osimertinib without EGFR testing" and guided the doctor to complete the supplementary test, thus eliminating irregular treatment behavior from the source.

[0046] 5. The system's generalization and adaptability have been greatly enhanced. Effect: When the statistical scope of national examination indicators is updated (such as adding mandatory items) or clinical guidelines are upgraded, the system adaptation and adjustment cycle is shortened from 1-2 weeks required by the rule-based method to less than 24 hours.

[0047] Corresponding technology and data: This is thanks to the visual configuration function of the dynamic rule engine. Administrators can directly update rules through the interface (such as adding a "PET-CT examination" rule for liver cancer) without coding, realizing "one-click publishing and real-time effect" of rules.

[0048] 6. Significantly reduced labor costs drive more refined management. Results: It freed up approximately 60% of the medical statistics and quality control personnel, allowing them to focus on more valuable clinical data analysis and quality improvement work instead of tedious data processing.

[0049] Corresponding Technology and Data: By leveraging the multi-dimensional data analysis capabilities provided by the visual dashboard (S900), hospital management can quickly pinpoint problematic departments and processes. For example, in one instance, a hospital, through system analysis, accurately identified the low lung cancer staging rate as being due to "low execution rate of brain MRI examinations," and improved this indicator from 78.5% to 89.6% within three months. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the system architecture of a specific embodiment of the present invention.

[0051] Figure 2 This is a screenshot of a specific embodiment of the present invention.

[0052] Figure 3 This is a screenshot of a specific embodiment of the present invention.

[0053] Figure 4 This is a schematic diagram of the system modules and workflow of a specific embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0055] This invention application claims a system for extracting complex indicators from medical documents based on a large language model, comprising a data preprocessing module, a model deployment module, a knowledge base management module, a model invocation module, a policy-fence rule engine module, an accuracy monitoring module, and a human-computer interaction and data link connection module, wherein: The data preprocessing module is used to perform standardized collection, data anonymization, data cleaning, and format standardization processing on medical document data.

[0056] The model deployment module is used to deploy pre-defined large language models on the server and provide concurrent inference services through the inference engine.

[0057] The knowledge base management module is used to build a vectorized tumor quality control knowledge base that includes policy document knowledge base, diagnosis and treatment guideline knowledge base, technical standard knowledge base and clinical knowledge base, and to provide semantic retrieval results to the model calling module.

[0058] The model invocation module is used to concatenate preprocessed medical document data, the recall results of the vectorized tumor quality control knowledge base, and prompt word templates into prompt words and input them into the large language model, and then generate structured indicator judgment results and inference chains.

[0059] The strategy fence rule engine module is used to perform term classification correction, consistency verification, and cancer type detection strategy verification on the input and output of the large language model, and to mark the verification results that do not conform to the preset rules.

[0060] The accuracy monitoring module is used to statistically evaluate the correctness, consistency, and relevance of the output of the large language model, and to trigger prompt word optimization or model optimization based on the evaluation results.

[0061] The human-computer interaction and data link docking module is used to transmit the indicator judgment results, inference chain and abnormal warning back to the hospital business system through the docking protocol and provide a visual interactive interface.

[0062] It should be noted that because current oncology medical quality control indicators rely on manual operation, and the data is scattered across multiple systems and lacks a unified standard for interpretation, existing technologies suffer from drawbacks such as low statistical efficiency, large deviations in standard implementation, high difficulty in data integration, and inability to intervene in clinical diagnosis and treatment in real time. Specifically: 1. The method of manually reviewing medical records and summarizing them in Excel takes 15-20 minutes to review a single medical record, and the monthly statistics for the entire hospital require 120 people per day.

[0063] 2. Different departments have different understandings of staging standards (such as TNM, CNLC, FIGO), resulting in a statistical error rate of 15%-20%, which affects the accuracy of the data.

[0064] 3. Patient diagnosis, treatment, and examination data are scattered across more than 14 systems, including EMR, PACS, LIS, and medical record homepage. During manual integration, 30% of cases with missing staging were not identified due to data omission.

[0065] 4. It can only complete historical data statistics and cannot verify key conditions such as "whether the staged examination has been completed" and "whether the pathology report has been uploaded" in real time when clinical treatment orders are issued, making it difficult to achieve immediate error correction and quality improvement of diagnosis and treatment.

[0066] It should be noted that the modules of this system are deployed using Docker containers and data interaction is achieved based on the RESTful API protocol. The overall architecture follows a closed-loop design of "input-processing-output-feedback" and supports unified monitoring of tumor quality control indicators for multiple hospitals and multiple cancer types.

[0067] It should be noted that the data anonymization method of the data preprocessing module is as follows: performing mask replacement processing on the patient name, ID number, contact information, and address fields while retaining some characters; the data cleaning method of the data preprocessing module is as follows: removing or marking medical document data with missing preset fields, and marking medical document data that meets the preset outlier judgment conditions; the format standardization method is as follows: segmenting the medical document data after OCR recognition according to the preset character threshold and unifying the text encoding.

[0068] It should be further clarified that the data collection scope of this system is: to collect the complete medical records of cancer patients within the past 12 months, including but not limited to: patient basic information (hospitalization number, age, gender, etc.), admission record, medical record, discharge summary, surgical record, pathology diagnosis record, examination orders, and auxiliary examination reports. The corresponding hospital data form codes are B02-1, B02-3, B04-1, B05-1, B06-1, B07-1, B08-1, B10-1, B16-1 / 2 / 3, etc.

[0069] It should be further explained that the data anonymization rules are as follows: a field-level anonymization algorithm is used, and sensitive information is handled in the following ways: Name: Keep the first and last characters, and replace the middle characters with "" (e.g., "Xu Qiyu" → "Xu Yu").

[0070] ID card number: retain the first 6 digits of the administrative division code and the last 4 digits of the check code, and replace the middle with "****".

[0071] Contact information: Keep only the first 3 and last 2 digits, and replace the middle with "****".

[0072] Address: Detailed house number is hidden; only province, city, and district information are retained.

[0073] It should be further explained that the data cleaning standards are as follows: Missing value handling: Medical records with missing key fields (hospitalization number, diagnosis code, treatment time) are directly removed; missing non-key fields (such as ethnicity) are marked as "not recorded".

[0074] Outlier handling: Medical records with hospital stays > 180 days or without specific results descriptions in the examination reports are marked as "abnormal and pending review".

[0075] It should be further explained that the standardization method is as follows: the garbled characters in the PDF scan are corrected by secondary recognition using OCR (using Tesseract-OCR 5.0), the text encoding is unified to UTF-8, and redundant spaces and special characters (such as meaningless symbols such as "\n", "\t", "×") are removed.

[0076] It should be further explained that the operation steps of the data preprocessing module are as follows: Sa1. Export target form data in batches via the JDBC interface of the hospital's HIS / EMR system, with an export frequency of daily incremental synchronization.

[0077] Sa2. Call a Python script (which requires pandas 2.0+ and numpy 1.24+) to perform data anonymization on the exported data.

[0078] Sa3. Run the cleaning program, filter valid medical records according to the above criteria, and generate a "List of Valid Medical Records" and a "List of Abnormal Medical Records".

[0079] 4. Store valid medical records in a distributed file system (such as MinIO) according to document type, with the file naming rule being "Hospital Number_Document Type_Timestamp.txt".

[0080] In this specific embodiment, the key parameters of the data preprocessing module are: OCR recognition threshold ≥ 0.9 (confidence level), text segment character count 2000 characters / segment, and data synchronization delay ≤ 5 minutes.

[0081] It should be noted that the model deployment module enables efficient deployment of large language models, ensuring the stability of model operation, high concurrency processing capabilities, and low latency.

[0082] In this specific embodiment, the system hardware configuration includes two optional options: Option 1 (Small-scale deployment): The server is configured with an Intel Xeon Gold 6330 CPU, 64GB DDR4 memory, dual NVIDIA RTX 3090 GPUs (24GB VRAM per card, totaling 48GB), and 2TB SSD storage.

[0083] Option 2 (Medium-to-large-scale deployment): The server is configured with an Intel Xeon Platinum 8480C CPU, 128GB DDR5 memory, 8 NVIDIA L20 GPUs (24GB of video memory per card, totaling 192GB), and 4TB SSD storage.

[0084] In this specific embodiment, the system's software environment is as follows: Operating system: Ubuntu 22.04 LTSServer 64-bit.

[0085] Dependencies: Python 3.9.16, PyTorch 2.0.1, CUDA 11.8, cuDNN 8.6.0, vLLM 0.2.0, Ollama 0.1.28, Docker 24.0.7.

[0086] It should be noted that the large language model is the Deepseek-R1-Distill-Qwen-32B-AWQ model; the large language model adopts the 4-bit AWQ quantization weight format; the large language model adopts a group weight quantization strategy with a quantization group size of 128; the large language model loads the quantized weights and cache data structures used to store intermediate results of attention calculation in historical context in the GPU memory, thereby reducing memory usage and maintaining inference accuracy without changing the model structure.

[0087] It should be further explained that the model deployment module runs the large language model through the vLLM inference engine; the vLLM inference engine sets the corresponding maximum concurrency according to the hardware configuration of the system server and dynamically allocates video memory; the tensor parallelism of the large language model is equal to the number of GPUs; the model deployment module encapsulates the model call interface through the FastAPI framework and supports batch requests.

[0088] It should be further explained that the deployment of the model deployment module includes the following process: Docker image building: Write a Dockerfile, encapsulate the model file, dependency libraries and execution scripts, and keep the image size within 20GB.

[0089] vLLM inference engine configuration: Set the maximum concurrency to 64 (Option 1) / 256 (Option 2), the memory allocation strategy to "dynamic allocation", and the target token generation speed to ≥100 tokens / s.

[0090] Service exposure: The model call interface is encapsulated through the FastAPI framework, with a timeout of 30 seconds, and batch requests are supported (up to 16 medical records can be processed in a single batch).

[0091] It should be further explained that the operation steps of the model deployment module are as follows: Sb1. Install Ubuntu 22.04 LTS system and configure GPU driver (NVIDIA Driver 525.125.06).

[0092] Sb2. Install Docker and its dependencies by executing pip install -r requirements.txt (requirements.txt contains the above-mentioned dependencies and their versions).

[0093] Sb3. Download the Deepseek-R1-Distill-Qwen-32B-AWQ model and place it in the / data / model / directory.

[0094] Sb4. Write a Dockerfile and execute dockerbuild-ttumor-llm:v1.0 to build the image.

[0095] Sb5. Start the container: docker run -d --gpusall -p 8000:8000 -v / data / model: / app / modeltumor-llm:v1.0.

[0096] Sb6. Call the vLLM startup script: pythondeploy_vllm.py --model / app / model --tensor-parallel-size2 (Solution 1) / 8 (Solution 2) --port8000.

[0097] In this specific embodiment, the maximum batch size of the model is 16, and the inference delay is ≤5 seconds per medical record.

[0098] It should be noted that the knowledge base management module includes: Policy document knowledge base: "Standards for the Evaluation of Tertiary Hospitals (2025 Edition)", "Operational Manual for Performance Appraisal of National Tertiary Public Hospitals", "Quality Control Indicators for Oncology Specialty (2023 Edition)".

[0099] Clinical Practice Guidelines Knowledge Base: Clinical application guidelines for new antitumor drugs (2024 edition), breast cancer diagnosis and treatment guidelines (2022 edition), and 15 other common cancer types.

[0100] Technical Standards Knowledge Base: National Examination Tumor Indicator Statistical Standards (V1.3), Tumor Corresponding Disease Coding Library (ICD-10), Anti-tumor Drug Catalog, Indicator Reading Document Scope Rules.

[0101] Clinical knowledge base: TNM / CNLC / FIGO staging criteria, and examination and assessment strategies for various cancer types (e.g., thyroid cancer requires "thyroid ultrasound + chest CR / CT + neck enhanced CT").

[0102] It should be further explained that the knowledge base management module uses the bge-m3 embedding model to convert the indicator definitions and knowledge base text into vectors of preset dimensions, and constructs the HNSW index structure through the Chroma vector database; the vector similarity calculation of the knowledge base management module uses cosine similarity, and recalls the corresponding knowledge entries when the cosine similarity is greater than the preset similarity threshold.

[0103] In this specific embodiment, the knowledge base management module is based on the Dify 0.10.0 open source platform and integrates the bge-m3 embedding model (768 dimensions) and the Chroma vector database (open source version v0.4.19).

[0104] It should be further explained that the knowledge base management module includes the following build process: Document parsing: Uses Dify's built-in document parsing tool, supporting PDF (version ≥1.7), Word (.docx / .doc), and TXT formats. During parsing, it divides the document into "chapter titles + paragraphs" and removes redundant content such as headers, footers, and tables of contents.

[0105] Text embedding: The bge-m3 model is called, the batch size is set to 32, the embedding dimension is 768, and the generated vector data is stored in the Chroma vector library. Cosine similarity is used for vector similarity calculation.

[0106] Knowledge Update: Supports administrators to upload new documents through the Dify visual interface (maximum of 10 files per upload, each file ≤100MB). After the update, incremental synchronization of the vector library is automatically triggered, and the synchronization completion time is ≤10 minutes.

[0107] It should be further explained that the knowledge base management module includes the following operation steps: Sc1. Install the Dify platform: docker-composeup-d (refer to the official Dify deployment documentation), and configure the administrator account and permissions.

[0108] Sc2. Collect the above knowledge base files and organize them according to the categories of "policy documents / treatment guidelines / technical standards / clinical knowledge".

[0109] Sc3. Log in to the Dify backend, go to the "Knowledge Base Management" module, upload files in batches, and set the segmentation mode to "by chapter + 2000 characters as a catch-all".

[0110] Sc4. Select bge-m3 as the embedding model, start the "Text Embedding and Vector Storage" task, and check the task log to ensure there are no failures.

[0111] Sc5. Configure knowledge base access permissions: Only the model calling module can read it, administrators can edit / delete it, and ordinary users can only view it.

[0112] In this specific embodiment, the bge-m3 model inference batchsize=32, the vector database index type is HNSW, the similarity threshold is ≥0.85 (recalling relevant knowledge), and the knowledge base is updated once a week.

[0113] It should be noted that the model invocation module achieves accurate extraction and calculation of tumor quality control indicators through prompt word engineering and thought chain guidance. The model invocation module adopts prompt word templates and structured output constraints; the prompt words contain no less than four thought chain guidance steps; the structured output constraints require the large language model output to include numerator markers, denominator markers, and inference chain arrays, and automatically trigger secondary prompt correction when the output result does not conform to the preset JSON structure.

[0114] In this specific embodiment, the prompt word template is shown below (designed by indicator and cancer type, taking "pre-treatment clinical staging assessment rate" of thyroid cancer as an example): System role definition: "You are an expert in the field of tumor quality control. You make judgments strictly in accordance with the 'Tumor Professional Quality Control Indicators (2023 Edition)' and the National Examination Statistical Standard V1.3. You make judgments based solely on the provided medical record texts and knowledge base information, without making subjective speculations, and you must demonstrate the complete reasoning process." Task Instructions: "Please complete the following tasks: 1. Confirm whether the patient's cancer type is thyroid cancer (ICD-10 code C73); 2. Determine whether the patient received any thyroid cancer-related treatments before admission (surgery, radiotherapy, anti-tumor drugs / traditional Chinese medicine); 3. Check if the medical record contains TNM / CNLC / FIGO staging descriptions; 4. Verify whether the examination items in the medical record conform to the thyroid cancer staging assessment strategy (mandatory item: thyroid ultrasound; optional items: chest CR / CT, neck enhanced CT / PET-CT); 5. If the following conditions are met simultaneously: 'cancer type is thyroid cancer + no treatment received before admission + staging description exists + examination items conform to the strategy,' then include it in the numerator; otherwise, do not include it; 6. Output the numerator / denominator count and ratio. The reasoning process must be marked with the relevant medical record paragraphs and knowledge base clauses." The following is an example of output format constraints: { "index_name":"Pre-treatment Clinical Staging Assessment Rate", "cancer_type":"Thyroid cancer", "medical_record_no":"1868789", "denominator_flag":true / false, / / Whether to include in the denominator (for patients receiving initial treatment) "numerator_flag":true / false, / / Whether to include in the numerator (complete the phased assessment) "reasoning_chain":[ "Based on 1: The discharge summary in the medical record diagnosed 'papillary thyroid microcarcinoma,' corresponding to ICD-10 code C73, which meets the requirements for cancer type (from the knowledge base: Tumor Disease Coding Library.csv);" "Based on criterion 2: The admission record shows no prior history of thyroid-related treatment. The patient underwent a total thyroidectomy after admission, meeting the definition of first-time treatment (from National Civil Service Examination Statistics Standard V1.3.txt);" "Based on 3: The pathology diagnosis record mentions 'pTmiN0M0, stage IA1', indicating a TNM staging description." "Criterion 4: The medical record includes thyroid ultrasound and chest CT scan records, which meets the assessment strategy (both mandatory and optional items are satisfied);" Conclusion: Take into account both the denominator and numerator. ], "rate":"90.0%" / / Single cases are only marked as either included or excluded; the percentage is calculated during aggregation. } It should be further explained that the model invocation module includes the following invocation flow: The front end sends a request through the API interface (http: / / {ip}:8000 / model / invoke), and the request parameters include "medical record text, indicator name, and cancer type".

[0115] After receiving the request, the backend first retrieves the statistical criteria and evaluation strategies for the corresponding indicators from the knowledge base and then concatenates them into the prompt words.

[0116] The vLLM inference engine is invoked to generate results, and the returned JSON format is validated (if the format is incorrect, a secondary prompt is triggered: "Please re-output according to the specified JSON format, no further explanation is required").

[0117] After successful verification, the results will be stored in a MySQL database (table name: tumor_index_result).

[0118] It should be further explained that the model invocation module includes the following steps: Sd1. Create a "Tumor Indicator Extraction" application on the Dify platform and enter the "Workflow Orchestration" module.

[0119] Sd2. Drag and drop the “LLM Call” component, select the deployed Deepseek-R1-Distill-Qwen-32B-AWQ model, and paste the above prompt word template.

[0120] Sd3. Configure the "Knowledge Base Retrieval" component, set the search keywords to "indicator name + cancer type", and retrieve the top 5 related knowledge.

[0121] Sd4. Add an "Output Format Validation" component, write a JSON format validation script (dependent on the json5 library), and set up secondary prompt logic.

[0122] Sd5. Publish the application, generate an API key, and the front end calls the model through the API request, passing in parameters such as medical record text.

[0123] Sd6 receives the results returned by the model, stores them in the database, and supports batch export.

[0124] In this specific embodiment, the key parameters of the model calling module are: prompt word character length ≤ 5000 characters, mind chain guidance steps ≥ 4 steps, JSON format verification pass rate ≥ 99%, and API call timeout time = 30 seconds.

[0125] It should be noted that the strategy fence rule engine module uses JSON-formatted terminology classification rules to perform terminology classification correction during the model input stage, performs consistency checks during the model output stage, and marks cases requiring manual review when consistency checks fail. The technical goal of the strategy fence rule engine module is to solve problems such as incorrect classification of professional terms and lack of consistency in results, thereby improving the accuracy of model judgments.

[0126] It should be further explained that the rule base design example of the policy fence rule engine module is as follows: { "terminology_classification":[ {"wrong_class":"chest CT","correct_class":"chest X-ray","rule":"chest X-ray is not classified as chest CT; descriptions containing 'chest X-ray', 'chest X-ray', or 'chest anteroposterior and lateral views' are classified as chest X-rays"}, {"wrong_class":"Abdominal ultrasound","correct_class":"Urinary tract ultrasound","rule":"In renal cancer-related medical records, 'abdominal ultrasound' specifically refers to 'urinary tract ultrasound, bilateral renal and ureteral ultrasound'"} ], "consistency_check":[ {"index":"Pre-treatment Clinical Staging Assessment Rate","key_flag":"Staging Description","check_rule":"When the same medical record is assessed multiple times, the staging description must be consistent; inconsistencies will trigger manual review."} {"index":"Postoperative pathological TNM staging rate","key_flag":"Number of lymph nodes dissected","check_rule":"Difference between the number of lymph nodes dissected in the surgical record and the pathology record is ≤2, otherwise marked as abnormal"} ], "cancer_strategy":[ { "cancer_type":"Thyroid cancer", "basic_strategy":{ "required_check":["Thyroid ultrasound"], "optional_check":["Chest CR","Chest CT","Neck Enhanced CT","PET-CT"], "document_scope":["Admission Record","Discharge Summary","Surgical Record","Pathology Record"] } }, { "cancer_type":"lung cancer", "basic_strategy":{ "required_check":["chest CT","abdominal ultrasound"], "optional_check":["Brain MRI","Bone Scan","PET-CT"], "document_scope":["Admission Record","Progress Record","Discharge Summary","Examination Orders"] } } ] } It should be further explained that the rule engine of the policy fence rule engine module includes the following workflow: Preprocessing stage: Rule matching is performed on medical terms in medical record texts to correct classification errors (such as adjusting "chest X-ray" from "chest CT" to the correct classification).

[0127] After model inference: perform consistency verification on the "key identifiers" (such as phase descriptions and cleaning quantities) output by the model. If they meet the rules, they pass; otherwise, they are marked as "awaiting manual review".

[0128] Rule updates: Administrators can edit JSON rules through the "Rule Management" interface on the Dify platform. The rules will take effect immediately after saving and no system restart is required.

[0129] It should be further explained that the policy fence rule engine module includes the following operation steps: Se1. Write a rule base JSON file (named tumor_strategy_rule.json), and enter the terminology classification, consistency verification, and cancer type strategy rules in the above format.

[0130] Se2. Create a "Rule Engine" component on the Dify platform, import the JSON rule library, and configure the rule matching priority (term classification > consistency check > cancer type strategy).

[0131] Se3. Embed the rule engine component into the model call workflow, set it to perform term classification after "Knowledge Base Retrieval" and before "LLM Call", and perform consistency verification after "Output Format Validation".

[0132] Se4. Test rule validity: Upload 100 medical records with incorrect terminology classification and verify that the correction accuracy rate is ≥95%; upload 50 medical records with duplicate judgments and verify that the consistency check pass rate is ≥98%.

[0133] In this specific embodiment, the key parameters of the policy fence rule engine module are: rule matching efficiency ≤ 1 second / medical record, rule update effective time ≤ 1 minute, and terminology classification correction accuracy ≥ 95%.

[0134] It should be noted that the technical goal of the accuracy monitoring module is to quantify the accuracy, consistency, and relevance of the model output, and to provide data support for model optimization.

[0135] It should be further explained that the accuracy monitoring module randomly selects no less than 10% of cases as test samples, and then calls the FactScore tool to compare the model inference chain with the knowledge base content for factual consistency. When the accuracy index or consistency index is lower than the preset monitoring threshold, prompt word optimization or model optimization is triggered. The accuracy index is required to be ≥90%, expressed as in formula (1): (1) Where: a represents the number of medical records whose model judgment results are consistent with manual annotation; n represents the total number of medical records tested.

[0136] The consistency index requirement is ≥95%, expressed as in formula (2): (2) Where: c represents the number of times the model results are consistent when the same medical record is called 3 times; m represents the total number of repeated calls.

[0137] The correlation index requires ≥98%, expressed as formula (3): (3) Where: r is used to characterize the number of medical records related to the model output inference chain and indicator requirements.

[0138] FactScore: Using the FactScore open-source tool (v0.5.0), the model's inference chain is compared with authoritative content in the knowledge base, and a score of ≥0.85 is considered acceptable.

[0139] It should be further explained that the accuracy monitoring module includes the following monitoring process: Sample selection: 10% of new medical records (no less than 50) are randomly selected each day as test samples, and are manually labeled back-to-back by two senior oncologists.

[0140] Automatic scoring: The system calls the FactScore tool and a custom scoring script to quantify and score the model results using four major metrics.

[0141] Report generation: A "Model Accuracy Monitoring Report" is generated weekly, which includes indicator scores, abnormal case analysis, and optimization suggestions.

[0142] Optimization trigger: When the accuracy is less than 90% or the consistency is less than 95%, the prompt word optimization or model fine-tuning process will be automatically triggered.

[0143] It should be further explained that the accuracy monitoring module includes the following specific operating steps: Sf1. Install the FactScore tool: pip install factscore==0.5.0, and configure the authoritative document path of the knowledge base as the comparison data source.

[0144] Sf2. Write a scoring script (score_calculation.py) to automatically calculate correctness, consistency, and relevance.

[0145] Sf3. Set up scheduled tasks on the Dify platform: extract test samples at 00:00 every day, execute automatic scoring at 01:00, and generate a monitoring report every Sunday at 23:00.

[0146] Sf4. Establish an optimization closed loop: After marking abnormal cases in the monitoring report (such as model judgment errors or inconsistent results), add them to the prompt word optimization sample library for updating the prompt word template.

[0147] In this specific embodiment, the key parameters of the accuracy monitoring module are: test sample ratio = 10%, consistency of manual annotation ≥ 97% (third-party arbitration), FactScore threshold ≥ 0.85, and optimization trigger cycle ≤ 3 days.

[0148] It should be noted that the technical goal of the human-computer interaction and data link integration module is to optimize the operating experience for clinicians, achieve seamless integration with the hospital's existing systems, and improve work efficiency. The human-computer interaction and data link integration module uses the HL7FHIRR4 standard data interface to synchronously transmit the model's output indicator judgment results and inference chain back to the hospital's business system.

[0149] It should be further explained that the design framework of the clinician's user interface is as follows: Core functional areas: Medical record query (supports fuzzy search by hospital number and patient name), indicator result display (displayed by cancer type and indicator, with abnormal items marked in red), and inference chain view (clicking on an indicator expands the model inference process).

[0150] Interactive controls: "Tumor indicator scope checklist" (such as TNM staging version selection, scope of examination items), "Manual correction button" (supports doctors to modify model judgment results and fill in the reason for correction), "Results export" (supports export in Excel / PDF format).

[0151] Technology stack: The front end uses Vue.js 3.2+ and ECharts 5.4+, and the back end uses SpringBoot 2.7+. The interface response time is ≤2 seconds.

[0152] It should be further explained that the data link connection framework for the human-computer interaction and data link connection module is as follows: Integration systems: Hospital doctor station system, HIS system, EMR system.

[0153] Interface protocol: Adopts HL7FHIRR4 standard, and realizes data interaction through WebService interface.

[0154] Core function: When doctors open a medical record on the doctor's station, they can click the "AI Tumor Indicator Analysis" button to directly jump to the indicator analysis interface of the corresponding medical record in this system without having to repeatedly enter information; the indicator results of this system can be synchronously transmitted back to the "quality control module" of the EMR system.

[0155] It should be further explained that the operation steps of the human-computer interaction and data link docking module are as follows: Sg1. Front-end development: The operation interface is built based on Vue.js, the control layout is designed according to the doctors' usage habits, and ECharts is integrated to realize the visualization of indicators.

[0156] Sg2. Backend Development: Use Spring Boot to write an interface adaptation layer to implement the conversion between the HL7FHIR protocol and the API of this system.

[0157] Sg3. System Integration Test: Collaborate with the hospital's information department to complete the interface integration test of the doctor's station / EMR system, ensuring a jump success rate of ≥99% and a data synchronization delay of ≤3 seconds.

[0158] Sg4. User Training: Demonstrate the operation process to clinicians, collect feedback and optimize the interface (such as adjusting the position of check boxes and simplifying the export steps).

[0159] It should be further noted that the key parameters of the human-computer interaction and data link docking module are: interface response time ≤ 2 seconds, jump success rate ≥ 99%, data synchronization delay ≤ 3 seconds, and doctor operation satisfaction ≥ 90%.

[0160] This invention application claims a method for extracting indicators from a complex indicator extraction system for medical documents based on a large language model, comprising the following steps: S1. The data preprocessing module performs standardized collection, data anonymization, data cleaning, and format standardization on medical document data to generate structured text that can be used for reasoning in large language models.

[0161] S2. Input the structured text into the model call module, and the knowledge base management module will synchronously provide the parameters and evaluation strategies corresponding to the indicator name and cancer type.

[0162] S3. Perform inference on the concatenated prompt words using a large language model, and then output the index judgment result and inference chain.

[0163] S4. The strategy fence rule engine module performs term classification correction, consistency verification, and cancer detection strategy verification on the output of the large language model to obtain the final indicator judgment result.

[0164] S5. The indicator judgment results obtained in step S5 are returned to the hospital business system through the human-computer interaction and data link docking module and displayed visually; at the same time, the accuracy monitoring module evaluates the indicator judgment results and triggers prompt word optimization or model optimization based on the evaluation results.

[0165] It should be noted that step S1 first establishes a multi-source data integration platform to acquire basic and diagnostic data of cancer patients. This is because the statistical analysis of cancer quality control indicators needs to cover the entire process of patient care from initial consultation to treatment. However, existing data is scattered across multiple systems such as EMR, PACS, and LIS. Therefore, standardized interfaces are needed to centrally integrate the data while ensuring that the data complies with the anonymization requirements of the Personal Information Protection Law, providing a complete and compliant data source for subsequent indicator calculations.

[0166] It should be further explained that the specific operation steps of step S1 are as follows: 1) Interface Development and Data Integration: Develop 12 types of standardized data interfaces to connect with the patient's basic information form (B02-1), patient diagnosis record (B02-3), medical record cover sheet (B04-1), admission record (B05-1), progress record (B06-1), discharge record (B07-1), inpatient surgery record (B08-1), inpatient medical order record (B10-1), routine examination record (B16-1), pathology examination record (B16-2), molecular pathology test record (B16-3), and clinical radiotherapy record (B18-1), achieving real-time data synchronization to the integration platform with a synchronization delay of ≤5 minutes.

[0167] 2) Data anonymization: Field-level anonymization algorithms are used to replace sensitive information such as patient name, ID number, and contact information (e.g., the middle character of the name is hidden with "*", and the first 6 and last 4 digits of the ID number are retained), while key statistical fields such as hospital number, diagnosis code, and treatment time are retained.

[0168] 3) Data integrity verification: Based on preset rules (such as "the difference between the number of records on the medical record front page and the number of records in the inpatient information table is ≤5%" and "pathological examination records must include specimen location and diagnostic conclusion"), missing data is automatically identified and a "Data Missing Report" is generated, prompting the information department to supplement and connect the missing data.

[0169] In this specific embodiment, taking lung cancer patient data from a tertiary hospital from January to December 2023 as an example, the data obtained through the interface is shown in Table 1 below: Table 1. Data on lung cancer patients It should be noted that steps S2-S3 utilize an AI large language model and SQL queries to calculate the numerator and denominator of the core quality control indicators. The reason and principle behind this step is that core quality control indicators (such as the pre-treatment clinical staging assessment rate) require processing both structured data (such as diagnostic codes) and unstructured data (such as staging descriptions in medical record texts). SQL can efficiently filter structured data, while the AI ​​large language model (LLM) can extract key information from medical records (such as "whether previous treatments were received" and "whether TNM staging exists") through natural language processing. The combination of these two methods enables automated indicator calculation, reducing human error.

[0170] It should be further explained that, taking the "pre-treatment clinical staging assessment rate" as an example, the specific implementation steps are as follows: S210. Define the scope of cancer types (SQL query) Patients with target cancer types were screened based on the ICD-10 coding range. The specific coding rules are shown in Table 2 below: Table 2. Cancer Type Coding Rules Use SQL statements to exclude patients with multiple primary cancers (the same patient's diagnosis record contains two or more different cancer type codes), and output a list of patients with the target cancer type (including hospital number, diagnosis code, and discharge time). S220. Determine the initial treatment status (processed by AI large language model) The Deepseek-R1-Distill-Qwen-32B-AWQ model was used, with a preset prompt template loaded: "Based on the past medical history and present medical history in the initial medical record (B06-1), admission record (B05-1), and discharge record (B07-1), determine whether the patient received 'surgical treatment', 'radiotherapy', 'anti-tumor drug treatment', or 'traditional Chinese medicine anti-tumor treatment' for [cancer type] before admission; determine whether the patient received the above treatments for [cancer type] after admission."

[0171] The model outputs results in JSON format, including "IsSczl" (whether treatment was received before admission: 1 = no treatment, 2 = received treatment), "issczl2" (whether treatment was received after admission: 1 = no treatment, 2 = received treatment), and "aisczlyj" (analysis basis). Patients with "IsSczl=1 and issczl2=2" are selected as "the number of patients with a certain type of cancer who received their first treatment in the same period" (denominator).

[0172] S230. Determine the completion status of clinical staging (AI + SQL combination) AI processing: The model reads the admission record (B05-1) and discharge record (B07-1), determines whether there is a TNM / CNLC / FIGO staging description (such as "cT2N1M0" or "CNLCIIb stage"), or whether it meets the staging examination and assessment strategy (such as breast cancer requiring "breast ultrasound / mammography / MRI + chest CT + abdominal ultrasound / PET-CT"), and outputs the judgment result of "whether staging is completed".

[0173] SQL processing: Query whether there is a clear staging code or description in the diagnostic record table (B02-3) and pathology examination record (B16-2), and cross-validate with the AI ​​results.

[0174] Patients who "completed staging assessment" were selected as "the number of patients with a certain type of cancer who completed clinical staging assessment before the first treatment" (numerator).

[0175] In this specific embodiment, taking a breast cancer patient (hospitalization number: 6725285) as an example, the calculation process of the indicators is shown in Table 3 below: Table 3. Model Calculation Flowchart It should be noted that in step S4, the establishment of a strategy fence rule engine module enables indicator quality control and clinical intervention. The reason and principle of the module is as follows: During the clinical diagnosis and treatment process, doctors may neglect to conduct staged examinations or pathological tests. It is necessary to block non-standard medical orders through real-time rule verification and prompt the supplementation of necessary diagnosis and treatment steps. At the same time, it supports hospitals to update custom rules according to guidelines to ensure system adaptability and scalability.

[0176] It should be further explained that the specific implementation steps of the policy fence rule engine module are as follows: Building a dynamic rule engine The system includes six pre-defined quality control rules for core indicators, such as "Molecular pathology testing must be completed before the first targeted therapy (the B16-3 form must contain EGFR / ALK testing records)" and "The number of lymph nodes dissected during colorectal cancer surgery must be ≥12 (both surgical record B08-1 and pathology record B16-2 must record the number of nodes dissected)". It also supports user-defined rules (such as "PET-CT examination is required for liver cancer"). Users can configure rule triggering conditions (such as "when a 'sorafenib' prescription is issued"), validation fields (such as "EGFR mutation status in the molecular pathology test record"), and alert methods (pop-up / SMS) through a visual interface.

[0177] Real-time early warning and intervention When a doctor issues a cancer treatment order (such as chemotherapy, surgery, or targeted therapy) in the HIS system, the system automatically retrieves PACS / LIS / EMR data to verify whether it complies with quality control rules.

[0178] If the rules are not met (e.g., "Osimertinib prescription issued, but no EGFR test record"), the system will pop up a window to block the prescription and prompt "EGFR molecular pathology test is required (refer to the 2022 edition of the Lung Cancer Diagnosis and Treatment Guidelines). The test record can be uploaded and then resubmitted." The early warning information is simultaneously pushed to the department's quality control personnel, generating a "Real-time Quality Control Early Warning List" to track the rectification progress.

[0179] In this specific embodiment, when a doctor prescribes osimertinib targeted therapy for a lung cancer patient (hospitalization number: 1872853), the system triggers the following process: Rule verification: A query of B16-3 molecular pathology testing records revealed no EGFR mutation test results.

[0180] Pop-up warning: "The patient has not completed EGFR molecular pathology testing, which does not meet the requirements for pre-targeted therapy testing in the 2022 edition of the Lung Cancer Diagnosis and Treatment Guidelines. Please complete the testing (test item: EGFR gene mutation testing, test method: NGS)."

[0181] Rectification tracking: The early warning information was pushed to the thoracic surgery quality control officer. The doctor's order could only be submitted normally after the patient completed the test (the "EGFR19 exon deletion" report was issued on 2023-11-05) and uploaded it to the system.

[0182] It should be noted that step S5 generates indicator statistical reports and visualization dashboards through the human-computer interaction and data link docking module, and outputs the final quality control results. The specific process includes the following steps: Indicator reports are generated automatically. Statistical reports are generated based on three dimensions: "cancer type - department - time". The reports include the numerator, denominator, percentage, target achievement rate, and difference analysis from the benchmark value (such as "benchmark value of 90% for clinical staging assessment rate before first treatment").

[0183] The report supports both Excel and PDF formats. The Excel format contains detailed data (hospitalization number, anonymized patient name, staging results, and treatment time), while the PDF format is a summary report (including charts and text analysis).

[0184] Visual Kanban Development Hospital-level dashboard: Real-time display of the hospital's compliance rates for 6 core indicators (such as "pre-treatment clinical staging assessment rate 85.13%" and "perioperative mortality rate 0.1%)", ranked by cancer type and department, with indicators that have not met the standards marked in red (compliance rate < benchmark value).

[0185] Department Dashboard: Displays the trend of department indicators (last 7 days / 30 days) and the distribution of problematic cases (e.g., "15 cases of lung cancer with missing staging, of which 8 cases were due to lack of brain MRI").

[0186] Doctor's Dashboard: Displays the completion status of individual quality control tasks (e.g., "Dr. Zhang's phased assessment rate this month is 88%, ranking in the top 15% of the department"), which is linked to performance.

[0187] In this specific embodiment, the indicator report is shown in Table 4 below: Table 4. Statistical Report on Core Indicators of Oncology Specialty in a Certain Hospital in 2023 (Partial) In this specific embodiment, the data accumulation of the present invention has collected more than 20,000 patient documents covering 15 common cancer types, and integrated 12 core data tables (including patient basic information table, diagnosis record table, medical record cover page, surgical record, molecular pathology test record, etc.). Among them, 800 documents have completed the labeling of national tumor quality control indicators, forming a basic dataset containing cross-system data association relationships. The data details are shown in Table 5 below.

[0188] Table 5. System Data Relationship Table Model Validation: Based on the Deepseek-R1-Distill-Qwen-32B-AWQ quantization model (4-bit AWQ, quantization group size 128), the extraction test of 6 core tumor quality control indicators for the national examination was completed. The test dataset consisted of 2000 real medical records covering 15 types of cancer (800 of which were labeled). The accuracy of the core indicators was significantly better than existing technologies. The test results are shown in Tables 6 and 7 below: Table 6. Model Test Report Table 7. Accuracy Comparison Table It should be noted that, as Figure 1 , 2 As shown in Figures 3 and 4, this invention has built an end-to-end system prototype that includes "multi-system data access - document preprocessing - prompt generation - LLM reasoning - rule verification - visualization output - feedback loop". It adopts Docker containerized deployment, realizes data interaction based on RESTful API and HL7FHIRR4 protocol, supports batch processing of national examination indicators (average 1200 documents per day) and visualization display, and can be seamlessly connected with the hospital's existing HIS / EMR system.

[0189] It should be further explained that because this technology utilizes a multi-source data integration platform to achieve unified data integration, combines AI large language models with SQL to automate indicator calculations, and employs a dynamic rule engine to enable real-time clinical intervention, the technology of this invention has the advantages of high statistical efficiency, strong data accuracy, good clinical adaptability, and the ability to achieve continuous quality improvement. It can be applied to extract key information from unstructured medical documents such as discharge summaries, pathology reports, imaging reports, and outpatient medical records, providing automated data processing capabilities for scenarios such as performance evaluation of national tertiary public hospitals, clinical decision support, scientific research data mining, and medical quality assessment. It is particularly suitable for the accurate extraction and statistical analysis of national tumor quality control indicators (such as the pre-treatment clinical staging assessment rate and post-operative pathological TNM staging rate).

[0190] Specifically, this manifests as follows: Efficiency improvement: The time for statistical analysis of indicators has been shortened from "120 patients / day per month for the whole hospital" to "automatic calculation ≤ 2 hours", and the processing speed reaches 3-5 long text medical records (about 15,000 words) per minute.

[0191] Improved accuracy: By standardizing the interpretation of staging standards (such as TNM 9th edition and CNLC) through AI models, the statistical error rate of indicators has been reduced from 15%-20% to ≤3%. At the same time, data omissions have been reduced through "AI+SQL" cross-validation (the identification rate of missing staging cases has been increased from 70% to 99%).

[0192] Strong clinical adaptability: The dynamic rule engine supports custom rules (such as adding a rule for "PSMA detection before targeted therapy for prostate cancer") and adapts to different cancer guidelines updates (such as seamless switching from the 2022 version of the "Guidelines for the Diagnosis and Treatment of Breast Cancer" to subsequent versions).

[0193] Continuous quality improvement: The real-time early warning function can correct errors immediately when diagnosis and treatment occur, avoiding the formalism of "recording after treatment". Combined with the trend analysis of the visual dashboard, it promotes the continuous improvement of the indicator compliance rate (for example, the clinical staging assessment rate before the first treatment in a certain hospital increased from 85.13% to 92.3% in 3 months).

[0194] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0195] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0196] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0197] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A system for extracting complex indicators from medical documents based on a large language model, characterized in that: It includes a data preprocessing module, a model deployment module, a knowledge base management module, a model invocation module, a policy-fence rule engine module, an accuracy monitoring module, and a human-computer interaction and data link integration module, among which: The data preprocessing module is used to perform standardized collection, data desensitization, data cleaning, and format standardization processing on medical document data. The model deployment module is used to deploy a pre-defined large language model on the server and provide concurrent inference services through the inference engine; The knowledge base management module is used to construct a vectorized tumor quality control knowledge base that includes a policy document knowledge base, a diagnosis and treatment guideline knowledge base, a technical standard knowledge base, and a clinical knowledge base, and to provide semantic retrieval results to the model calling module; The model invocation module is used to concatenate the preprocessed medical document data, the recall results of the vectorized tumor quality control knowledge base, and the prompt word template into prompt words and input them into the large language model, and then generate structured index judgment results and inference chains; The policy fence rule engine module is used to perform terminology classification correction, consistency verification, and cancer detection strategy verification on the input and output of the large language model, and to mark the verification results that do not conform to the preset rules. The accuracy monitoring module is used to statistically evaluate the correctness, consistency and relevance of the output of the large language model, and trigger prompt word optimization or model optimization based on the evaluation results; The human-computer interaction and data link docking module is used to transmit the indicator judgment results, inference chain and abnormal warning back to the hospital business system through the docking protocol and provide a visual interactive interface.

2. The medical document complex indicator extraction system based on a large language model according to claim 1, characterized in that: The data desensitization method of the data preprocessing module is as follows: performing mask replacement processing on the patient name, ID number, contact information, and address fields while retaining some characters; the data cleaning method of the data preprocessing module is as follows: removing or marking the medical document data with missing preset fields, and marking the medical document data that meets the preset outlier judgment conditions; the format standardization method is as follows: segmenting the medical document data after OCR recognition according to the preset character threshold and unifying the text encoding.

3. The medical document complex indicator extraction system based on a large language model according to claim 2, characterized in that: The large language model is a Deepseek-R1-Distill-Qwen-32B-AWQ model; the large language model adopts a 4-bit AWQ quantization weight format; the large language model adopts a group weight quantization strategy with a quantization group size of 128; the large language model loads the quantized weights and a cache data structure for storing intermediate results of attention calculation in historical context in the GPU memory, thereby reducing memory usage and maintaining inference accuracy without changing the model structure.

4. The medical document complex indicator extraction system based on a large language model according to claim 3, characterized in that: The model deployment module runs the large language model through the vLLM inference engine; the vLLM inference engine sets the corresponding maximum concurrency according to the hardware configuration scheme of the system server and dynamically allocates video memory; the tensor parallelism of the large language model is equal to the number of GPUs; the model deployment module encapsulates the model call interface through the FastAPI framework and supports batch requests.

5. The medical document complex indicator extraction system based on a large language model according to claim 4, characterized in that: The knowledge base management module uses the bge-m3 embedding model to convert the indicator definitions and knowledge base text into vectors of a preset dimension, and constructs an HNSW index structure through the Chroma vector database. The vector similarity calculation of the knowledge base management module uses cosine similarity, and recalls the corresponding knowledge entries when the cosine similarity is greater than a preset similarity threshold.

6. The medical document complex indicator extraction system based on a large language model according to claim 5, characterized in that: The model invocation module uses the prompt word template and structured output constraints; the prompt word contains no less than four thought chain guidance steps; the structured output constraints require the large language model output to include numerator markers, denominator markers and inference chain arrays, and automatically trigger secondary prompt correction when the output result does not conform to the preset JSON structure.

7. The medical document complex indicator extraction system based on a large language model according to claim 6, characterized in that: The policy fence rule engine module performs term classification correction on the model input stage using term classification rules stored in JSON format, performs consistency verification on the model output stage, and marks cases awaiting manual review when the consistency verification fails.

8. The medical document complex indicator extraction system based on a large language model according to claim 7, characterized in that: The accuracy monitoring module randomly selects no less than 10% of cases as test samples, then calls the FactScore tool to compare the model inference chain with the knowledge base content for factual consistency. When the accuracy index or the consistency index is lower than a preset monitoring threshold, prompt word optimization or model optimization is triggered. The accuracy index is expressed by the following formula: Where: a represents the number of medical records whose model judgment results are consistent with manual annotation; n represents the total number of tested medical records; The consistency index is expressed as follows: Where: c represents the number of times the model results are consistent when the same medical record is called 3 times; m represents the total number of repeated calls. The correlation index is expressed as follows: Where: r is used to characterize the number of medical records related to the model output inference chain and indicator requirements.

9. The medical document complex indicator extraction system based on a large language model according to claim 8, characterized in that: The human-computer interaction and data link docking module uses the HL7 FHIR R4 standard data interface to synchronously transmit the indicator judgment results output by the model and the inference chain back to the hospital business system.

10. A method for extracting indicators using the complex indicator extraction system for medical documents based on a large language model as described in any one of claims 1 to 9, characterized in that: Includes the following steps: S1. The medical document data is processed by the data preprocessing module through standardized collection, data desensitization, data cleaning, and format standardization to generate structured text that can be used for reasoning in the large language model; S2. Input the structured text into the model calling module, and the knowledge base management module synchronously provides parameters and evaluation strategies corresponding to the indicator name and cancer type; S3. Perform inference on the concatenated prompt words using the large language model, and then output the indicator judgment result and the inference chain; S4. The output of the large language model is corrected by the terminology classification, the consistency check and the cancer detection strategy check through the strategy fence rule engine module, so as to obtain the final indicator judgment result. S5. The indicator judgment result obtained in step S5 is returned to the hospital business system through the human-computer interaction and data link docking module and displayed visually; at the same time, the accuracy monitoring module evaluates the indicator judgment result and triggers prompt word optimization or model optimization based on the evaluation result.

Images