A medical data consistency compression method suitable for knowledge-enhanced retrieval

Abstract

The application provides a medical data consistency compression method suitable for knowledge enhanced retrieval, and is characterized in that the method comprises the following steps: (1) multi-source medical data analysis and patient data normalization: format recognition and differential analysis are performed on multi-format multi-source medical files, content and original text tracing information are extracted, a patient unique identification aggregation method based on hospitalization number + birth date + name hash is used, all data are mapped to an extended patient-level data structure PDO, and an extended PDO in a unified format is output; (2) clinical scene driven medical attribute dynamic grading: based on the extended PDO, a three-level grading mechanism of basic classification + dynamic weight adjustment + association grading binding is constructed, and a grading data set with dynamic weight score and association evidence group binding identification is output. The application has a significant technical advantage in the application of medical data processing combined with RAG. Through a strict structured compression process, the data length can be effectively reduced.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary technology of artificial intelligence in the field of medical data processing, specifically involving an intelligent consistency compression method for multi-source heterogeneous medical data. Background Technology

[0002] As healthcare institutions become increasingly information-driven, a large amount of heterogeneous medical data is generated from various information systems such as Electronic Medical Records (EMR), Hospital Information Systems (HIS), Picture Archiving and Communication Systems (PACS), and Laboratory Information Systems (LIS). This data includes structured test results and medical orders, as well as semi-structured or unstructured admission records, imaging reports, and patient case summaries. Furthermore, this data is presented in inconsistent formats across different files, including PDFs and Excel files, with many Excel files containing records from multiple patients. In AI-driven scenarios such as clinical decision support, intelligent diagnostic assistance, and disease risk prediction, models often need to integrate data from different sources to build a complete patient disease profile.

[0003] However, these data, after simple merging, are often extremely lengthy. For example, laboratory data often contains dozens of test records, each containing hundreds of indicators; imaging reports are lengthy and repetitive with templated expressions; admission records often contain automatically generated paragraphs, which can be quickly ignored by humans, but accumulate as redundant text for the model.

[0004] In classic RAG systems, data must be chunked before entering the vector database. If the original text is too long, chunking will inevitably disrupt semantic continuity. For example, abnormal items and diagnostic significance in a test report may be split into different chunks; key descriptions in an imaging report may be fragmented, affecting recall and inference performance.

[0005] Current technologies often attempt to automatically generate summaries using large models to reduce text length, but this introduces serious risks. Medical data is highly rigorous, especially the numerical values ​​of test results, positive findings in image descriptions, drug dosages, and chronological order, all of which require precise consistency. Large model-generated summaries may introduce interpretative bias, semantic omissions, incorrect event ordering, or even numerical manipulation, none of which are clinically acceptable.

[0006] Even though some existing technologies attempt to address the aforementioned issues through tiered compression and post-verification, two core technical shortcomings remain, failing to meet the dual requirements of clinical applications and medical compliance: First, static medical attribute tiering and categorized independent compression mechanisms cannot adapt to the dynamic medical weights of individual patient clinical scenarios and lack cross-data type medical semantic association constraints. Using predefined fixed classification rules, consistency levels are determined solely based on the data type itself, without dynamically adjusting the compression strategy according to the patient's primary diagnosis, disease stage, and treatment scenario. This leads to a two-way problem of redundant retention of non-core data and excessive compression of core diagnostic and treatment data details. Simultaneously, categorized independent compression easily causes temporal misalignment and broken evidence chains in cross-modal medical cues, preventing the RAG system from recalling complete diagnostic and treatment evidence and reducing the accuracy of assisted diagnosis. Second, post-verification mechanisms cannot cover medical semantic equivalence verification throughout the entire compression process and lack end-to-end audit and traceability capabilities for the processing. It can only perform numerical, terminological, and single-point traceability checks on the compressed results, but cannot identify medical logic deviations and feature mismatches in the compression process, resulting in blind spots in clinical security. At the same time, it only achieves single-point reverse traceability from the compressed result to the original text fragment, without establishing a full-process operation audit trail, which does not meet the compliance requirements for auditability of the entire life cycle of sensitive medical data. Furthermore, the verification deviations cannot be quickly traced and dealt with, resulting in extremely low data processing efficiency.

[0007] Therefore, achieving controllable compression and structured representation of multi-source medical data while ensuring the accuracy of medical information is a critical issue that RAG-assisted diagnostic systems urgently need to address. However, currently, no single method can simultaneously meet the following requirements: First, it must automatically normalize and integrate multi-source data into patient-level content; second, it must be completely independent of large-scale model summaries, ensuring consistency of medical facts; third, it must significantly reduce data volume while maintaining core semantics; fourth, the compression results must be seamlessly usable by the RAG system without semantic fragmentation; fifth, all data must have a clear original text traceability chain, meeting clinical compliance requirements; sixth, the compression strategy can be dynamically adjusted to adapt to individual patient clinical scenarios, ensuring the integrity of core diagnostic and treatment information; and seventh, it can achieve medical semantic equivalence verification and end-to-end audit traceability throughout the compression process, eliminating blind spots in clinical security.

[0008] This invention proposes a medical data consistency compression method that combines consistency, traceability, dynamic adaptability, clinical safety, and high compression rate to address the aforementioned pain points across all dimensions. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide a medical data consistency compression method suitable for knowledge-enhanced retrieval. It aims to ensure that medical data maintains consistency, traceability and integrity during cross-system aggregation, structured transformation, text rearrangement, semantic compression and vectorized indexing. It adapts to the dynamic weight requirements of clinical diagnosis and treatment scenarios and the compliance audit requirements of the entire life cycle of medical data. It is applicable to clinical-grade intelligent auxiliary diagnosis, intelligent retrieval, clinical auxiliary decision-making systems and multimodal medical data management platforms with RAG (Retrieval-Augmented Generation) as the core.

[0010] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A medical data consistency compression method suitable for knowledge-enhanced retrieval includes: (1) Multi-source medical data parsing and patient data normalization: format identification and differential parsing of multi-format and multi-source medical documents, extracting content and original text traceability information, adopting a patient unique identifier aggregation method based on hospital number + date of birth + name hash, mapping all data to extended patient-level data structure PDO, and outputting extended PDO in a unified format; (2) Clinical scenario-driven dynamic grading of medical attributes: Based on extended PDO, a three-level grading mechanism of basic classification + dynamic weight adjustment + associated grading binding is constructed. Combined with the rule base of the Chinese Medical Association's specialty diagnosis and treatment guidelines and the core observation index base of ICD-10 disease coding, a grading dataset with dynamic weight scores and associated evidence group binding labels is output. (3) Structured semantic compression processing under association constraints: Based on the hierarchical results, differentiated compression is performed on medical data with different consistency levels. Combined with the association evidence group constraint, the integrity of the association evidence chain is ensured. Each compression operation is recorded in real time, and structured compressed data blocks with association evidence group binding and single-step operation recording are output. (4) Full-process closed-loop medical consistency verification and audit: Construct a four-layer verification mechanism to verify the compression process and results throughout the entire process, realize automatic traceability and correction of deviations, and output the VerifiedDataBlock that has passed the verification and a complete audit and verification report; (5) Enhanced RAG-friendly structure generation and semantic consistency segmentation: Construct a two-layer structure of core fact layer and original evidence layer, perform semantic block segmentation based on medical event and related evidence group constraints, and finally encapsulate to generate an enhanced RAGReadyPackage that can be directly used in the RAG system.

[0011] Furthermore, the multi-format medical files in step (1) include PDF, Excel and semi-structured text. When parsing PDF files, line numbers, page numbers, paragraph levels and coordinate information are retained. When parsing Excel files, the header is automatically identified, merged cells are disassembled and mixed multi-patient data is separated.

[0012] Furthermore, the extended PDO includes four core fields: original content field, clinical scenario metadata field, diagnosis and treatment event association identifier field, and end-to-end audit log field. The clinical scenario metadata field includes the patient's primary diagnosis ICD code, comorbidity code, disease stage, department visited, and diagnosis and treatment goal.

[0013] Furthermore, the basic classification layer in step (2) divides medical data into three categories: strong consistency data, weak consistency data, and low consistency requirement data. Among them, strong consistency data is numerical content such as test indicators, vital signs, and drug dosages; weak consistency data is narrative text such as descriptions of imaging examinations, admission records, and medical records; and low consistency requirement data is administrative content such as departments and bed numbers.

[0014] Furthermore, in step (2), the dynamic weight adjustment layer, based on the clinical scenario metadata in PDO, calls the standardized diagnosis and treatment guidelines and the core disease observation indicator library to dynamically quantify and score the medical weight of each data item, and adjust its consistency level and compression strategy priority.

[0015] Furthermore, in step (3), the strong consistency data compression first normalizes the test item name, unit, reference range and numerical format through CanonicalSchema, and then performs differential compression based on the dynamic weight grading results. The test indicators corresponding to the positive findings in the weak consistency data completely retain the time series information.

[0016] Furthermore, the weak consistency data compression in step (3) extracts core medical information and restructures it through syntactic reconstruction, semantic role positioning and extraction of key medical fields, removes template-based redundant statements, and maintains the temporal and logical correspondence between the structured restructure and the timestamps and indicator results of the associated strong consistency data.

[0017] Furthermore, the four-layer verification mechanism in step (4) includes: single-step real-time consistency verification, medical semantic equivalence verification, full-link traceability verification, and deviation automatic source tracing and correction mechanism, wherein the medical semantic equivalence verification is implemented based on SNOMEDCT, LOINC, ICD standard medical terminology database and clinical logic rule database.

[0018] Furthermore, the enhanced two-layer structure in step (5) includes a core fact layer containing structured key medical facts, associated evidence group IDs, and clinical scenario-based retrieval tags, suitable for vectorized input; the original evidence layer corresponds to the core fact layer item by item, preserving complete original text fragments, location indexes, and full-link audit traceability entry points.

[0019] Furthermore, in step (5), the semantic consistency segmentation is based on the medical event and the semantic blocks are segmented. The content within the same related evidence group is not split into different chunks, so as to ensure the semantic integrity of the medical event and the integrity of the related evidence chain.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects:

[0021] This invention offers significant technical advantages in the application of medical data processing combined with RAG. Through a rigorous structured compression process, it effectively reduces data length, typically by 40%–80%, while ensuring that all critical medical facts are not lost or altered. Based on a classification compression mechanism and a consistency check chain, the entire compression process does not rely on large models, avoiding "interpretive bias" or "medical illusions," thus greatly improving clinical usability and safety. A two-layer output strategy enables RAG to achieve high recall and high contextual completeness during retrieval, avoiding common semantic fragmentation problems. The original text traceability mechanism ensures that all data is medically auditable, meeting the requirements of medical regulations for data verifiability.

[0022] In summary, this invention has significant innovation and practical value in the compression, secure processing, structured transformation, and downstream AI applications of multi-source heterogeneous medical data. Attached Figure Description

[0023] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments of the invention and, together with the specification, further serve to explain the principles of the invention and enable those skilled in the art to practice and use the invention.

[0024] Figure 1 This is a schematic diagram illustrating the overall principle of a medical data consistency compression method suitable for knowledge-enhanced retrieval. Figure 2 This is one of the framework diagrams for multi-source medical data parsing and patient data normalization, which is a medical data consistency compression method suitable for knowledge-enhanced retrieval. Figure 3 The second part of the framework diagram for a medical data consistency compression method suitable for knowledge-enhanced retrieval: multi-source medical data parsing and patient data normalization. Figure 4A clinical scenario-driven dynamic hierarchical framework diagram of medical attributes for a medical data consistency compression method suitable for knowledge-enhanced retrieval; Figure 5 This is one of the full-process closed-loop medical consistency verification and auditing framework diagrams for a medical data consistency compression method suitable for knowledge-enhanced retrieval; Figure 6 The second part of the full-process closed-loop medical consistency verification and audit framework diagram for a medical data consistency compression method suitable for knowledge-enhanced retrieval; Figure 7 This is a framework diagram for enhanced RAG-friendly structure generation and semantic consistency segmentation of a medical data consistency compression method suitable for knowledge-enhanced retrieval.

[0025] As shown in the figure, specific structures and devices are marked in the figure to clearly illustrate the structure of the embodiments of the present invention. However, this is only for illustrative purposes and is not intended to limit the present invention to this specific structure, device and environment. Those skilled in the art can adjust or modify these devices and environments according to specific needs. Detailed Implementation

[0026] The following is a detailed description of a medical data consistency compression method suitable for knowledge-enhanced retrieval provided by the present invention, with reference to the accompanying drawings and specific embodiments. It should be noted that, to make the embodiments more detailed, the following embodiments are the best and preferred embodiments; those skilled in the art can also use other alternative methods to implement some well-known technologies; and the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0027] Please refer to the attached figures. This invention proposes a medical data consistency compression method suitable for knowledge-enhanced retrieval. The entire method consists of five main stages: multi-source medical data parsing and patient data normalization, dynamic hierarchical classification of medical attributes driven by clinical scenarios, structured semantic compression under association constraints, closed-loop medical consistency verification and auditing throughout the entire process, and enhanced RAG-friendly structure generation and semantic consistency segmentation. These five parts form a complete end-to-end processing chain, enabling complex medical data originally from multiple files, multiple formats, and multiple patients to be uniformly parsed, dynamically adapted and compressed, and rigorously verified throughout the entire process. While maintaining medical rigor, it achieves high adaptability to clinical-grade RAG systems, does not rely on generative large models throughout the process, and completely eliminates the risks of illusion and interpretability bias.

[0028] (I) Multi-source medical data analysis and patient data normalization After receiving various file types (including PDF, Excel, and semi-structured text) from the hospital information system, the system first identifies the file format and then employs different parsing strategies based on the file type. For PDF files, text content is extracted using layout structure parsing methods, while retaining line numbers, page numbers, paragraph levels, and coordinate information to facilitate the subsequent establishment of a traceability chain. For Excel files, the system needs to automatically identify table headers, break down and merge cells, and resolve the issue of multiple patient data points mixed within the same file.

[0029] To achieve cross-file data alignment, this invention employs a patient unique identifier aggregation method based on hospital number + date of birth + name hash, mapping all source content to an extended patient-level data structure PDO (PatientDataObject). This structure makes core extensions to the original storage dimensions, fully including: ① Original content field, uniformly stored according to field, line number, timestamp, source file name, and content hash; ② Clinical scenario metadata field, storing core clinical information such as the patient's primary diagnosis ICD code, comorbidity code, disease stage, department visited, and treatment goals; ③ Treatment event association identifier field, generating a unique "treatment event ID" for the same patient and the same treatment event, marking it in all associated original data items; ④ End-to-end audit log field, reserving slots for recording the entire process of operations from data parsing onwards, used to store the execution rules, timestamps, operation subjects, parameter configurations, and intermediate result hash values ​​for each step of the operation.

[0030] The output of this stage is a unified format extended PDO, which solves problems such as heterogeneous multi-source formats, difficulty in aligning different files, and mixed patient data, providing stable and standardized input for subsequent dynamic grading, associated compression, and end-to-end auditing.

[0031] (II) Dynamic grading of medical attributes driven by clinical scenarios After constructing the extended PDO, a three-level hierarchical mechanism is formed, consisting of basic classification, dynamic weight adjustment, and association hierarchical binding. The entire process is based on the rule base of the Chinese Medical Association's specialist diagnosis and treatment guidelines and the core observation indicator base of ICD-10 disease coding, without any major model dependency.

[0032] Basic classification layer: The three-level classification rules serve as a hierarchical safety baseline, ensuring that the safety threshold for core high-risk data is not breached. The first category is highly consistent data, including numerical data such as test indicators, vital signs, and drug dosages. This type of data poses a high risk to medical decision-making, as any change in the values ​​may affect the diagnostic results, and absolute consistency must be maintained. The second category is weakly consistent data, including descriptions of imaging examinations, admission records, and progress notes. This type of data is narrative text. Although the content is redundant, it contains key medical semantics and can be rearranged in a structured manner, but the medical semantics cannot be changed. The third category is data with lower consistency requirements, including administrative content such as departments, bed numbers, and inpatient information. This type of content can be directly templated.

[0033] Dynamic Weight Adjustment Layer: Based on clinical scenario metadata in PDO, this layer calls upon standardized treatment guidelines and a database of core disease observation indicators to dynamically quantify and score the medical weight of each data item, adjusting its consistency level and compression strategy priority. For example, for patients with a primary diagnosis of type 2 diabetes, the weights of core treatment indicators such as glycated hemoglobin and fasting blood glucose are increased to the highest level, and the compression strategy is enforced to retain the entire time series, with zero horizontal compression and only merging of longitudinally completely duplicated values. For patients in critical care settings, the weights of vital signs are increased to the highest level, and the compression strategy is enforced to retain minute-level timestamps and only merge completely duplicated values. Non-core indicators maintain the basic compression strategy, reasonably increasing the compression rate.

[0034] Association-based hierarchical binding layer: Based on the "diagnosis and treatment event ID", cross-type related data items under the same diagnosis and treatment event are bound into the same related evidence group, and a uniform consistency level lower limit is set to ensure that related data items are not excessively compressed individually, thus avoiding the loss of related clues from the hierarchical stage.

[0035] In this phase, the grading rules, weighting criteria, and binding relationships are all written into the PDO audit log, and a grading dataset with dynamic weight scores and associated evidence group binding identifiers is output. This enables the adaptation of differentiated compression strategies for different types of data and different clinical scenarios, avoiding the loss of medical semantics caused by indiscriminate compression, and laying the foundation for subsequent structured semantic compression.

[0036] (III) Structured semantic compression processing under association constraints Strong consistency data compression: First, a unified CanonicalSchema is used to normalize the test item names, units, reference ranges, and numerical formats, resolving format incompatibility issues between different hospitals and systems. Then, differentiated compression is performed based on dynamic weighting: For the highest-weighted core indicators, only data with completely identical values ​​at multiple consecutive time points are merged, without any horizontal filtering or deletion; for regular-weighted indicators, both longitudinal time compression and horizontal indicator compression are performed, retaining only items with outliers, significant trend changes, or belonging to key medical indicators, while deleting unchanged or medically insignificant portions. Simultaneously, based on the constraint of associated evidence groups, test indicators corresponding to positive findings in weakly consistent data, regardless of whether the values ​​are abnormal, must not be horizontally compressed or deleted; the time series information must be fully preserved to ensure the integrity of the associated evidence chain.

[0037] Weakly consistent data compression: Through syntactic reconstruction, semantic role localization, and extraction of key medical fields, the system identifies imaging sites, positive findings, quantitative indicators, spatial relationships, and diagnostic hints, and reconstructs the original text in structured field format. Numerous template statements in imaging reports, such as those incorporating clinical context, are removed, eliminating redundant text while retaining all significant medical information. Based on the constraints of associated evidence groups, the structured rearrangement must maintain temporal and logical correspondence with the timestamps and indicator results of strongly consistent data, and the semantic content of the same associated event must not be split to avoid breaking the association clues. All rearrangements retain the original line numbers and hash values ​​of character ranges, allowing for retrospective analysis at any time.

[0038] Data compression with low consistency requirements: Use a fixed template for field rearrangement to ensure a consistent overall output structure.

[0039] Real-time single-step operation tracking: For each compression operation of each data item (vertical merging, horizontal filtering, redundant sentence removal, structured rearrangement, etc.), the operation rules, content hash values ​​before and after the operation, and original text anchor points are recorded in real time and synchronously written to the PDO full-link audit log, so that every step of the compression process can be traced.

[0040] Through the above steps, this invention can significantly reduce text length without losing any key medical facts or disrupting the chain of related evidence, while ensuring that the medical semantics are not tampered with. This stage outputs structured compressed data blocks with linked evidence groups and single-step operation traces.

[0041] (iv) Closed-loop medical consistency verification and audit throughout the entire process First layer: Single-step real-time consistency verification: Each step of the compression process is embedded in the front end. After the single-step compression operation is completed, the numerical and terminological consistency verification of that step is performed immediately. If the verification fails, it is immediately rolled back to the previous version to avoid the accumulation of deviations. At the same time, the verification results are written to the audit log to control risks from the source.

[0042] The second layer: Medical semantic equivalence verification: Based on the standard medical terminology database (SNOMEDCT, LOINC, ICD) and the clinical logic rule database, a pure rule-driven medical semantic equivalence verification model is constructed to perform a full verification of the content before and after compression at the medical logic level. The core includes: verification of the consistency of positive / negative findings to ensure that the feature descriptions correspond completely with the subject without mismatches or omissions; verification of the consistency of medical logical relationships to ensure that the medical logic of lesion features, test indicators, time series, and causal relationships has not changed, with no feature splitting or logical misalignment; and verification of the integrity of associated evidence groups to ensure that cross-type data within a group still maintain complete semantic association and temporal correspondence after compression, with no broken evidence chains.

[0043] The third layer: end-to-end traceability verification: Based on the audit logs in PDO, verify whether the operation records of the entire process from data parsing to compressed output are complete, whether each operation has rules and basis, whether the hash value is verifiable, and whether it can be completely traced back to the original text, to ensure that the entire process complies with the compliance requirements of the "Administrative Measures for Network Security of Medical and Health Institutions" for auditability of the entire life cycle of medical sensitive data.

[0044] The fourth layer: Automatic deviation tracing and correction mechanism: For content that fails the verification, the mechanism automatically locates the link, step and rule basis where the deviation occurred based on the audit log, and distinguishes between deviations that can be automatically corrected (incorrect terminology mapping, non-standard format) and deviations that cannot be automatically corrected (semantic logic misalignment, numerical tampering). The former is automatically corrected and recorded, while the latter is automatically rolled back and marked with manual review prompts, which greatly improves the efficiency of anomaly handling.

[0045] This verification chain ensures that no factual errors or medical logic deviations occur during data compression, achieving auditability and traceability throughout the entire medical data processing workflow, and meeting the clinical safety and compliance requirements of the medical industry. This stage outputs a verified VerifiedDataBlock, along with a complete end-to-end audit report and verification result report.

[0046] (V) Enhanced RAG-friendly structure generation and semantically consistent segmentation An enhanced two-layer structure is constructed: The first layer is the core fact layer, composed of structured and compact key medical facts. It features a small text volume and clear structure, and includes newly added associated evidence group IDs and clinical scenario-based search tags, making it suitable as vectorized input. The second layer is the original text evidence layer, corresponding item-by-item to the content of the first layer. It stores complete original text fragments and their location indexes, as well as end-to-end audit traceability entry points, facilitating the loading of original evidence during model inference. Cross-type data within the same associated evidence group are bound and stored in the two-layer structure, ensuring that the associated clues in the core fact layer are presented centrally, and the corresponding fragments in the original text evidence layer are synchronously bound.

[0047] Semantic consistency segmentation: The system performs semantic block segmentation based on medical events (such as a test, an imaging report, or a disease progression node), while adding constraints on the segmentation of related evidence groups to ensure that the content within the same related evidence group cannot be split into different chunks. This ensures that the chunk segmentation of RAG does not destroy the semantic integrity of medical events and the completeness of the related evidence chain, thus avoiding the problem of semantic fragmentation at its source.

[0048] Final output encapsulation: The final output is an enhanced RAGReadyPackage that can be directly used for vector retrieval and evidence fusion, which fully includes FactLayer (core fact layer), EvidenceLayer (original text evidence layer), ChunkIndex (semantic block index), AuditReport (full-link audit report), and SceneTag (clinical scenario-based search tag).

[0049] Through this enhanced structure, the present invention not only ensures retrieval efficiency, but also enables the model to cite traceable original text evidence and complete chain of related evidence when generating medical answers, thereby improving the credibility of the answers and meeting medical safety requirements.

[0050] A patient hospitalized in the endocrinology department of a tertiary hospital, with a primary diagnosis of type 2 diabetes mellitus with poor glycemic control and comorbid pulmonary nodules, generated heterogeneous medical data from multiple sources during their hospitalization, including: 12 complete blood count and biochemical test Excel files, 3 chest CT imaging reports in PDF format, 1 admission record in PDF format, semi-structured text of the medical record, and administrative information on the patient's medical record cover sheet. The present invention's method is needed to achieve consistent data compression and output a standardized data package that can be directly used in a RAG-assisted diagnostic system.

[0051] Multi-source medical data analysis and implementation of patient data normalization After receiving all the above files, the system first performs format recognition and then performs differentiated parsing for different file types: For PDF files (chest CT reports, admission records): the full text content is extracted using the layout structure parsing method, while retaining the page number, line number, paragraph level, and coordinate information of each paragraph, and generating the original text position anchor points; For Excel test files: automatically identify headers, break down merged cells, separate data from other patients in the same ward mixed in the file, extract full test data for the target patient, and retain the row number, timestamp, and source file name of each row of data; For semi-structured medical records: extract text content, recording time, and physician signature information, while retaining paragraph hierarchy and original text location information.

[0052] A unique patient identifier is generated based on the hospital admission number, date of birth, and name hash. Data from all sources is then mapped to an extended PDO structure. Original content field: categorized by test data, image reports, medical records, and administrative information, storing original content, timestamps, source files, content hashes, and original text location anchors; Clinical scenario metadata fields: Enter the primary diagnosis ICD-10 code E11.6, comorbidity ICD-10 code R91, disease stage as inpatient treatment, department visited as endocrinology, and treatment goals as blood glucose control and lung nodule assessment; Treatment event association identifier: Generate a unique ID for the pulmonary nodule treatment event during this hospitalization cycle, and bind the 3 chest CT reports, corresponding tumor marker test data, and pulmonary nodule follow-up content in the medical record to this ID; Generate a unique ID for the diabetes blood glucose control treatment event, and bind the 12 blood glucose-related tests, medication records, and blood glucose assessment content in the medical record to this ID; End-to-end audit log slot: Records the time, parsing rules, and file processing results of this data parsing, and completes the initial log writing.

[0053] This step outputs a standardized extended PDO, completing the patient-level normalization aggregation of multi-source data.

[0054] Clinical scenario-driven dynamic classification of medical attributes Based on extended PDO, a three-level dynamic hierarchical structure is implemented: Basic classification: Laboratory indicators, blood glucose levels, and drug dosages are classified as strongly consistent data; chest CT reports, admission records, and progress notes are classified as weakly consistent data; and administrative information such as departments, bed numbers, and hospital numbers are classified as low-consistency data to establish a safety baseline. Dynamic weight adjustment: Based on clinical scenario metadata, the rule bases of the endocrinology diabetes diagnosis and treatment guidelines and the respiratory pulmonary nodule diagnosis and treatment guidelines are called to assign weight scores to data items: the weights of core diabetes indicators such as glycated hemoglobin, fasting blood glucose, and 2-hour postprandial blood glucose are raised to the highest level, and the compression strategy is set to merge only completely duplicated continuous values ​​without horizontal screening; the weights of tumor markers and chest CT-related indicators are raised to the second highest level, and the compression strategy is set to remove only templated content with no medical significance, while retaining positive findings and time series; routine test indicators such as liver function and kidney function are set to routine weights, and standard vertical and horizontal compression is performed. Association-based hierarchical binding: All data items in the evidence group associated with pulmonary nodule diagnosis and treatment events are uniformly set to a weak consistency level to ensure that the data within the group is not over-compressed; All data items in the evidence group associated with diabetes blood glucose control diagnosis and treatment events are uniformly set to a strong consistency level to ensure the absolute security of core diagnosis and treatment data.

[0055] Write all hierarchical rules, weighting criteria, and binding relationships into the PDO audit log, and output a hierarchical dataset with dynamic weights and associated evidence group identifiers.

[0056] Implementation of Structured Semantic Compression Processing under Association Constraints Differentialized, linked compression is performed on the three types of datasets after classification: Strong consistency test data compression: First, CanonicalSchema is used to normalize the test item names, units, and reference ranges to resolve the issue of inconsistent or identical item names across different batches of tests. For the highest-weighted blood glucose-related indicators, only items with completely identical values ​​for three consecutive days are merged into a single time period record without any horizontal deletion. For liver and kidney function indicators with moderate weight, longitudinal compression is performed to merge consecutive duplicate values, while horizontal compression retains only outliers and indicators with fluctuations exceeding 20%, removing unchanged routine items. For tumor marker indicators associated with lung nodules, based on the constraints of the associated evidence set, the entire time series values ​​are retained without horizontal compression, ensuring a complete chain of evidence with positive findings in CT reports. In this example, the original test data contained 12 tests and 126 indicators; after compression, only 32 core indicators were retained, achieving a compression rate of 74.6%, with 100% of the core diagnostic and treatment indicators retained.

[0057] Weakly consistent text data compression: For chest CT reports, core fields such as examination site, positive findings, nodule size, density characteristics, spatial relationships, and diagnostic suggestions are extracted through syntactic reconstruction and semantic role localization. Templated redundant statements such as "further examination based on clinical recommendations" are removed, and the original text is reconstructed in structured field form, retaining line numbers and character range hashes. For admission records and progress notes, core content such as present medical history, positive signs, treatment plans, and disease assessments are extracted. Meaningless template paragraphs automatically generated by the system are removed. Based on associated evidence group constraints, descriptions related to lung nodules and blood glucose are kept chronologically aligned with corresponding test data without splitting associated semantic content. In this example, the total number of words in the original three CT reports was 2860. After compression, the structured core content was reduced to only 420 words, achieving a compression rate of 85.3%. All positive findings and medical features were completely preserved without semantic tampering.

[0058] Low-consistency administrative data compression: Use a fixed template for field rearrangement to uniformly output administrative content such as patient basic information and hospitalization information, and remove redundant formats.

[0059] For each compression operation, the operation rules, content hashes before and after the operation, and original text anchors are recorded in real time and written to the PDO full-link audit log to achieve full traceability of each single operation.

[0060] This step outputs a structured compressed data block with associated evidence groups bound to it.

[0061] Implementation of Closed-Loop Medical Consistency Verification and Audit For structured compressed data blocks, perform a four-layer closed-loop verification: Single-step real-time consistency verification: For each step of the compression process, the numerical values ​​and terms have been compared in real time. The pass rate of all single-step operations is 100%, with no cumulative deviation. Medical semantic equivalence verification: Based on SNOMEDCT, ICD-10 terminology database, and clinical logic rule base, the consistency of medical semantics before and after compression was verified: it was confirmed that the nodule features in the CT report were completely consistent with the subject attribution, with no feature mismatch; it was confirmed that the test indicators corresponded completely with the temporal and causal relationships of the medical records, with no logical misalignment; it was confirmed that the content of the two related evidence groups was complete, with no broken evidence chains, and the verification pass rate was 100%. End-to-end traceability verification: Verify the integrity of the audit logs in PDO to confirm that every step from data parsing to compression completion has rules to follow, hash values ​​are verifiable, and can be fully traced back to the original text. The entire process has a 100% traceability rate, which meets the medical data compliance requirements. Automatic deviation tracing and correction: No abnormal content was found in this verification, and a full-link audit report and verification result report were automatically generated.

[0062] This step outputs a verified VerifiedDataBlock, along with a complete audit and verification report.

[0063] Enhanced RAG-friendly structure generation and semantically consistent segmentation implementation For data blocks that pass verification, construct an enhanced RAG adapter structure: Enhanced two-layer structure construction: The core fact layer integrates compressed abnormal test results, CT positive findings, blood glucose change trends, drug dosage, key disease milestones and other core medical facts, binds the associated evidence group ID, and marks the contextual search tags for diabetes diagnosis and treatment and lung nodule assessment; the original text evidence layer corresponds to the core fact layer item by item, stores complete original text fragments, original text location anchors, audit traceability entry points, and the content of the same associated evidence group is stored together. Semantic consistency segmentation: Based on a medical event of a test, a CT report, and a disease progress node, chunk segmentation is performed. At the same time, the constraints of the associated evidence group are strictly followed. The associated content of the lung nodule diagnosis and treatment event and the diabetes blood glucose control event are treated as independent semantic blocks without splitting. Finally, 6 independent semantic blocks are generated without any semantic separation or splitting of associated clues. Final encapsulation: An enhanced RAGReadyPackage is generated, fully encompassing the core fact layer, original evidence layer, semantic block index, end-to-end audit report, and clinical scenario-based search tags. It can be directly imported into a vector database and seamlessly integrated with RAG-assisted diagnostic systems. This embodiment fully implements the entire technical solution of this invention, achieving an average compression rate of over 70% while ensuring 100% consistency of medical facts, complete preservation of core diagnostic and treatment information, unbroken cross-modal evidence chains, and end-to-end auditability and traceability, fully meeting the application requirements of clinical-grade RAG systems.

[0064] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0065] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for consistent compression of medical data suitable for knowledge-enhanced retrieval, characterized in that, include: (1) Multi-source medical data parsing and patient data normalization: format identification and differential parsing of multi-format and multi-source medical documents, extracting content and original text traceability information, adopting a patient unique identifier aggregation method based on hospital number + date of birth + name hash, mapping all data to extended patient-level data structure PDO, and outputting extended PDO in a unified format; (2) Clinical scenario-driven dynamic grading of medical attributes: Based on extended PDO, a three-level grading mechanism of basic classification + dynamic weight adjustment + associated grading binding is constructed. Combined with the rule base of the Chinese Medical Association's specialty diagnosis and treatment guidelines and the core observation index base of ICD-10 disease coding, a grading dataset with dynamic weight scores and associated evidence group binding labels is output. (3) Structured semantic compression processing under association constraints: Based on the hierarchical results, differentiated compression is performed on medical data with different consistency levels. Combined with the association evidence group constraint, the integrity of the association evidence chain is ensured. Each compression operation is recorded in real time, and structured compressed data blocks with association evidence group binding and single-step operation recording are output. (4) Full-process closed-loop medical consistency verification and audit: Construct a four-layer verification mechanism to verify the compression process and results throughout the entire process, realize automatic traceability and correction of deviations, and output the VerifiedDataBlock that has passed the verification and a complete audit and verification report; (5) Enhanced RAG-friendly structure generation and semantic consistency segmentation: Construct a two-layer structure of core fact layer and original evidence layer, perform semantic block segmentation based on medical event and related evidence group constraints, and finally encapsulate to generate an enhanced RAGReadyPackage that can be directly used in the RAG system.

2. The method according to claim 1, characterized in that, The multi-format medical files in step (1) include PDF, Excel and semi-structured text. When parsing PDF files, line numbers, page numbers, paragraph levels and coordinate information are retained. When parsing Excel files, the header is automatically identified, merged cells are disassembled and mixed multi-patient data is separated.

3. The method according to claim 1, characterized in that, The extended PDO includes four core fields: original content field, clinical scenario metadata field, diagnosis and treatment event association identifier field, and end-to-end audit log field. The clinical scenario metadata field includes the patient's primary diagnosis ICD code, comorbidity code, disease stage, department visited, and diagnosis and treatment goal.

4. The method according to claim 1, characterized in that, The basic classification layer in step (2) divides medical data into three categories: strong consistency data, weak consistency data, and low consistency requirement data. Among them, strong consistency data is numerical content such as test indicators, vital signs, and drug dosages; weak consistency data is narrative text such as descriptions of imaging examinations, admission records, and medical records; and low consistency requirement data is administrative content such as departments and bed numbers.

5. The method according to claim 1, characterized in that, The dynamic weight adjustment layer in step (2) uses the clinical scenario metadata in PDO to call the standardized diagnosis and treatment guidelines and the core disease observation index library to dynamically quantify and score the medical weight of each data item, and adjust its consistency level and compression strategy priority.

6. The method according to claim 1, characterized in that, In step (3), the strong consistency data compression first normalizes the test item name, unit, reference range and numerical format through CanonicalSchema, and then performs differential compression based on the dynamic weight grading results. The test indicators corresponding to the positive findings in the weak consistency data completely retain the time series information.

7. The method according to claim 1, characterized in that, The weak consistency data compression in step (3) extracts core medical information and restructures it through syntactic reconstruction, semantic role positioning and extraction of key medical fields, removes template-based redundant statements, and maintains the temporal and logical correspondence between the structured restructure and the timestamps and indicator results of the associated strong consistency data.

8. The method according to claim 1, characterized in that, The four-layer verification mechanism in step (4) includes: single-step real-time consistency verification, medical semantic equivalence verification, full-link traceability verification, and automatic deviation tracing and correction mechanism. Among them, the medical semantic equivalence verification is implemented based on the SNOMEDCT, LOINC, ICD standard medical terminology library and clinical logic rule library.

9. The method according to claim 1, characterized in that, The enhanced two-layer structure in step (5) includes a core fact layer containing structured key medical facts, associated evidence group IDs, and clinical scenario-based search tags, which is suitable for vectorized input; the original evidence layer corresponds to the core fact layer item by item, and saves complete original text fragments, location indexes, and full-link audit traceability entry points.

10. The method according to claim 1, characterized in that, The semantic consistency segmentation in step (5) is based on the medical event to segment semantic blocks, and the content within the same related evidence group is not split into different chunks, so as to ensure the semantic integrity of the medical event and the integrity of the related evidence chain.

Images