A method and system for unstructured data extraction of medical examination reports

Abstract

The present application relates to the technical field of unstructured data processing, in order to solve the problems of insufficient precision of existing systems for medical report extraction, incomplete scene coverage and the like, a kind of medical examination report unstructured data extraction method and system are provided, through the five core steps of "multi-modal input classification and preprocessing, text block feature clustering partition, adaptive rule generation, entity extraction, man-machine collaborative iterative optimization", effectively solve the technical problems of poor adaptability, low accuracy and high maintenance cost of the prior art, suitable for various medical, examination report key information automatic extraction scene in medical field.

Description

Technical Field

[0001] This invention relates to the field of unstructured data processing technology, specifically to a feature extraction method and system for medical examination reports, applicable to the automated extraction of key information from various medical and examination reports in the medical field. Background Technology

[0002] Medical examination reports are the core basis for medical diagnosis, treatment planning, and prognosis assessment. Descriptive features in these reports, such as the size, color, arrangement, and shape of pathological lesions, are key indicators for doctors to determine the nature and stage of development of lesions. Currently, the extraction of these unstructured data features mainly relies on manual input, which has the following technical limitations: Inefficient: Manual extraction requires reading each report and organizing the information, which is difficult to meet the high-efficiency processing needs of clinical and scientific research when faced with a large number of reports; High error rate: Manual extraction is easily affected by subjective factors and fatigue, resulting in problems such as omissions, errors, and inconsistent formats, which affect the accuracy of subsequent data applications; Poor adaptability: The report formats of different hospitals and departments vary greatly, and the expression methods are flexible. Traditional fixed rule matching or general OCR tools are difficult to adapt to diverse scenarios, and the extraction accuracy is limited. High maintenance costs: When adding new hospital templates or encountering rare expressions, rules need to be rewritten manually, making dynamic adaptation impossible.

[0003] In existing technologies, some solutions attempt to extract data using general text extraction tools or simple keyword matching. However, these solutions are not optimized for the professionalism and diverse formats of medical reports, resulting in insufficient extraction accuracy and incomplete scenario coverage. They fail to meet the high requirements of the medical field for accurate and stable data extraction. Therefore, there is an urgent need for a solution that adapts to multimodal input, accurately extracts the core features of reports, and can be dynamically optimized for unstructured data extraction. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method and system for extracting unstructured data from medical examination reports.

[0005] On the one hand, a method for extracting unstructured data from medical examination reports includes the following steps: S1. Automatically classify, identify, and preprocess multimodal reports to be processed, and output standardized text blocks and feature vectors to provide data sources for subsequent partitioning and extraction; S2. Perform text block feature mining and clustering partitioning on the standardized text blocks and feature vectors obtained in S1 to accurately locate the core feature description region. S3. Through small sample learning, core feature classification extraction rules adapted to different expression habits are automatically generated to achieve dynamic adaptation; S4. Based on the classification extraction rules and the core region of the localization, the entity extraction module completes the extraction of core features and outputs the extraction results.

[0006] Furthermore, by leveraging human-machine collaborative feedback, the rules and dictionary are dynamically iterated, continuously improving the accuracy of core feature extraction.

[0007] On the other hand, a system for extracting unstructured data from medical examination reports includes at least: Classification preprocessing module: used to realize automatic classification and recognition of medical reports, perform differentiated preprocessing on raw documents and scanned copies, and output standardized text blocks and feature information; Feature clustering module: used to extract multidimensional features of text blocks, automatically divide functional regions through unsupervised clustering algorithm, and accurately locate the core description area; Rule generation module: Used to automatically generate and optimize extraction rules through small sample association mining, and build a dynamic rule base; Entity extraction module: Extracts core features based on a dynamic rule base and inputs them into the result output module; Results output module: Used to verify the rationality of extracted entity features and output structured JSON data, supporting integration with medical systems; Iterative optimization module: used to display extraction results, record manual correction feedback, realize incremental updates of rules and dictionary, and iteratively optimize extraction performance.

[0008] Technical effects: This invention forms a complete technical closed loop through the innovative design of five core steps: multimodal input classification and preprocessing, text block feature clustering and partitioning, adaptive rule generation, entity extraction, and human-machine collaborative iterative optimization. It has achieved significant technical breakthroughs and beneficial effects in the extraction of unstructured data from medical reports.

[0009] Specifically, this invention first innovates in multimodal input classification and preprocessing, employing a dual-judgment mechanism to automatically classify report types and specifically optimize the preprocessing algorithm, resolving the contradiction between efficiency and accuracy in multi-carrier report processing, unlike traditional single preprocessing schemes. Building upon this, it innovates in text block feature clustering and partitioning, using unsupervised clustering combined with multidimensional features to automatically divide functional areas, adapting to the format differences of medical reports from over 80% of hospitals without manual template annotation, overcoming the limitations of traditional fixed-area positioning. Subsequently, it innovates in adaptive rule generation, using small-sample association mining to automatically generate dynamic rules and support… Prioritization and sorting, combined with rule matching, syntactic parsing fusion, and cross-text block feature fusion in entity extraction innovation, effectively solve the extraction errors caused by sentence interference and expression splitting. It achieves high-precision extraction with an accuracy rate of ≥95% for clear original documents, ≥88% for blurry scanned documents, and an overall accuracy rate of ≥92%. At the same time, relying on the fully automated processing, the processing time of a single report is ≤3 seconds (including OCR processing), which significantly reduces labor costs. Finally, through human-machine collaborative iterative optimization, a lightweight feedback closed loop is built to achieve collaborative optimization of rules, dictionaries, and models through incremental updates without full retraining, which significantly reduces long-term maintenance costs.

[0010] In summary, compared with existing general OCR or fixed rule solutions, this invention has achieved comprehensive technological breakthroughs in multimodal adaptability, dynamic rule generation, accurate partition extraction, and continuous optimization capabilities. It effectively solves the core pain points of existing technologies, such as poor adaptability, low accuracy, and high maintenance costs. Its structured output format facilitates subsequent storage, analysis, and integration into medical systems, and can be widely used in scenarios such as clinical diagnostic assistance, medical research data organization, and medical quality management. It has outstanding technological advancements and broad medical application value. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the overall process of the unstructured data extraction method in this embodiment. Detailed Implementation

[0012] This embodiment focuses on five key steps: multimodal input classification and preprocessing, text block feature clustering and partitioning, adaptive rule generation, entity extraction, and human-machine collaborative iterative optimization. It constructs a complete core feature extraction process for medical reports and achieves accurate and efficient extraction of core features through multimodal adaptation, clustering and partitioning, dynamic rules, and collaborative optimization mechanisms.

[0013] On the one hand, such as Figure 1 As shown, S1, as a fundamental step in the extraction process, addresses the multi-carrier characteristics of medical reports by implementing input classification and targeted preprocessing to provide a high-quality data source for subsequent partitioning and extraction. Specifically, it includes: Based on a dual approach of document metadata and image features, the system automatically distinguishes between input types: native electronic documents (PDF / Word) and scanned documents / photos, avoiding efficiency losses and accuracy degradation caused by uniform processing. For native electronic documents, it quickly identifies them through file header information and copyable text attributes. Specifically, it uses the document parsing tool PyMuPDF to extract text content and layout information (font size, character coordinates, paragraph spacing, paragraph position) for targeted preprocessing, preserving the spatial relationships and hierarchical relationships of text blocks while removing redundant formatting from the document itself. For scanned documents / photos, it completes recognition through image clarity and character edge continuity features, achieving a classification accuracy of ≥99%. Specifically, it employs a four-step algorithm for scanned documents / photos: "adaptive denoising → tilt correction → binarization enhancement → edge repair," specifically addressing issues such as blurry printing of medical reports, stamp shadows, paper wrinkles, and tilt distortion to complete preprocessing. Then, it uses an OCR tool to extract text and character coordinates, simulating the text block structure of the native document to improve the accuracy of subsequent processing.

[0014] The S2 algorithm uses an unsupervised clustering algorithm to perform feature mining and region segmentation on the preprocessed text blocks, accurately locating the core feature description regions and solving the problem of region positioning for different report formats. Specifically, it includes: Text block feature extraction involves extracting multi-dimensional features from each text block to construct a text block feature vector, including spatial features (center coordinates X / Y, aspect ratio, and spacing with adjacent text blocks), layout features (font size, font weight, and whether it is bold), and content features (text length and feature word density). The feature vector is then standardized to eliminate dimensional differences and ensure the stability of the clustering results in the next step.

[0015] Unsupervised clustering partitioning is performed using the DBSCAN unsupervised clustering algorithm to cluster the feature vectors of the text blocks, automatically dividing the report into functional areas such as patient information, core description, diagnosis conclusion, and signature / remarks. This eliminates the need for manual template annotation, adapting to the different layouts of various hospitals. Next, adaptive optimization of clustering parameters is performed, automatically adjusting the cluster neighborhood radius (eps) and minimum sample size (min_samples) through text block density statistics to avoid partitioning errors caused by fixed parameters. Regions containing high-frequency feature words such as "size," "presentation," "arrangement," "shape," and "color" are selected from the clustering results as core feature description regions, achieving precise target region identification.

[0016] S3 specifically includes S31, sample feature mining: Input 10-20 medical field annotation reports with different templates, extract the expression features of each core feature, including keyword context, sentence structure, and feature value association patterns; use the Apriori association rule mining algorithm to mine the association relationship between feature keywords and feature values, and the matching rules between keywords and sentence structure, to form an initial association rule set.

[0017] S32. Dynamic Rule Generation and Optimization: Based on the initial association rule set, classification extraction rules are automatically generated: size features correspond to "keyword + number + unit" regularization rules, and color / shape / arrangement features correspond to "keyword + feature value" semantic association rules; Furthermore, redundant rules can be removed, high-frequency rules can be given high priority, and rare rules can be given low priority, thus completing rule deduplication and priority sorting to ensure extraction efficiency and accuracy. At the same time, a rule extension channel is reserved to support the addition of new rules through human-machine collaborative feedback without refactoring the core code.

[0018] The S4 method employs a combined strategy of "rule matching + syntactic parsing" to achieve both numerical and textual feature extraction. Numerical feature extraction is based on adaptively generated regularization rules, capturing the "number + unit" combination in the core feature description region. It supports single-dimensional (diameter) and multi-dimensional (length × width × height) numerical extraction, while simultaneously standardizing units, such as converting them to cm or mm. Textual feature extraction is based on semantic association rules, matching keywords with feature values, and combining part-of-speech tagging (filtering nouns and adjectives) to initially extract target entities.

[0019] Furthermore, the preliminary results were refined. Using the dependency parsing tool Spacy, semantic associations between keywords and feature values ​​were established, eliminating mismatches caused by sentence inversion and parenthetical phrases (e.g., in "bluish-gray, nipple-shaped," accurately associating "bluish-gray" with "color" and "nipple-shaped" with "shape"). Simultaneously, for feature descriptions that might be split after clustering (e.g., size descriptions spanning two lines), cross-text block feature fusion was performed based on text block spatial association to ensure the completeness of the extracted results.

[0020] In this embodiment, a closed loop of "extraction-verification-optimization" is further constructed. Through human-machine collaborative feedback, the rules and dictionary are dynamically iterated to continuously improve extraction accuracy. Specifically, this includes: manually correcting data marked as "to be verified" or "suspected new features," recording correction logs, including error type (missed / incorrectly extracted / mismatched), original text, correct feature values, new expression patterns, etc. New expression patterns in the manually corrected logs are automatically converted into new rules, added to the rule base, and their priorities are updated. New feature values ​​confirmed manually (such as rare colors or arrangement patterns) are automatically added to the corresponding feature value dictionary. Furthermore, a high-efficiency parameter fine-tuning technique is employed to update the clustering and extraction logic based on corrected samples, eliminating the need for full retraining and reducing optimization costs.

[0021] On the other hand, a corresponding unstructured data extraction system is constructed based on unstructured data extraction methods. Specifically, this includes a classification preprocessing module for automatic classification and recognition of medical reports, performing differentiated preprocessing on raw documents and scanned copies, and outputting standardized text blocks and feature information; a feature clustering module for extracting multi-dimensional features from text blocks, automatically dividing functional regions through unsupervised clustering algorithms, and accurately locating core descriptive areas; a rule generation module for automatically generating and optimizing extraction rules through small-sample association mining, and constructing a dynamic rule base; an entity extraction module for extracting core features based on the dynamic rule base and inputting them into the result output module; a result output module for verifying the rationality of extracted entity features, outputting structured JSON data, and supporting integration with medical systems; and an iterative optimization module for displaying extraction results, recording manual correction feedback, implementing incremental updates of rules and dictionaries, and iteratively optimizing extraction performance.

[0022] The implementation based on the aforementioned unstructured data extraction system begins with the installation of software and dependent libraries (PaddleOCR, OpenCV, Jieba, Spacy, etc.), downloading and configuring the syntactic analysis model, and building a human-computer interaction verification interface. Ten to twenty labeled medical report samples are input, and the rule generation module initializes the rule base while simultaneously constructing an initial feature value dictionary. The reports to be processed are then input into the unstructured data feature extraction system. The classification preprocessing module automatically completes the classification, performs differential preprocessing, and outputs standardized text blocks and feature vectors. The feature clustering module extracts multi-dimensional features from the text blocks, clusters them using the DBSCAN algorithm, and identifies the core descriptive regions. Based on the initialized rule base, the entity extraction module performs initial extraction and syntactic refinement of the core features, outputting preliminary results. The results are manually verified, abnormal data is corrected, and the iterative optimization module automatically updates the rule base and dictionary, optimizing the clustering and extraction logic. The optimized extraction results are then validated for reasonableness, and structured JSON data is output, thus completing the entire extraction process.

[0023] The comparison results with existing technologies are shown in the table below: Table 1

[0024] The following shows the extraction results of two special medical reports after processing in this embodiment.

[0025] Special Case 1: Blurred scanned document report (with stamp interference) Input report text fragment (after OCR extraction): "Microscopic findings: Lesion diameter 8mm, light pink in color, oval in shape, with diffuse cell distribution. [Stamped interfering text]" Extraction results: { "Medical size": "0.8cm", "Medical Color": "Light Pink" "Arrangement characteristics": "Diffuseness" "Medical Shape": "Oval" "Verification Status": "Normal", "Suspected new feature value": [], Source extracted from: "Chapter viewed under the microscope" } Test results: Extraction accuracy 95%, processing time 2.8 seconds.

[0026] Special Case 2: Inverted Sentence Structure + Rare Expression Report Enter the report text snippet: "Gross examination: bluish-gray, papillary, approximately 1.2 × 0.9 cm in size, with cells arranged in a sieve pattern." Extraction results: { Medical Size: 1.2cm × 0.9cm "Medical color": null, Arrangement characteristics: "sieve-like" "Medical Shape": "Nipple-like" Verification Status: "Pending Verification" "Suspected new feature value": ["bluish-gray"], Source of extraction: "General Examination Section" } Test results: After manual correction, "cyan-gray" was added to the color feature value dictionary, and subsequent similar descriptions can be automatically extracted. The processing time was 2.1 seconds.

[0027] All content not described in detail in this specification belongs to the prior art known to those skilled in the art. Furthermore, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for extracting unstructured data from medical examination reports, characterized in that, Includes the following steps: S1. Automatically classify, identify, and preprocess multimodal reports to be processed, and output standardized text blocks and feature vectors to provide data sources for subsequent partitioning and extraction; S2. Perform text block feature mining and clustering on the standardized text blocks and feature vectors obtained in S1 to accurately locate the core feature description region. S3. Through small sample learning, core feature classification extraction rules adapted to different expression habits are automatically generated to achieve dynamic adaptation; S4. Based on the classification extraction rules and the core region of the localization, the entity extraction module completes the extraction of core features and outputs the extraction results.

2. The method for extracting unstructured data from medical examination reports according to claim 1, characterized in that, It also includes using human-machine collaborative feedback to achieve dynamic iteration of rules and dictionaries, continuously improving the accuracy of core feature extraction.

3. The method for extracting unstructured data from medical examination reports according to claim 1, characterized in that, S2 extracts multi-dimensional features from each text block to construct a text block feature vector, including spatial features, layout features, and content features. At the same time, the feature vector is standardized. The DBSCAN unsupervised clustering algorithm is used to cluster the text block feature vectors. Then, the clustering parameters are adaptively optimized. The clustering neighborhood radius and minimum number of samples are automatically adjusted by the text block density statistics. The regions with high-frequency feature words are selected from the clustering results as the core feature description regions to accurately lock the target region.

4. The method for extracting unstructured data from medical examination reports according to claim 1, characterized in that, S3 specifically includes S31, Sample Feature Mining: Input multiple different template reports, extract the expression features of each core feature, including keyword context, sentence structure, and feature value association patterns; use the Apriori association rule mining algorithm to mine the association relationship between feature keywords and feature values, and the matching rules between keywords and sentence structures, to form an initial association rule set; S32, Dynamic Rule Generation and Optimization: Based on the initial association rule set, automatically generate classification extraction rules: size features correspond to "keyword + number + unit" regularization rules, and color / shape / arrangement features correspond to "keyword + feature value" semantic association rules.

5. The method for extracting unstructured data from medical examination reports according to claim 4, characterized in that, Remove redundant rules, set high priority for frequently used rules and low priority for rarely used rules, and complete rule deduplication and priority sorting.

6. The method for extracting unstructured data from medical examination reports according to claim 1, characterized in that, The S4 includes numerical feature extraction and text feature extraction. The numerical feature extraction is based on adaptively generated regular rules to capture the "number + unit" combination in the core feature description region. The text feature extraction is based on semantic association rules to match keywords and feature values.

7. A system for extracting unstructured data from medical examination reports, characterized in that, At least including: Classification preprocessing module: Used to automatically classify and recognize medical reports, perform differentiated preprocessing on raw documents and scanned copies, and output standardized text blocks and feature information; Feature clustering module: used to extract multidimensional features of text blocks, automatically divide functional regions through unsupervised clustering algorithm, and accurately locate the core description area; Rule generation module: Used to automatically generate and optimize extraction rules through small sample association mining, and build a dynamic rule base; Entity extraction module: Extracts core features based on a dynamic rule base and inputs them into the result output module; Results output module: Used to verify the rationality of extracted entity features and output structured JSON data, supporting integration with medical systems; Iterative optimization module: used to display extraction results, record manual correction feedback, realize incremental updates of rules and dictionary, and iteratively optimize extraction performance.

Images