Medical guide document analysis and content extraction system and method based on natural language processing and computer vision technology

Abstract

The invention relates to a medical guide document analysis and content extraction system based on natural language processing and computer vision technology, the system comprises a layout detection model, a formula detection model, a table recognition model, a formula recognition model and an OCR (optical character recognition) model, the layout detection model is used for positioning different elements in a document; the formula detection model is used for positioning a formula in a document; and the table identification model is used for detecting table boundaries, segmenting table cells and analyzing a complex structure. The invention further provides a medical guide analysis and content extraction method applying the medical guide document analysis and content extraction system based on the natural language processing and computer vision technology. According to the invention, efficient analysis and accurate extraction of the medical guide document are realized. The system can accurately identify the text, table, formula and picture information in the document, improves the medical information processing efficiency, enables the acquisition of medical knowledge to be more convenient and efficient, and facilitates the improvement of the medical service quality and research efficiency.

Description

Technical Field

[0001] The present invention relates to a medical guideline document parsing and content extraction system and method, and in particular to a medical guideline document intelligent parsing and content extraction system and method based on natural language processing (NLP) and computer vision technology, belonging to the field of medical information processing and artificial intelligence technology. Background Art

[0002] With the rapid development of informatization in the medical industry, professional documents such as medical guidelines, clinical pathways, and technical specifications are increasingly being published electronically. PDF, in particular, has become the primary medium for storing and disseminating medical literature. However, medical guidelines often contain complex layouts, such as multi-column layouts, nested tables, mathematical formulas, charts, and specialized terminology. This makes it difficult for traditional text extraction tools to meet the requirements of high accuracy and efficiency.

[0003] In recent years, the rapid development of artificial intelligence and natural language processing technologies has provided new solutions for parsing medical guideline documents. Deep learning models can accurately identify different regions within documents, optical character recognition (OCR) models excel at recognizing medical terminology and complex typography, formula recognition technology based on specialized models accurately parses mathematical formulas, and table parsing models effectively process complex tables and generate structured data. By combining these advanced technologies, structured data can be efficiently and accurately extracted from medical guideline documents, providing a high-quality data foundation for subsequent medical knowledge management, intelligent question-answering systems, and clinical decision support. This tool not only enhances the automation of medical information processing but also provides strong technical support for the digital transformation of the healthcare industry.

[0004] Chinese patent document CN119848169A discloses a document processing method and system based on text content extraction. It verifies and identifies secure documents from uploaded documents and saves them to a distributed repository. It also constructs a secure document status table to represent the relationship between document attributes and storage addresses, thereby realizing orderly distributed storage of documents and high-disaster-tolerant storage of documents. It calls the Apache Tika library to extract the text content of secure documents from the distributed repository, generates index text corresponding to the secure documents, performs full-text retrieval on this basis, and feeds the retrieval results back to the front end, thereby improving the accuracy and efficiency of document retrieval. It downloads the complete document from the distributed repository and generates a decoded data stream. Based on the working status of the front end, it previews the decoded data stream online and adjusts the document delivery status of the distributed repository. It uses the Apache Tika library to realize full-text text retrieval of the document, thereby improving the document retrieval matching degree and achieving efficient and accurate search. It also provides online preview of documents to improve the convenience and sensitivity of document editing and use.

[0005] Chinese patent document CN115546817A discloses a method and device for document parsing. By using an OCR model to identify the text content and coordinates in an image, and a layout model to identify the layout structure category and coordinates in an image, and calibrating the coordinates of each part of the layout structure, accurate text classification can be achieved, ultimately obtaining a more accurate and complete parsing result. During operation, the user can parse file information in a targeted manner based on the layout structure category. The document parsing method and device provided in this application can identify a variety of layout structures and can customize personalized data parsing solutions for different layout categories, making it more flexible and professional.

[0006] However, there are currently no reports on intelligent parsing and content extraction systems and methods for medical guideline documents based on natural language processing (NLP) and computer vision technology. Summary of the Invention

[0007] The first object of the present invention is to address the deficiencies in the prior art and provide a medical guideline document parsing and content extraction system based on natural language processing and computer vision technology.

[0008] The second object of the present invention is to apply a medical guideline document parsing and content extraction method of a medical guideline document parsing and content extraction system based on natural language processing and computer vision technology.

[0009] To achieve the above-mentioned first purpose, the technical solution adopted by the present invention is: a medical guide document parsing and content extraction system based on natural language processing and computer vision technology, including a layout detection model, a formula detection model, a table recognition model, a formula recognition model and an OCR model. The layout detection model: locates different elements in the document; the formula detection model: locates formulas in the document, including in-line formulas and inter-row formulas; the table recognition model: detects table boundaries, splits cells, and parses complex structures; uses the TableMaster model to detect the located table area; the formula recognition model: recognizes the formula image as LaTeX source code; the OCR model: extracts text content from the image, including position recognition and content recognition, and uses the PaddleOCR model for extraction.

[0010] As a preferred example, the layout detection model is fine-tuned based on iterative data of the LayoutLMv3 model, and the fine-tuned model performs well on diverse documents; the formula detection model is fine-tuned based on the YOLOv8 model, and its detection accuracy of the formula area is optimized through fine-tuning; the formula recognition model is fine-tuned based on the UniMERNet model, and its accuracy of formula recognition is optimized through fine-tuning.

[0011] As a preferred example, the different elements in the positioning document include images, tables, text, titles, and formulas; the TableMaster decomposes the task into four subtasks: table detection: determining the overall bounding box of the table, cell segmentation: dividing the table into rows, columns, and cells, text recognition: extracting the text content in the cell in combination with the PaddleOCR model, and structure analysis: analyzing the row and column structure of the table and the relationship between merged cells.

[0012] To achieve the second objective, the present invention adopts a technical solution: a method for parsing and extracting medical guideline content using a medical guideline document parsing and content extraction system based on natural language processing and computer vision technology, comprising the following steps:

[0013] S1: Data preprocessing, consisting of the following steps:

[0014] S1-1: Data screening: Screen out files that cannot be processed and cannot process encrypted documents;

[0015] S1-2: Language recognition; supports processing Chinese or English documents;

[0016] S1-3: Document type classification: distinguish between scanned and text-based PDFs;

[0017] S1-4: Content garbled code detection;

[0018] S1-5: Page metadata extraction;

[0019] S2: Document content analysis, consisting of the following specific steps:

[0020] S2-1: Layout Analysis: Divide the document into regions based on the fine-tuned layout detection model, accurately locate the position of each element in the document by returning coordinates, and analyze the spatial relationship between elements;

[0021] Reading order analysis: identify single-column and multi-column layouts, and handle multi-column layouts or complex typesetting;

[0022] S2-2: Formula detection and recognition: The formula detection model is used to locate the formula area, and the formula recognition model is used to convert the formula image into LaTeX format;

[0023] S2-3: Table recognition: TableMaster and StructEqTable perform table recognition tasks and can execute Table-to-LaTex tasks;

[0024] S2-4: OCR recognition;

[0025] S3: Document content post-processing, which consists of the following specific steps:

[0026] S3-1: Processing bounding boxes: removing formulas and text blocks contained within image and table areas, as well as boxes contained within formula boxes;

[0027] S3-2: Information integration;

[0028] S4: Format conversion.

[0029] As a preferred example, the S2-4: OCR recognition: after excluding tables, formulas, and images in the document, use the OCR model to perform text area recognition; for text blocks containing inline formulas, first use the coordinates provided by the formula detection model to mask the formula, then perform OCR, and finally reinsert the formula into the OCR result.

[0030] As a preferred example, S3-1: For partially overlapping text boxes, avoid mutual coverage by shrinking vertically and horizontally. For partial overlap between text and tables and images, temporarily ignore the tables and images. S3-2: Based on the output of S, remove useless information in the header and footer, divide the entire page into multiple areas, each area contains multiple bounding boxes, and each area contains at most one column.

[0031] As a preferred example, S4: format conversion, generates markdown or json format according to the result of S3-2 processing.

[0032] As a preferred example, the aforementioned S1-3: for text-type PDFs, directly use PyMuPDF for text extraction; for scanned PDFs, an OCR model is required; S1-4: content garbled code detection: some text-based documents will have garbled code when copying text, and such documents must be identified in advance during the preprocessing stage; S1-5: page metadata extraction: extract the total number of pages and page size of the document.

[0033] As a preferred example, the output formats of the S2-1 are two: JSON format and visual annotation; the S2-1 layout analysis divides the document into regions based on the fine-tuned layout detection model, locates the positions of titles, text, pictures, tables, formulas, footnotes, side notes, and page number elements, and analyzes the spatial relationship between elements, namely nesting, overlapping, and contextual association; S2-1 reading order analysis: judge single column and multi-column based on the position information of the text block, determine the reading path based on the order of element arrangement, and handle multi-column layout or complex typesetting.

[0034] The advantages of this invention are that it enables efficient parsing and precise extraction of medical guideline documents. The system can accurately identify text, tables, formulas, and images within documents, improving the efficiency of medical information processing and making the acquisition of medical knowledge more convenient and efficient, thereby enhancing the quality of medical services and research efficiency.

[0035] 1. Technical Effect

[0036] 1) Improve guideline analysis efficiency: Model-automated case analysis can save 50% of manual editing

[0037] Above the time, the structured accuracy is >95%.

[0038] 2) Accuracy: The expert review pass rate for guideline interpretation in terms of completeness, format standardization, and terminology consistency reached over 95%;

[0039] 3) Clinical decision support: The accuracy of preliminary diagnosis of common internal medicine diseases has been improved by approximately

[0040] 20%, and provide a traceable chain of evidence.

[0041] 4) Data traceability: Completely record the model reasoning process to meet medical regulatory requirements.

[0042] 2. Economic effects

[0043] 1) Reduce the cost of manual guide compilation by approximately 50%;

[0044] 2) Shorten the diagnosis and treatment cycle, increase bed turnover rate, and help improve the overall hospital admission rate

[0045] Ability and income level.

[0046] 3. Social impact

[0047] 1) Improve the quality and efficiency of medical services and enhance patients’ medical experience;

[0048] 2) Promote the development of smart healthcare and precision diagnosis and treatment, and promote the optimal allocation of medical resources and the improvement of overall public health levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Flowchart of medical guide PDF parsing and content extraction.

[0050] Figure 2 Layout analysis output and visualization results.

[0051] Figure 3 Formula detection and recognition visualization results.

[0052] Figure 4 Output and visualization of table recognition results.

[0053] Figure 5 Output and visualization of image recognition results.

[0054] Figure 6 Final Markdown output result.

[0055] Figure 7 The final json output result. DETAILED DESCRIPTION

[0056] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content of the present invention record, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the application's appended claims.

[0057] Example 1

[0058] A medical guideline document parsing and content extraction system based on natural language processing and computer vision technology, mainly composed of the following models:

[0059] Layout Detection Model: This model locates different elements in a document, including images, tables, text, titles, and formulas. Existing layout detection algorithms have difficulty identifying diverse documents. We collected documents from different types, screened representative samples through cluster analysis, established fine-grained annotation standards, and annotated these samples. We iteratively fine-tuned the data based on the LayoutLMv3 model (Microsoft's multimodal Transformer model, a pre-trained model for document image analysis). The resulting model performed well on diverse documents.

[0060] Formula Detection Model: This model locates formulas within documents, including both inline and interline formulas. Inline formulas are those interspersed between text, while interline formulas occupy a single line. This model is fine-tuned based on the YOLOv8 model (Ultralytics' YOLO family of object detection algorithms) to optimize its detection accuracy for formula areas.

[0061] Table recognition model: Detects table boundaries, segments cells, and parses complex structures. Uses the TableMaster model to detect the located table area. TableMaster breaks down this task into four subtasks.

[0062] Table detection: Determine the overall bounding box of the table.

[0063] Cell Splitting: Split the table into rows, columns and cells.

[0064] Text recognition: Combined with the PaddleOCR model to extract the text content in the cell.

[0065] Structural analysis: Analyze the row and column structure of the table and the relationship between merged cells.

[0066] Formula Recognition Model: This model recognizes formula images as LaTeX source code. LaTeX source code is a text-based markup language specifically used for typesetting and writing content containing complex mathematical formulas, scientific documents, and technical documentation. This model is fine-tuned based on the UniMERNet model to optimize its formula recognition accuracy.

[0067] OCR model: Extracts text from images, including location and content recognition. Uses the PaddleOCR model for extraction.

[0068] Example 2:

[0069] See Figure 1 , Figure 1 This is a flowchart of medical guide PDF parsing and content extraction, a medical guide parsing and content extraction method according to application embodiment 1, comprising the following steps:

[0070] Step 1: Data preprocessing, which consists of the following specific steps:

[0071] Step 1-1: Data screening: Filter out files that cannot be processed (encrypted documents and password-protected documents). Encrypted documents cannot be processed.

[0072] Step 1-2: Language identification; supports processing Chinese or English documents. The OCR model needs to specify the language type to help improve recognition accuracy.

[0073] Steps 1-3: Document type classification: Distinguish between scanned and text-based PDFs. For text-based PDFs, directly use PyMuPDF for text extraction. PyMuPDF is a high-performance Python library for data extraction, analysis, conversion, and manipulation of PDF documents. For scanned PDFs, an OCR model is required. Scanned PDFs are recognized based on their characteristics: large image areas covering the entire page and an average text length per page close to zero.

[0074] Steps 1-4: Detecting Garbled Content: Some text-based documents may display garbled text when copied. Pre-processing these documents allows for OCR text recognition in the next step, thus avoiding content extraction errors caused by garbled text.

[0075] Step 1-5: Page metadata extraction: Extract the total number of pages and page dimensions (width and height) of the document.

[0076] Step 2: Document content analysis, consisting of the following specific steps:

[0077] Step 2-1: Layout Analysis: Based on the fine-tuned layout detection model, the document is divided into regions and the positions of the title, body text, images, tables, formulas, footnotes, side notes, and page numbers are located. The returned coordinates accurately locate each element within the document. Spatial relationships between elements (nesting, overlap, and contextual associations) are analyzed.

[0078] Reading order analysis: Determine single-column and multi-column layouts based on the position of text blocks, determine the reading path based on the order of element arrangement, and handle multi-column layouts or complex typesetting.

[0079] There are two output formats: JSON format: contains element type, coordinates, text content, and hierarchical relationship metadata. Visual annotation: PDF with bounding boxes to intuitively display the detection results.

[0080] Step 2-2: Formula detection and recognition: Use the formula detection model to locate the formula area (in-line formulas and inter-line formulas), and use the formula recognition model to convert the formula image into LaTeX format to ensure accuracy.

[0081] Step 2-3: Table recognition: TableMaster and StructEqTable perform table recognition tasks and can execute Table-to-LaTex tasks.

[0082] Steps 2-4: OCR Recognition: After excluding tables, formulas, and images from the document, the OCR model is used to identify text regions. To avoid text order errors caused by full-page OCR, OCR is performed based on the text regions (titles, paragraphs) detected by layout analysis. For text blocks containing inline formulas, the formulas are first masked using the coordinates provided by the formula detection model, followed by OCR. Finally, the formulas are reinserted into the OCR results, ensuring accurate text recognition and reading order.

[0083] Layout analysis output and visualization results are shown in Figure 2 , the visualization results of formula detection and recognition are shown in Figure 3 , the output and visualization results of table recognition are shown in Figure 4 , Figure 5 The output and visualization results of image recognition can be found in Figure 5 .

[0084] Step 3: Document content post-processing, which consists of the following specific steps:

[0085] Step 3-1: Process bounding boxes: Remove formulas and text blocks within image and table areas, as well as boxes within formula boxes. For partially overlapping text boxes, shrink them vertically and horizontally to prevent overlap, ensuring their final position and content are intact and facilitating subsequent sorting. For partial overlap between text, tables, and images, temporarily ignore the tables and images to ensure text integrity.

[0086] Step 3-2: Information integration. Based on the output of step 2, remove useless information from headers and footers, divide the entire page into multiple areas, each area contains multiple bounding boxes, and each area contains at most one column. Ensure that the text is read line by line in a natural reading order, determine the reading order of each element in the document, and integrate the content.

[0087] Step 4: Format conversion: Based on the results of step 3-2, markdown and json formats can be generated.

[0088] The final Markdown output is shown in Figure 6 , the final json output result is shown in Figure 7 .

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the method of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A medical guideline document parsing and content extraction system based on natural language processing and computer vision technologies, comprising a layout detection model, a formula detection model, a table recognition model, a formula recognition model, and an OCR model, characterized by: The layout detection model locates different elements in the document; the formula detection model locates formulas in the document, including in-line formulas and inter-row formulas; the table recognition model detects table boundaries, splits cells, and parses complex structures; the located table area is detected using the TableMaster model; the formula recognition model recognizes formula images as LaTeX source code; the OCR model extracts text content from images, including position recognition and content recognition, using the PaddleOCR model for extraction.

2. The medical guideline document parsing and content extraction system according to claim 1, characterized in that: The layout detection model is fine-tuned based on iterative data of the LayoutLMv3 model, and the fine-tuned model performs well on diverse documents. The formula detection model is fine-tuned based on the YOLOv8 model, and its detection accuracy for formula areas is optimized through fine-tuning. The formula recognition model is fine-tuned based on the UniMERNet model, and its accuracy for formula recognition is optimized through fine-tuning.

3. The medical guideline document parsing and content extraction system according to claim 1, characterized in that: The different elements in the positioning document include images, tables, text, titles, and formulas. The TableMaster decomposes the task into four subtasks: table detection: determining the overall bounding box of the table, cell segmentation: dividing the table into rows, columns, and cells, text recognition: extracting the text content in the cell in combination with the PaddleOCR model, and structural analysis: analyzing the row and column structure of the table and the relationship between merged cells.

4. A method for parsing and extracting medical guideline content using the system of any one of claims 1 to 3, comprising the following steps: S1: Data preprocessing, consisting of the following steps: S1-1: Data screening: Screen out files that cannot be processed and cannot process encrypted documents; S1-2: Language recognition; supports processing Chinese or English documents; S1-3: Document type classification: distinguish between scanned and text-based PDFs; S1-4: Content garbled code detection; S1-5: Page metadata extraction; S2: Document content analysis, consisting of the following specific steps: S2-1: Layout Analysis: Divide the document into regions based on the fine-tuned layout detection model, accurately locate the position of each element in the document by returning coordinates, and analyze the spatial relationship between elements; Reading order analysis: identify single-column and multi-column layouts, and handle multi-column layouts or complex typesetting; S2-2: Formula detection and recognition: The formula detection model is used to locate the formula area, and the formula recognition model is used to convert the formula image into LaTeX format; S2-3: Table recognition: TableMaster and StructEqTable perform table recognition tasks and can execute Table-to-LaTex tasks; S2-4: OCR recognition; S3: Document content post-processing, which consists of the following specific steps: S3-1: Processing bounding boxes: removing formulas and text blocks contained within image and table areas, as well as boxes contained within formula boxes; S3-2: Information integration; S4: Format conversion.

5. The method for parsing and extracting medical guideline content according to claim 4, wherein: S2-4: OCR recognition: After excluding tables, formulas, and images in the document, the OCR model is used to perform text area recognition; for text blocks containing inline formulas, the coordinates provided by the formula detection model are first used to mask the formula, then OCR is performed, and finally the formula is reinserted into the OCR result.

6. The method for parsing and extracting medical guideline content according to claim 4, wherein: S3-1: For partially overlapping text boxes, avoid mutual coverage by shrinking them vertically and horizontally. For partial overlap between text and tables and images, temporarily ignore the tables and images. S3-2: Based on the output of S, remove useless information in the header and footer, and divide the entire page into multiple regions. Each region contains multiple bounding boxes, and each region contains at most one column.

7. The method for parsing and extracting medical guideline content according to claim 4, wherein: S4: Format conversion, generating markdown or json format according to the results of S3-2 processing.

8. The method for parsing and extracting medical guideline content according to claim 4, wherein: The aforementioned S1-3: For text-based PDFs, PyMuPDF is used directly for text extraction. For scanned PDFs, an OCR model is required. S1-4: Content garbled code detection: Some text-based documents will have garbled code when copying text. Such documents should be identified in advance during the preprocessing stage. S1-5: Page metadata extraction: Extract the total number of pages and page size of the document.

9. The method for parsing and extracting medical guideline content according to claim 4, wherein: The S2-1 output format has two types: JSON format and visual annotation. The S2-1 layout analysis divides the document into regions based on the fine-tuned layout detection model, locates the position of the title, text, pictures, tables, formulas, footnotes, marginal notes, and page numbers, and analyzes the spatial relationship between elements, such as nesting, overlap, and contextual association. S2-1 Reading order analysis: Determine single column and multi-column based on the position information of the text block, determine the reading path based on the order of element arrangement, and handle multi-column layout or complex typesetting.

Images