Medical examination report detection and recognition method based on MTVDBNet
By using the MTVDBNet method, combined with feature enhancement and multi-scale feature fusion, the problems of low accuracy and poor robustness in text detection and recognition of medical examination report images are solved, achieving efficient and reliable text detection and recognition that is adaptable to multi-scale text features and complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUJIAN FUJITSU COMM SOFTWARE CO LTD
- Filing Date
- 2026-03-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies have poor anti-interference capabilities, low accuracy, and weak model adaptability when processing low-quality, high-noise medical examination report images. They are also unable to effectively capture multi-scale text features, resulting in poor text detection and recognition performance.
We adopt an MTVDBNet-based approach that combines feature enhancement, multi-scale feature fusion, and adaptive post-processing. By introducing an attention mechanism and a visual language pre-trained model, we optimize the text detection and recognition process, suppress noise interference, and enhance the model's ability to fuse multi-scale text features.
It improves the accuracy and adaptability of text detection, reduces noise interference, enhances the robustness of the model to complex scenarios, and achieves efficient automated processing of medical examination reports.
Smart Images
Figure CN122392082A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting and recognizing medical examination reports based on MTVDBNet. Background Technology
[0002] With the rapid development of medical informatization and smart healthcare, the demand for the digitization and automated processing of documents such as medical examination reports and outpatient medical records is becoming increasingly urgent. Traditionally, the extraction of text information from these medical images mainly relies on manual input or basic OCR technology, which is inefficient and prone to errors. Existing automated methods mostly use traditional text detection algorithms or general-purpose deep learning models, but they perform poorly when processing low-quality, high-noise medical images, with the following specific shortcomings:
[0003] (1) Poor anti-interference ability and low accuracy: Medical examination report images usually contain complex background noise, such as stains, scratches, handwriting, overlapping stamps, etc. Traditional text detection algorithms lack effective feature enhancement mechanisms. Under the interference of these noises, it is difficult to accurately distinguish between text regions and non-text regions, resulting in a significant decrease in the recall and precision of text localization.
[0004] (2) Weak model adaptability and insufficient generalization ability: Medical documents have diverse formats, and the text font, size, and orientation vary. Traditional methods or single-scale detection models are difficult to effectively capture multi-scale text features. When faced with reports from different hospitals and in different formats, the model's generalization ability is insufficient, and the detection results are unstable. Summary of the Invention
[0005] The purpose of this invention is to provide a medical examination report detection and recognition method based on MTVDBNet, to solve the problems of low accuracy and poor robustness in existing medical examination report image text detection and recognition technologies. By introducing the MTVDBNet method based on feature enhancement and multi-scale feature fusion, combined with a visual language pre-trained model, the following objectives are achieved:
[0006] 1. Improve the accuracy and adaptability of text detection and reduce noise interference.
[0007] 2. Enhance the model's ability to fuse multi-scale text features.
[0008] 3. Improve overall recognition efficiency by optimizing detection results through a learnable post-processing mechanism.
[0009] 4. Provides an efficient and reliable solution for the automated processing of medical examination reports.
[0010] The technical solution adopted in this invention is:
[0011] The method for detecting and recognizing medical examination reports based on MTVDBNet includes the following steps:
[0012] Obtain images of medical examination reports;
[0013] The medical examination report image is input into a pre-constructed MTVDBNet model for text detection and recognition;
[0014] The MTVDBNet model performs operations including feature enhancement, multi-scale feature fusion, and adaptive post-processing on the input image to detect and recognize text information in the image.
[0015] Output the structured text recognition results.
[0016] Furthermore, the MTVDBNet model includes:
[0017] An efficient feature enhancement module is used to enhance text region features of the input image and suppress background noise by introducing an attention mechanism;
[0018] A multi-scale feature fusion module, connected to the efficient feature enhancement module, is used to fuse feature maps from different scales to generate a segmentation feature map rich in contextual information.
[0019] An adaptive post-processing and recognition module, connected to the MSFM module, is used to convert the fused segmentation feature map into a text bounding box and perform text content recognition.
[0020] Furthermore, the efficient feature enhancement module highlights text regions and suppresses background noise by cascading multi-level convolutional features and introducing a spatial attention mechanism.
[0021] Furthermore, the multi-scale feature fusion module fuses feature maps from different depths and resolutions from the EFEM module to detect text of different sizes.
[0022] Furthermore, the adaptive post-processing and recognition module includes:
[0023] A learnable post-processing unit is used to adaptively determine the threshold between text and background for each pixel in the segmentation feature map through a differentiable binarization operation, generating an accurate binary map to separate the text region.
[0024] The text recognition unit is used to recognize the separated text regions and obtain the original text string.
[0025] Furthermore, the adaptive post-processing and recognition module also includes a visual language pre-trained model, which is used to fuse the features and semantic information corresponding to the binary image to enhance the model's inference ability when the text is partially occluded or blurred.
[0026] Furthermore, the visual language pre-trained model is the oCLIP model.
[0027] Furthermore, the step of outputting the structured text recognition result includes:
[0028] Based on preset medical keywords, the identified raw text strings are categorized into predefined structured fields;
[0029] The system uses a medical dictionary to automatically correct errors in the identified raw text strings.
[0030] Furthermore, the method for constructing the MTVDBNet model includes:
[0031] Collect diverse medical examination report images from different medical institutions to construct training and testing datasets;
[0032] The images in the dataset are labeled, and the bounding boxes of the text regions and the corresponding text content are marked.
[0033] Construct a basic feature library, which includes at least common noise patterns in medical documents and prior text information;
[0034] The MTVDBNet model is trained using the labeled dataset and the aforementioned basic feature library to optimize the model parameters.
[0035] Furthermore, the noise pattern includes one or more of stamps, handwriting, and stains; the prior text information includes specific terms and field layouts.
[0036] The main technical features of this invention include:
[0037] 1. Feature Enhancement and Multi-Scale Fusion: The efficient feature enhancement module (EFEM module) introduces spatial attention mechanism and cascade structure to achieve efficient feature enhancement; the multi-scale feature fusion module (MSFM module) fuses multi-scale features to improve the model's ability to detect text of different sizes.
[0038] 2. Learnable post-processing mechanism: Differentiable binarization method is adopted to adaptively set the threshold, reducing the error caused by the traditional fixed threshold.
[0039] 3. Visual Language Pre-trained Model Integration: The oCLIP model is combined with the segmentation module to enhance visual and semantic representations by leveraging its large-scale pre-trained knowledge, thereby improving adaptability to complex medical image scenarios.
[0040] 4. End-to-end text detection and recognition process: Seamlessly connects the detection and recognition stages to optimize overall processing efficiency.
[0041] Beneficial technical effects of the present invention:
[0042] 1. High accuracy and robustness: The EFEM and MSFM modules effectively suppress noise interference, significantly improving text detection accuracy in low-resolution medical images; oCLIP integration enhances the model's understanding of semantic content, reducing false positives and false negatives.
[0043] 2. High efficiency and adaptability: The EFEM module is designed to be lightweight, ensuring that the method is computationally efficient in practical applications; multi-scale fusion enables the model to adapt to various text sizes and layouts.
[0044] 3. Scalability: This method can be generalized to other medical document images (such as prescriptions and laboratory reports), providing a reliable foundation for medical informatization.
[0045] 4. High degree of automation: End-to-end processing reduces manual intervention, lowers medical data management costs, and increases processing speed. Attached Figure Description
[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;
[0047] Figure 1 This is an architecture diagram of the MTVDBNet model in this invention. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0049] This embodiment provides a method for detecting and recognizing medical examination reports based on MTVDBNet, including the following steps:
[0050] First, obtain images of the medical examination report. These images can come from various sources, such as scans from the hospital information system, electronic archives from the examination center, or photos of the paper report taken by medical staff using their mobile phone cameras.
[0051] Next, the acquired medical examination report image is input into a pre-built MTVDBNet model for text detection and recognition. This model is an end-to-end deep learning network specifically designed for the characteristics of medical document images.
[0052] Subsequently, the MTVDBNet model performs operations on the input image, including feature enhancement, multi-scale feature fusion, and adaptive post-processing, to sequentially locate the text region in the image and recognize the text content.
[0053] Finally, the structured text recognition results are output. The original recognized text strings are post-processed and categorized according to a preset medical data structure (such as patient name, examination items, and numerical results) to form standardized formats such as JSON or XML, which are easy to import directly into the hospital information system (HIS) or electronic medical record system.
[0054] In this embodiment, the MTVDBNet model deeply integrates feature enhancement, multi-scale fusion, and adaptive post-processing. Its core architecture includes three main modules, such as... Figure 1 As shown:
[0055] The Efficient Feature Enhancement Module (EFEM) enhances text region features in the input image and suppresses background noise by introducing an attention mechanism. This module uses cascaded multi-level convolutional features and introduces a spatial attention mechanism to highlight text regions, suppress background noise, and enhance the model's ability to capture key information.
[0056] The Multi-Scale Feature Fusion (MSFM) module, connected to the efficient feature enhancement module, is used to fuse feature maps from different scales to generate segmentation feature maps rich in contextual information. Specifically, the MSFM module fuses feature maps of different depths and resolutions to generate a unified segmentation feature map rich in contextual information, enabling the detection of text of different sizes and ensuring that text of different sizes (such as medical numerical indicators) can be effectively detected.
[0057] The adaptive post-processing and recognition module, connected to the multi-scale feature fusion module, transforms the fused segmented feature map into text bounding boxes and performs text content recognition. This module accurately separates text regions through learnable post-processing operations (such as differentiable binarization) and enhances its adaptability to complex scenes by combining a visual language pre-trained model. Finally, the text recognition unit completes the content recognition. Specifically, the adaptive post-processing and recognition module includes:
[0058] The learnable post-processing unit adaptively determines the thresholds for text and background for each pixel in the segmentation feature map through a differentiable binarization operation, generating an accurate binary map to precisely separate the text regions.
[0059] The text recognition unit is used to recognize the separated text region image to obtain the original text string.
[0060] The adaptive post-processing and recognition module also includes a visual language pre-trained model, which fuses the features corresponding to the binary image with semantic information, enabling the model to understand the semantic context of medical text and enhance its inference ability when the text is partially occluded or blurred. In this embodiment, the visual language pre-trained model is the oCLIP model, which utilizes its large-scale pre-trained knowledge to enhance visual and semantic representations and improve adaptability to complex medical image scenes.
[0061] The specific steps for outputting the structured text recognition results include:
[0062] Based on preset medical keywords, the identified raw text strings are categorized into predefined structured fields. For example, the identification results are categorized into predefined structured fields based on keywords such as "name," "blood pressure," and "reference range."
[0063] Automatic error correction is performed on the identified raw text string using a medical dictionary. This includes automatically correcting errors in the recognition results, such as using a medical dictionary to correct character recognition errors caused by image blur (e.g., misrecognizing "mg" as "rng").
[0064] The final output is a standardized data structure that is easy to import directly into a hospital information system (HIS).
[0065] The construction method of the MTVDBNet model includes the following steps:
[0066] Collect diverse medical examination report images from different medical institutions (such as scanned copies from hospital information systems, electronic archives from medical examination centers, or photos of paper reports taken by medical staff using their mobile phone cameras) to construct training and testing datasets.
[0067] The images in the dataset are labeled, and the bounding boxes of the text regions and the corresponding text content are marked.
[0068] Build a high-quality, standardized basic feature library. The feature library contains noise patterns commonly found in medical documents, such as stamps, handwriting, and stains, as well as textual prior information such as specific terms and field layouts.
[0069] The MTVDBNet model was trained using the labeled dataset and basic feature library to optimize the model parameters.
[0070] To ensure the generalization ability of the MTVDBNet model in real-world scenarios, system testing and performance evaluation are required after training. Testing is conducted using medical report images from new institutions that were not present in the training set. Evaluation dimensions include:
[0071] Detection performance: Calculate the recall, precision, and F1 score of the text detection;
[0072] Recognition performance: Calculate character accuracy and word accuracy;
[0073] Robustness: The performance retention of the test model under different noise levels, different lighting conditions and different image resolutions;
[0074] End-to-end efficiency: The average processing time for a single image.
[0075] Based on the feedback from the test evaluation, the model is iterated and optimized:
[0076] Based on feedback from the testing and evaluation, this is a crucial cycle of continuous improvement.
[0077] Optimize network structure: for example, adjust the implementation of the attention mechanism in EFEM or the fusion strategy of MSFM;
[0078] Fine-tuning model parameters: Use newly encountered counterexample images to fine-tune the model and enhance its generalization ability;
[0079] Update the feature library: Incorporate new types of noise and layouts into the basic feature library to continuously expand the model's adaptability.
[0080] Through multiple iterations, the entire system has been continuously optimized in terms of accuracy, robustness, and processing speed, ultimately achieving high-quality automated extraction of medical report information.
[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. In addition to the methods described in the above embodiments, the present invention can also be implemented in the following ways:
[0082] Modular cloud service deployment: The core MTVDBNet detection and text recognition are decoupled and encapsulated as independent microservices. These are provided externally via a RESTful API, allowing hospital information systems to call them on demand, achieving high-concurrency, elastically scalable distributed processing and reducing the coupling of a single system.
[0083] Lightweight integration for mobile devices: Model pruning and quantization were performed on the EFEM and MSFM modules to develop a lightweight version. This version can be integrated into medical apps, enabling healthcare professionals to quickly extract key information (such as patient name and abnormal indicators) by taking real-time photos of reports using their mobile phone cameras, meeting the needs of mobile scenarios such as bedside diagnosis.
[0084] Structured enhancement using a large language model: After text recognition, the original recognition results are input into a large language model fine-tuned with medical knowledge, replacing traditional keyword-based rules. Leveraging its powerful semantic understanding and reasoning capabilities, it automatically corrects OCR errors, completes abbreviations, and generates more accurate and relevant structured diagnostic suggestions, which are then directly entered into electronic medical records.
[0085] These implementation methods expand the application boundaries of the solution, providing more value in terms of deployment flexibility, mobility, and depth of intelligent decision-making.
[0086] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Unless otherwise specified, the embodiments and features described and illustrated in this application can be combined with each other. The components of the embodiments of this application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
Claims
1. A method for detecting and recognizing medical examination reports based on MTVDBNet, characterized in that: Includes the following steps: Obtain images of medical examination reports; The medical examination report image is input into a pre-constructed MTVDBNet model for text detection and recognition; The MTVDBNet model performs operations including feature enhancement, multi-scale feature fusion, and adaptive post-processing on the input image to detect and recognize text information in the image. Output the structured text recognition results.
2. The medical examination report detection and recognition method based on MTVDBNet according to claim 1, characterized in that: The MTVDBNet model include: An efficient feature enhancement module is used to enhance text region features of the input image and suppress background noise by introducing an attention mechanism; A multi-scale feature fusion module, connected to the efficient feature enhancement module, is used to fuse feature maps from different scales to generate a segmentation feature map rich in contextual information. An adaptive post-processing and recognition module, connected to the MSFM module, is used to convert the fused segmentation feature map into a text bounding box and perform text content recognition.
3. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 2, characterized in that: The efficient feature enhancement module highlights text regions and suppresses background noise by cascading multi-level convolutional features and introducing a spatial attention mechanism.
4. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 2, characterized in that: The multi-scale feature fusion module fuses feature maps of different depths and resolutions from the EFEM module to detect text of different sizes.
5. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 2, characterized in that: The adaptive post-processing and recognition module includes: A learnable post-processing unit is used to adaptively determine the threshold between text and background for each pixel in the segmentation feature map through a differentiable binarization operation, generating an accurate binary map to separate the text region. The text recognition unit is used to recognize the separated text regions and obtain the original text string.
6. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 5, characterized in that: The adaptive post-processing and recognition module also includes a visual language pre-trained model, which is used to fuse the features and semantic information corresponding to the binary image to enhance the model's inference ability when the text is partially occluded or blurred.
7. The medical examination report detection and recognition method based on MTVDBNet according to claim 6, characterized in that: The visual language pre-trained model is the oCLIP model.
8. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 1, characterized in that: The steps for outputting the structured text recognition results include: Based on preset medical keywords, the identified raw text strings are categorized into predefined structured fields; The system uses a medical dictionary to automatically correct errors in the identified raw text strings.
9. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 1, characterized in that: The method for constructing the MTVDBNet model includes: Collect diverse medical examination report images from different medical institutions to construct training and testing datasets; The images in the dataset are labeled, and the bounding boxes of the text regions and the corresponding text content are marked. Construct a basic feature library, which includes at least common noise patterns in medical documents and prior text information; The MTVDBNet model is trained using the labeled dataset and the aforementioned basic feature library to optimize the model parameters.
10. The method for detecting and recognizing medical examination reports based on MTVDBNet according to claim 9, characterized in that: The noise pattern includes one or more of stamps, handwriting, and stains; the prior text information includes specific terms and field layouts.