Processing method for medical report structured information extraction and privacy protection

Abstract

The invention discloses a processing method for structural information extraction and privacy protection of a medical report, which relates to the field of medical report processing and can efficiently and accurately extract text information in a table by integrating advanced image preprocessing, dynamic layout analysis, OCR (optical character recognition) technologies and a dual-channel privacy detection model. Establishing a coordinate mapping relation between the image description paragraph and the corresponding image; the generated JSON structure containing the text, table and image-text mapping relation ensures the integrity and accuracy of the data; the problems of data dislocation, context splitting, privacy disclosure and the like when the medical document with complex typesetting is processed in the prior art are solved; besides, an improved Transform framework is constructed on the basis of a Qwen model, a logical reasoning chain is injected, meanwhile, a case reading large model is obtained by integrating clinical diagnosis rules and multi-modal contrast learning strategy training, the model is utilized to generate a high-quality structured abstract, and the abstract generation accuracy is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of medical report processing, and in particular to a processing method for extracting structured information and protecting privacy in medical reports. Background Art

[0002] In the field of intelligent medical information processing, the combination of structured optical character recognition (OCR) and privacy-enhancing technologies has become an important means of enhancing the value of medical data. While existing solutions have made significant progress in digitizing reports, they still face several technical bottlenecks in actual clinical applications. First, current OCR systems typically use general algorithms for layout analysis, which struggle to cope with the unique multimodal structural characteristics of medical documents, including various data areas such as tables of test indicators, image description paragraphs, and handwritten annotations.

[0003] The complex layout features of medical reports, such as tables extending across pages and nested examination items, pose challenges to traditional OCR technology. While these systems can recognize standard text, they perform poorly when parsing tables with dynamic column widths or mixed text and graphics, with accuracy rates below 35%. This low accuracy leads to data misalignment and loss of context in the digitized results, seriously compromising the integrity of subsequent data analysis.

[0004] On the other hand, there's an inherent conflict between privacy protection mechanisms and medical information extraction. Current mainstream solutions rely on global desensitization or regular expression filtering to protect sensitive information. However, these methods have limited ability to identify implicit identifiers, such as the association between "test number, patient age, and diagnosis." Research has shown that desensitization methods that rely solely on keyword matching carry a 12.7% risk of implicit privacy leakage, while over-desensitization can result in a 19.3% loss of critical diagnostic information, limiting the secondary use value of medical data.

[0005] Existing systems face the challenge of deeply integrating domain knowledge when generating diagnostic summaries. Template-based methods exhibit errors as high as 28% in handling the dynamic range of test indicators and drug dosage units, and are unable to establish a medically logical connection between test values and diagnostic conclusions. Experiments have shown that when processing complex reports containing more than eight abnormal indicators, the summary completeness of traditional methods is less than 62%, which is insufficient for the high-precision multidisciplinary consultations required in tertiary hospitals.

[0006] Furthermore, compatibility issues with standardized medical data output have long plagued the industry's development. Most systems prefer XML output, which conflicts structurally with the JSON or YAML interface protocols of hospital HIS and PACS systems. Consequently, when integrating clinical systems, additional format conversion modules often need to be developed, increasing deployment costs by over 40% and hindering the practical application and promotion of the technology. In summary, the root cause of these issues lies in the failure of existing solutions to establish a collaborative processing framework guided by medical features. The OCR module lacks prior knowledge of medical document structure, the privacy protection layer lacks an effective diagnostic semantic association analysis mechanism, the summary generation process lacks support from a clinical diagnostic rule engine, and the data output ignores the heterogeneous nature of medical information systems. By establishing a comprehensive, multi-layered processing system, an effective balance between medical data integrity and security can be achieved, providing new impetus for advancement in the medical industry. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a processing method for extracting structured information from medical reports and protecting privacy, comprising:

[0008] Preprocessing the medical report image to obtain a first image, performing dynamic layout analysis on the first image based on a medical document structure feature library to parse a test index table, and using optical character recognition (OCR) technology to obtain text information in the table; obtaining a coordinate mapping relationship between the image description paragraph in the first image and the corresponding image to obtain an image-text mapping relationship; and generating a JSON structure containing text, a table, and an image-text mapping relationship based on the obtained and parsed data;

[0009] The dual-channel privacy detection model is used to identify explicit and implicit privacy information in the JSON structure, remove the explicit privacy information, and apply the differential privacy mechanism to perturb the implicit privacy information to blur the privacy information, thus obtaining the processed data.

[0010] Based on the Qwen model, an improved Transformer framework was constructed by combining a multi-head attention mechanism, a pointer generation network, and a cascade annotation strategy. A logical reasoning chain from indicator abnormalities to pathological inference was constructed using preset clinical diagnostic rules, and this logical reasoning chain was then injected into the improved Transformer framework.

[0011] The Transformer framework is trained using a multimodal contrastive learning strategy, a cross-modal alignment loss function, and a training set to obtain a large case reading model; the training set includes multiple categories of medical reports, i.e., training samples; a training sample includes one or more data modalities; the types of data modalities include text and images;

[0012] The case reading model extracts examination information from the processed data and generates a structured summary based on the examination information, including diagnostic conclusions, interpretation of abnormal indicators and clinical recommendations.

[0013] Furthermore, the medical document structure feature library includes a plurality of standard templates for different file categories; the medical report image is preprocessed to obtain a first image, and the first image is dynamically analyzed based on the medical document structure feature library to parse the test index table, specifically:

[0014] An adaptive median filter is used to eliminate noise in the medical report image, and a CLAHE algorithm is used to enhance the contrast of the text area in the medical report image to obtain a first image;

[0015] Determining the file category to which the first image belongs through a ResNet target model, loading layout feature parameters of a standard template corresponding to the file category through a template matching engine of a medical document structure feature library, and performing dynamic layout analysis on the first image based on the loaded layout feature parameters;

[0016] Dynamically adjusting the column width threshold of the table in the first image based on the results of the dynamic layout analysis and the structural characteristics of the table in the standard template corresponding to the file category;

[0017] Determining whether a table spans two pages based on the titles, identifiers, and page layout features (such as page numbers and continuity marks) of each table portion in the first image, and if so, concatenating the table portions spanning two pages to form a complete table;

[0018] The XY-Cut algorithm is used to segment the cross-page table or the non-cross-page table using the adjusted column width threshold to parse the inspection index table.

[0019] Furthermore, the coordinate mapping relationship between the image description paragraph and the corresponding image in the first image is obtained as follows:

[0020] Use the YOLOv5 model to detect annotation boxes in the image and extract the image features within each annotation box; use the natural language processing model to extract the feature vectors of text paragraphs and obtain text features that correspond one-to-one to the text paragraphs;

[0021] Calculate the similarity between each image feature and all text features and generate an attention weight matrix; Calculate the similarity between each text feature and all image features and generate an attention weight matrix;

[0022] Based on the above two attention weight matrices, new image feature representation and text feature representation are generated by weighted summing of image features and text features respectively;

[0023] Calculate the similarity between the new image feature representation and the new text feature representation to obtain the image-text semantic consistency score;

[0024] Determine whether the image-text semantic consistency score is higher than the set threshold. If so, establish a coordinate mapping relationship between each image description paragraph, i.e., text paragraph, and the corresponding image feature. If not, re-detect the annotation box and extract the text features.

[0025] Furthermore, determining the file category to which the first image belongs by using the ResNet target model specifically includes:

[0026] Obtain a data set containing multiple training samples; each training sample corresponds to a standard template in a medical document structure feature library; (these samples should cover all 12 types of standard templates) the training samples are marked with corresponding file categories;

[0027] The ResNet-18 model is used as the base model, and a fully connected layer is connected to its last layer to obtain the initial model. The initial model is trained using the dataset and the cross entropy loss function to obtain the ResNet target model.

[0028] The first image is input into the ResNet target model to obtain the probability values corresponding to various file categories contained in the medical document structure feature library output by the fully connected layer, and the file category with the highest probability value is selected as the classification result.

[0029] Furthermore, the dual-channel privacy detection model includes:

[0030] The explicit privacy channel is used to identify explicit privacy information in the JSON structure using a pre-trained BiLSTM-CRF model and a NER dictionary. It then verifies the ID card number in the explicit privacy information using regular expressions and the Luhn algorithm. After verification, the identified explicit privacy information is removed.

[0031] An implicit privacy channel is used to construct a semantic graph network based on information in a JSON structure; the semantic graph network includes multiple nodes, each node represents a clinical entity; the clinical entity is a test indicator, a diagnostic conclusion, or patient information; a connection relationship, i.e., an association weight edge, is established between nodes in the semantic graph network based on clinical guidelines and expert knowledge; multiple clinical entity combinations with privacy leakage risks are set; based on the clinical entity combinations, it is determined whether a three-hop node combination in the semantic graph network has a leakage risk. If so, a differential privacy mechanism is applied to perturb the implicit privacy information, i.e., the three-hop node combination with the leakage risk.

[0032] Furthermore, the formula expression of the cross-modal alignment loss function is:

[0033]

[0034] Where I represents the image feature, T represents the text description, τ represents the temperature coefficient; s(I,T) represents the similarity score between the image feature I and the text description T; K represents the number of negative samples; I k Represents the kth negative sample image feature; the negative sample indicates that the image feature I does not match its corresponding text description T; L align Represents the cross-modal alignment loss value.

[0035] Furthermore, the logic reasoning chain is injected into the improved Transformer framework, specifically:

[0036] Storing the logical reasoning chain in the form of a decision tree results in a clinical decision tree containing multiple diagnostic rules;

[0037] Define the weight of each diagnostic rule in the clinical decision tree. The definition formula is:

[0038] w r =σ(w g [h t ; r j ]);

[0039] Where w r represents the weight of the rth diagnostic rule; σ represents the Sigmoid function; w g represents the parameter matrix in the gating mechanism; h t represents the hidden state of the decoder, r j represents the embedded representation of the rth diagnostic rule;

[0040] The clinical decision trees with defined weights are dynamically fused through a gating mechanism and injected into the improved Transformer framework.

[0041] Furthermore, the case reading model extracts examination information from the processed data and generates a structured summary containing diagnostic conclusions, abnormal indicator interpretations, and clinical recommendations based on the examination information, specifically:

[0042] Extract inspection information from processed data using the Qwen model;

[0043] Locate inspection item blocks in the extracted inspection information through a cascade annotation strategy and extract indicator data from them; generate an input sequence using the located inspection items, extracted indicator data, and text information in the table, and input the sequence into a pointer generation network;

[0044] The attention weight of each element in the input sequence is calculated through a pointer generation network and a multi-head attention mechanism. The key elements are determined based on the attention weights, and the corresponding diagnostic rules in the clinical decision tree are selected based on the key elements. The attention weight of each element in the input sequence is adjusted based on the elements involved in the diagnostic rules and the indicator data corresponding to the elements, and a structured summary is generated that includes diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations.

[0045] Furthermore, the processing method further includes:

[0046] A privacy attack simulator is constructed through a generative adversarial network. The generated structured summary is simulated with the privacy attack simulator, and the privacy leakage risk in the structured summary is evaluated. Preprocessing steps are performed based on the evaluation results.

[0047] Furthermore, the pre-processing step comprises:

[0048] Perform data obfuscation or anonymization on the generated structured summaries;

[0049] Optimize the parameter settings of the pointer generation network.

[0050] Compared with the prior art, the present invention has at least the following beneficial effects:

[0051] (1) By integrating advanced image preprocessing, dynamic layout analysis, OCR technology, and a dual-channel privacy detection model, the present invention can efficiently and accurately extract text information in the test index table from the medical report image and establish a coordinate mapping relationship between the image description paragraph and the corresponding image; the generated JSON structure containing text, table, and image-text mapping relationship not only ensures the integrity and accuracy of the data; this method solves the problems of data dislocation, context fragmentation, and privacy leakage encountered by the existing technology when processing complex typeset medical documents, and greatly improves the quality and security of medical report digitization; in addition, based on the Qwen model, an improved Transformer framework is constructed by combining a multi-head attention mechanism, a pointer generation network, and a cascade annotation strategy, and a logical reasoning chain is injected. At the same time, a large case reading model is obtained by integrating clinical diagnostic rules and a multimodal comparative learning strategy. The model can also automatically generate high-quality structured summaries based on the extracted information, greatly improving the accuracy of summary generation;

[0052] (2) By preprocessing medical report images using an adaptive median filter and the CLAHE algorithm, and performing dynamic layout analysis after determining the file category using a ResNet target model, the present invention can more accurately identify and parse test index tables. This method is particularly suitable for medical documents containing complex typesetting features (such as tables spanning multiple pages, dynamic column widths, etc.), improving the accuracy of data extraction, reducing data dislocation and context fragmentation problems, and thus improving the quality of subsequent data analysis.

[0053] (3) This paper uses the YOLOv5 model to detect image annotation boxes and extract image features. It also uses a natural language processing model to extract text paragraph features, calculates similarity, and generates an attention weight matrix to establish a semantic consistency score between the image and text. This method effectively addresses the limitations of traditional OCR technology when processing mixed image and text areas. By accurately establishing the coordinate mapping relationship between the image description paragraph and the corresponding image, the integrity and accuracy of the information extracted are ensured, which helps to generate a more reliable structured summary.

[0054] (4) Determining the file category to which the first image belongs through the ResNet target model not only improves the accuracy of classification, but also dynamically adjusts the column width threshold of the table according to different standard templates. This method overcomes the problem that existing general layout analysis algorithms are difficult to adapt to the multimodal structural characteristics of medical documents, and significantly improves the accuracy of table parsing, especially for medical reports with complex layout features;

[0055] (5) In this invention, the dual-channel privacy detection model combines explicit privacy channels and implicit privacy channels. It can not only identify and eliminate explicit privacy information (such as ID card numbers), but also discover implicit privacy leakage risks by constructing a semantic graph network and apply differential privacy mechanisms for perturbation processing. This method effectively solves the problems of over-desensitization or privacy leakage risks existing in existing privacy protection mechanisms, protects patient privacy, and retains key diagnostic information, greatly enhancing the security and availability of medical data.

[0056] (6) In the present invention, the application of the cross-modal alignment loss function optimizes the similarity score between image features and text descriptions. Through this optimization, data from different modalities (such as images and corresponding text descriptions) can be matched more accurately. This not only improves the consistency and reliability of the data, but also provides a solid foundation for subsequent data processing and analysis, further improving the performance and practicality of the entire system.

[0057] (7) In this invention, the logical reasoning chain is stored in the form of a decision tree, and clinical decision tree rules are dynamically integrated through a gating mechanism. This method can select appropriate diagnostic rules based on the attention weights of the elements in the input sequence, thereby generating a high-quality structured summary. In this way, the system can not only improve the accuracy of diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations, but also better meet the needs of clinical diagnosis and treatment, and improve the quality and efficiency of medical services. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The figure is a flow chart of a processing method for extracting structured information and protecting privacy from medical reports in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0060] In order to enhance the quality of structured summaries and meet the dual clinical demands for accuracy and safety, such as Figure 1 As shown, an embodiment of the present invention proposes a processing method for extracting structured information and protecting privacy from medical reports, including:

[0061] Preprocess the medical report image to obtain a first image, perform dynamic layout analysis on the first image based on a medical document structure feature library (including 12 standard templates such as test orders and imaging reports) to parse out a test index table, and use optical character recognition (OCR) technology to obtain text information in the table; obtain the coordinate mapping relationship between the image description paragraph in the first image and the corresponding image to obtain an image-text mapping relationship; and generate a JSON structure containing text, table, and image-text mapping relationship based on the obtained and parsed data;

[0062] In this embodiment, during the structured OCR processing, the method of the present invention also uses a conditional random field (CRF) model constrained by a medical dictionary for post-processing to improve recognition accuracy and semantic integrity. Specifically, text information is first extracted through OCR technology. In order to ensure that the extracted information is accurate and medically meaningful, a CRF model trained with a medical dictionary is used to correct the OCR results. The model specifically sets a forced state transition rule for the test indicators, which means that when a specific test indicator is identified, the model will force the recognition result to match the most likely correct value based on the pre-defined state transition logic. For example, if the standard range of a certain test indicator is 3.0-5.0, and the result recognized by OCR exceeds this range, the CRF model will automatically correct the value based on the knowledge in the medical dictionary to make it consistent with the expected range. This method significantly reduces the impact of character-level recognition errors on semantic integrity, thereby improving the reliability and accuracy of the overall data, especially in terms of key medical test indicators. This step ensures that even in the presence of character-level errors, the final extracted information still maintains a high degree of medical accuracy and consistency.

[0063] The medical document structure feature library includes a plurality of standard templates for different file categories; the medical report image is preprocessed to obtain a first image, and the first image is dynamically analyzed based on the medical document structure feature library to parse the test index table, specifically:

[0064] An adaptive median filter is used to eliminate noise in the medical report image, and a CLAHE algorithm is used to enhance the contrast of the text area in the medical report image to obtain a first image;

[0065] Determining the file category to which the first image belongs through a ResNet target model, loading layout feature parameters of a standard template corresponding to the file category through a template matching engine of a medical document structure feature library, and performing dynamic layout analysis on the first image based on the loaded layout feature parameters;

[0066] Determining the file category to which the first image belongs by using the ResNet target model specifically includes:

[0067] Obtain a data set containing multiple training samples; each training sample corresponds to a standard template in a medical document structure feature library; (these samples should cover all 12 types of standard templates) the training samples are marked with corresponding file categories;

[0068] The ResNet-18 model is used as the base model, and a fully connected layer is connected to its last layer to obtain the initial model. The initial model is trained using the dataset and the cross entropy loss function to obtain the ResNet target model.

[0069] The first image is input into the ResNet target model to obtain the probability values corresponding to various file categories contained in the medical document structure feature library output by the fully connected layer, and the file category with the highest probability value is selected as the classification result.

[0070] Dynamically adjusting the column width threshold of the table in the first image based on the results of the dynamic layout analysis and the structural characteristics of the table in the standard template corresponding to the file category;

[0071] Determining whether a table spans two pages based on the titles, identifiers, and page layout features (such as page numbers and continuity marks) of each table portion in the first image, and if so, concatenating the table portions spanning two pages to form a complete table;

[0072] The XY-Cut algorithm is used to segment the cross-page table or the non-cross-page table using the adjusted column width threshold to parse the inspection index table.

[0073] The coordinate mapping relationship between the image description paragraph and the corresponding image in the first image is obtained as follows:

[0074] Use the YOLOv5 model to detect annotation boxes in the image and extract the image features within each annotation box; use the natural language processing model to extract the feature vectors of text paragraphs and obtain text features that correspond one-to-one to the text paragraphs;

[0075] Calculate the similarity between each image feature and all text features and generate an attention weight matrix; Calculate the similarity between each text feature and all image features and generate an attention weight matrix;

[0076] Based on the above two attention weight matrices, new image feature representation and text feature representation are generated by weighted summing of image features and text features respectively;

[0077] Calculate the similarity between the new image feature representation and the new text feature representation to obtain the image-text semantic consistency score;

[0078] Determine whether the image-text semantic consistency score is higher than the set threshold. If so, establish a coordinate mapping relationship between each image description paragraph, i.e., text paragraph, and the corresponding image feature. If not, re-detect the annotation box and extract the text features.

[0079] The dual-channel privacy detection model is used to identify explicit and implicit privacy information in the JSON structure, remove the explicit privacy information, and apply the differential privacy mechanism to perturb the implicit privacy information to blur the privacy information, thus obtaining the processed data.

[0080] The dual-channel privacy detection model includes:

[0081] The explicit privacy channel is used to identify explicit privacy information in JSON structures using a pre-trained BiLSTM-CRF model and a NER dictionary (a specialized medical NER dictionary). Specifically, it implements 38 entity recognition categories. It then verifies the ID card number in the explicit privacy information using regular expressions and the Luhn algorithm. After verification, the identified explicit privacy information is removed.

[0082] An implicit privacy channel is used to construct a semantic graph network DSG based on the information in the JSON structure; the semantic graph network includes multiple nodes, each node represents a clinical entity; the clinical entity is a test indicator, a diagnostic conclusion or patient information; a connection relationship, i.e., an association weight edge, is established between the nodes in the semantic graph network based on clinical guidelines and expert knowledge; multiple clinical entity combinations with privacy leakage risks are set (for example, "test number + abnormal indicator + age" is a clinical entity combination); based on the clinical entity combination, it is judged whether the three-hop node combination in the semantic graph network has a leakage risk. If so, the differential privacy mechanism is applied to perturb the implicit privacy information, i.e., the three-hop node combination with the leakage risk.

[0083] In this embodiment:

[0084] A three-hop node combination refers to a path between any two nodes in the graph that passes through at most three edges. For example, if certain test indicators of "Patient A" are directly or indirectly associated with a specific diagnosis, then even if these test indicators or diagnosis are anonymous when viewed individually, their combination may reveal the identity of a specific individual.

[0085] Suppose in a medical report:

[0086] Node A: "Age = 35 years old";

[0087] Node B: "Gender = Male";

[0088] Node C: "A specific test indicator is significantly abnormal";

[0089] Each of these three nodes alone is insufficient to identify a specific patient, but when considered together, they may be able to infer information about a specific individual because this particular combination is relatively rare in the population. Therefore, when such a three-hop node combination (for example, from node A to node B to node C) is determined to be a clinical entity combination with a privacy risk, an alarm will be triggered, indicating that privacy protection processing is required for this combination, such as adding noise or anonymization.

[0090] The specific perturbation processing is as follows: numerical indicators are perturbated using Laplace noise of ε=0.5, and text fields are anonymized with k=10 to ensure data privacy and security.

[0091] Based on the Qwen model, an improved Transformer framework was constructed by combining a multi-head attention mechanism, a pointer generation network, and a cascade annotation strategy. A logical reasoning chain from abnormal indicator values to pathological inference was constructed using preset clinical diagnostic rules (specifically 327 preset clinical diagnostic rules, such as the association between the AST / ALT ratio and liver disease), and this logical reasoning chain was injected into the improved Transformer framework.

[0092] The logical reasoning chain is injected into the improved Transformer framework, specifically:

[0093] Storing the logical reasoning chain in the form of a decision tree results in a clinical decision tree containing multiple diagnostic rules;

[0094] Define the weight of each diagnostic rule in the clinical decision tree. The definition formula is:

[0095] w r =σ(w g [h t ; r j ]);

[0096] Where w r represents the weight of the rth diagnostic rule; σ represents the Sigmoid function; w g represents the parameter matrix in the gating mechanism; h t represents the hidden state of the decoder, r j represents the embedded representation of the rth diagnostic rule;

[0097] The clinical decision trees with defined weights are dynamically fused through a gating mechanism and injected into the improved Transformer framework.

[0098] The Transformer framework is trained using a multimodal contrastive learning strategy, a cross-modal alignment loss function, and a training set to obtain a large case reading model; the training set includes multiple categories of medical reports, i.e., training samples; a training sample includes one or more data modalities; the types of data modalities include text and images;

[0099] The formula expression of the cross-modal alignment loss function is:

[0100]

[0101] Where I represents the image feature, T represents the text description, τ represents the temperature coefficient; s(I,T) represents the similarity score between the image feature I and the text description T; K represents the number of negative samples; I kRepresents the kth negative sample image feature; the negative sample indicates that the image feature I does not match its corresponding text description T; L align Represents the cross-modal alignment loss value.

[0102] The case reading model extracts examination information from the processed data and generates a structured summary based on the examination information, including diagnostic conclusions, interpretation of abnormal indicators and clinical recommendations.

[0103] The case reading model extracts examination information from the processed data and generates a structured summary based on the examination information, including diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations, specifically:

[0104] Extract inspection information from processed data using the Qwen model;

[0105] The inspection item blocks are located in the extracted inspection information through a cascade annotation strategy, and the indicator data is extracted from them. An input sequence is generated through the located inspection items, the extracted indicator data, and the text information in the table, and the sequence is input into the pointer generation network;

[0106] The attention weight of each element in the input sequence is calculated through a pointer generation network and a multi-head attention mechanism. The key elements are determined based on the attention weights, and the corresponding diagnostic rules in the clinical decision tree are selected based on the key elements. The attention weight of each element in the input sequence is adjusted based on the elements involved in the diagnostic rules and the indicator data corresponding to the elements, and a structured summary is generated that includes diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations.

[0107] Specifically:

[0108] Determining key elements based on attention weights can be done in two ways:

[0109] 1. Select the top N elements with the highest weights: You can select the top N elements with the highest attention weights as key elements. The number N here can be adjusted according to the specific application scenario.

[0110] 2. Set a threshold: You can also set a threshold, and only those elements whose attention weight exceeds the threshold will be selected as key elements.

[0111] Once the key elements are identified, they can be used to guide the selection of appropriate diagnostic rules from the clinical decision tree. The specific operation can be:

[0112] Direct mapping: If a key element has a direct correspondence with a specific diagnostic rule, the corresponding rule can be directly selected.

[0113] Fuzzy matching: If there is no direct correspondence, the diagnostic rule corresponding to the closest key element can be found through similarity matching and other methods.

[0114] Consider an input sequence of medical reports. After passing through the Pointer Generation Network, attention weights are assigned to each element. Following a pre-defined strategy (e.g., selecting the top five high-weighted elements), several key elements are identified, such as "abnormal white blood cell count" and "fever symptoms persisting for one week." Based on these key elements, the system then selects relevant diagnostic rules from the clinical decision tree, such as the infectious disease diagnostic pathway for abnormal white blood cell count and prolonged fever.

[0115] The processing method further comprises:

[0116] A privacy attack simulator is constructed through a generative adversarial network. The generated structured summary is simulated with the privacy attack simulator, and the privacy leakage risk in the structured summary is evaluated. Preprocessing steps are performed based on the evaluation results.

[0117] The pre-processing step comprises:

[0118] Perform data obfuscation or anonymization on the generated structured summaries;

[0119] Optimize the parameter settings of the pointer generation network.

[0120] The processing method further comprises:

[0121] The generated structured summary is dynamically adapted to different data exchange formats such as JSON, YAML, or HL7 through a schema converter. This allows the structured summary to be seamlessly connected to the hospital's information systems, including HIS, PACS, etc.

[0122] The present invention adopts the Qwen model as the basis, combines the multi-head attention mechanism, the pointer generation network and the cascade annotation strategy to construct an improved Transformer framework, and uses the preset clinical diagnosis rules to build a logical reasoning chain, thereby solving the problem of low accuracy of the existing medical OCR system when processing complex typesetting (such as dynamic column width tables and mixed text and image areas). At the same time, the present invention uses a dual-channel privacy detection model to identify and protect explicit and implicit privacy information, effectively overcoming the risks of over-desensitization or privacy leakage in traditional privacy protection methods. In addition, the large case reading model obtained by training with a multimodal contrast learning strategy and a cross-modal alignment loss function not only improves the accuracy from test indicators to pathological inference, but also enhances the quality of structured summaries, meeting the clinical dual needs for accuracy and safety. This comprehensive solution greatly enhances the value of medical data, promotes the efficiency and reliability of digital processing of medical reports, and provides strong technical support for achieving higher quality medical services.

[0123] Example 2

[0124] To fully validate the effectiveness of the proposed method, this example designed a series of experiments to evaluate its performance in multiple aspects. Key evaluation indicators include medical entity recognition accuracy, table reconstruction quality, image-text semantic association capability, privacy protection effectiveness, diagnostic summary generation quality, and clinical applicability. Based on these evaluation indicators, two core datasets were prepared to comprehensively cover and test the various functions of the proposed method. The verification process included:

[0125] 1. Dataset preparation for the improved Transformer framework

[0126] The dataset preparation covers two core data sources: medical report structured dataset and cross-modal medical understanding dataset. The details are as follows:

[0127] The Medical Report Structured Dataset integrates 12,850 real clinical reports publicly available online, covering digital documents from seven departments, including radiology and laboratory medicine. Each report is annotated by experts and includes detailed medical entity labels, which are used to verify entity recognition and table reconstruction performance.

[0128] Cross-modal Medical Understanding Dataset: Contains 4,200 image-report pairs, covering data from multiple modalities such as CT and MRI, and focuses on evaluating the cross-modal understanding ability of semantic associations between images and text.

[0129] 2. Parameters and experimental settings

[0130] The experiments of the present invention were run on 4 NVIDIA RTX 4090GPUs equipped with the deep learning framework PyTorch 1.11.0, and the compilation environment was Python 3.10. In this embodiment, the basis of the case reading large model is the Qwen model (specifically the Qwen-VL2-7B model). After fine-tuning, two versions of the case reading large model are obtained, which are respectively recorded as: Our Method-7B and Our Method-13B. On this basis, this embodiment introduces a reinforcement learning fine-tuning framework to realize a dynamic policy optimization mechanism. In the pre-training stage, the case reading large model integrates multi-source medical corpus and enhances the ability to understand knowledge in the medical field. Subsequently, the proximal policy optimization (PPO) algorithm is used for targeted fine-tuning, and the model performance is optimized by designing a two-layer reward function:

[0131] Basic Reward R b :The Jaccard similarity calculation model is used to predict the overlap rate between entities and labeled entities to evaluate the coverage of medical entities (entities represent patient information, indicator data, etc.) generated in the summary.

[0132] Strategy reward R p : Quantify the degree of compliance with diagnostic rules based on rule activation frequency and confidence weighted calculation.

[0133] Experimental results show that this reinforcement learning framework significantly improves the performance of the model, increasing the clinical applicability score of structured summaries by 27.6% and the recall rate of abnormal indicators to 93.8%, which is significantly better than traditional supervised fine-tuning methods.

[0134] Comparative experiments further validated the effectiveness of this approach, demonstrating its superior performance in extracting case information and preserving privacy. In particular, in structured report and multimodal understanding tasks, the "Our Method" model demonstrated higher accuracy and robustness compared to other baseline models (such as the Llama2 and Alpaca series).

[0135] 3. Comparative Experimental Design

[0136] Table 1 shows the classification results of our method-7B and method-13B models, along with three baseline models, on a structured medical report dataset and a cross-modal medical understanding dataset. The results show that our method model outperformed the other baseline models across all experimental conditions. In particular, our method-13B achieved a human-assessed accuracy of 72.6% on the structured medical report dataset and 73.3% on the multi-modal medical understanding dataset. This demonstrates that our method offers superior accuracy and robustness in handling case information and multimodal understanding.

[0137] Table 1:

[0138]

[0139] In Table 1:

[0140] Structured Report (GT): This column represents the degree to which each model matches the ground truth labels when generating structured reports (i.e., structured summaries). Higher values indicate that the reports generated by the model are closer to the ground truth labels.

[0141] Structured Report (Human): The data values in this column represent the scores of human experts who evaluated the structured reports generated by each model. Higher scores indicate that the reports generated by the model more closely meet the expectations and standards of human experts.

[0142] Multimodal Understanding (GT): This column reflects how well each model matches the true labels in the multimodal understanding task. Higher values indicate better performance in understanding and processing the combined image and text information.

[0143] Multimodal Understanding (Human): This column represents the scores of human experts evaluating each model's performance on the multimodal understanding task. Higher scores indicate a model's performance on the task more closely matches human cognition and judgment.

[0144] 4. Doctor's evaluation results

[0145] As shown in Table 2, our method shows significant advantages in the four core dimensions of speed, medical accuracy, privacy compliance, and diagnostic relevance. In particular, in terms of speed, the 13B model far surpasses all baseline models with a score of 53.1 (the highest baseline is 35.5 points), thanks to the pipeline optimization design of the five-level collaborative architecture. In terms of medical accuracy, the 13B version of our method achieved an expert review pass rate of 94.7%, an increase of 30.9 percentage points over the best baseline model (72.3% of Alpaca-13B). In addition, in terms of privacy compliance, our method achieved a "completely shielded" rating, while all baseline models had privacy leakage risks, mainly due to the dual-channel design of the hierarchical privacy protection architecture - the explicit privacy channel uses the BiLSTM-CRF model to realize 38 categories of sensitive entity recognition, and the implicit channel detects cross-entity association risks through the semantic graph network. In terms of diagnostic relevance score, version 13B of the method of the present invention scored 9.5 points (out of 10 points), which is 31.9% higher than the best baseline model (7.2 points of Llama2-13B).

[0146] It should be noted that the five-level collaborative architecture specifically refers to:

[0147] 1. Medical image preprocessing layer:

[0148] The medical report image is preprocessed (using adaptive median filter and CLAHE algorithm) to obtain the first image. This step aims to eliminate noise and enhance the contrast of the text area to prepare for subsequent information extraction and analysis.

[0149] 2. Structured information extraction layer:

[0150] Based on the medical document structure feature library, the first image is dynamically analyzed to parse out the test index table, and the text information in the table is obtained using OCR technology.

[0151] The coordinate mapping relationship between the image description paragraph and the corresponding image in the first image is obtained to generate an image-text mapping relationship.

[0152] Based on the parsed data above, a JSON structure containing text, tables, and image-text mapping relationships is generated to achieve structured representation of information.

[0153] 3. Image and text semantic association layer:

[0154] The YOLOv5 model is used to detect annotation boxes in the image and extract image features within each annotation box; at the same time, the natural language processing model is used to extract feature vectors of text paragraphs.

[0155] Calculate the similarity between image features and text features, generate an attention weight matrix, and generate new image feature representation and text feature representation through weighted summation.

[0156] Calculate the similarity score between the newly generated image feature representation and the text feature representation, establish a coordinate mapping relationship between each image description paragraph and the corresponding image feature, and improve the ability to understand cross-modal data.

[0157] 4. Privacy protection layer:

[0158] A dual-channel privacy detection model is used to identify explicit and implicit privacy information in JSON structures. The explicit privacy channel uses a BiLSTM-CRF model to identify and remove explicit privacy information, and uses regular expressions and the Luhn algorithm to verify sensitive information such as ID numbers.

[0159] Implicit privacy channel: A semantic graph network is constructed based on the information in the JSON structure, and a differential privacy mechanism is applied to perturb three-hop node combinations that are at risk of leakage, ensuring that patient privacy is effectively protected.

[0160] 5. Diagnostic summary generation and optimization layer:

[0161] A large case reading model was obtained by adopting a multimodal contrastive learning strategy, a cross-modal alignment loss function, and an improved Transformer framework for training with training sets.

[0162] The model extracts examination information from the processed data and generates a structured summary based on the extracted information, including diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations.

[0163] A two-layer reward function (base reward and strategy reward) is designed to evaluate the quality of the generated summary and the degree to which it follows clinical diagnosis rules, thereby improving the accuracy and applicability of the diagnostic summary.

[0164] Table 2 shows the doctor evaluation results of the proposed solution in actual medical scenarios:

[0165] Language Model speed Medical accuracy Privacy compliance Diagnostic relevance Llama2-7B-withdata 25.3 66.3％ Unshielded 6.3 Llama2-13B-withdata 27.1 53.5％ Unshielded 7.2 Alpaca-7B-withdata 30.2 68.5％ Unshielded 6.6 Alpaca-13B-withdata 35.5 72.3％ Unshielded 6.3 Our Method-7B 45.8 90.0％ Completely shielded 8.8 Our Method-13B 53.1 94.7％ Completely shielded 9.5

[0166] In Table 2:

[0167] 1. Speed: The values in this column (e.g., 25.3, 27.1, etc.) represent the speed at which each language model processes tasks, expressed in time. Smaller values indicate faster processing times and higher efficiency.

[0168] 2. Medical Accuracy: The percentage values in this column (e.g., 66.3%, 90.0%, etc.) indicate the medical accuracy of the results generated by each language model. Higher values indicate a greater degree of alignment between the generated content and medical standards and professional knowledge, and thus a higher degree of accuracy.

[0169] 3. Privacy Compliance: The descriptions in this column (e.g., "Unmasked" or "Completely Masked") reflect how well each language model protects patient privacy when processing data. "Unmasked" indicates that the model does not effectively mask sensitive information, while "Completely Masked" indicates that the model effectively identifies and masks all sensitive information, ensuring privacy.

[0170] 4. Diagnostic Relevance: The values in this column (e.g., 6.3, 8.8, etc.) indicate the degree of relevance between the content generated by each language model and clinical diagnosis. The higher the value, the more the model-generated content reflects and assists clinical diagnosis, and thus has greater practical value.

[0171] In summary, the method based on the five-level collaborative processing architecture proposed in the present invention realizes the intelligent upgrade of the entire process of medical report processing through the case reading large model and reinforcement learning fine-tuning framework. This method integrates medical field knowledge enhancement, dynamic rule gating mechanism and hierarchical privacy protection technology, and breaks through the limitations of traditional methods in key links such as medical image preprocessing and diagnostic summary generation. Experimental results show that compared with baseline models such as Llama and Alpaca, the present invention has improved the medical accuracy and diagnostic relevance dimensions by 30.9% and 31.9% respectively, and achieved a 100% privacy compliance rating. In particular, it has shown significant advantages in cross-modal data association and abnormal indicator recall, successfully solving the core pain points in medical report processing such as disconnection between images and text, privacy leakage and diagnostic logic fragmentation, and providing an innovative solution for efficient, accurate and secure report processing in smart medical scenarios.

[0172] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0173] In addition, in the present invention, descriptions such as "first," "second," and "one" are for descriptive purposes only and should not be understood to indicate or imply their relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0174] In the present invention, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0175] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A method for extracting structured information from medical reports and protecting privacy, characterized in that: include: Preprocessing the medical report image to obtain a first image, performing dynamic layout analysis on the first image based on a medical document structure feature library to parse a test index table, and using optical character recognition (OCR) technology to obtain text information in the table; obtaining a coordinate mapping relationship between the image description paragraph in the first image and the corresponding image to obtain an image-text mapping relationship; and generating a JSON structure containing text, a table, and an image-text mapping relationship based on the obtained and parsed data; The dual-channel privacy detection model is used to identify explicit and implicit privacy information in the JSON structure, remove the explicit privacy information, and apply the differential privacy mechanism to perturb the implicit privacy information to blur the privacy information, thus obtaining the processed data. Based on the Qwen model, an improved Transformer framework is constructed by combining the multi-head attention mechanism, the pointer generation network, and the cascade annotation strategy; Build a logical reasoning chain from indicator abnormalities to pathological inference through preset clinical diagnostic rules, and inject the logical reasoning chain into the improved Transformer framework; The Transformer framework is trained using a multimodal contrastive learning strategy, a cross-modal alignment loss function, and a training set to obtain a large case reading model; the training set includes multiple categories of medical reports, i.e., training samples; a training sample includes one or more data modalities; the types of data modalities include text and images; The case reading model extracts examination information from the processed data and generates a structured summary based on the examination information, including diagnostic conclusions, interpretation of abnormal indicators and clinical recommendations.

2. A method for extracting structured information from medical reports and protecting privacy according to claim 1, characterized in that: The medical document structure feature library includes a plurality of standard templates for different file categories; the medical report image is preprocessed to obtain a first image, and the first image is dynamically analyzed based on the medical document structure feature library to parse the test index table, specifically: An adaptive median filter is used to eliminate noise in the medical report image, and a CLAHE algorithm is used to enhance the contrast of the text area in the medical report image to obtain a first image; Determining the file category to which the first image belongs through a ResNet target model, loading layout feature parameters of a standard template corresponding to the file category through a template matching engine of a medical document structure feature library, and performing dynamic layout analysis on the first image based on the loaded layout feature parameters; Dynamically adjusting the column width threshold of the table in the first image based on the results of the dynamic layout analysis and the structural characteristics of the table in the standard template corresponding to the file category; Determining whether there is a table that spans two pages based on the titles, identifiers, and page layout features of each table portion in the first image, and if so, splicing the table portions that span two pages to form a complete table; The XY-Cut algorithm is used to segment the cross-page table or the non-cross-page table using the adjusted column width threshold to parse the inspection index table.

3. A method for extracting structured information from medical reports and protecting privacy according to claim 2, characterized in that: The coordinate mapping relationship between the image description paragraph and the corresponding image in the first image is obtained as follows: Use the YOLOv5 model to detect annotation boxes in the image and extract the image features within each annotation box; use the natural language processing model to extract the feature vectors of text paragraphs and obtain text features that correspond one-to-one to the text paragraphs; Calculate the similarity between each image feature and all text features and generate an attention weight matrix; Calculate the similarity between each text feature and all image features and generate an attention weight matrix; Based on the above two attention weight matrices, new image feature representation and text feature representation are generated by weighted summing of image features and text features respectively; Calculate the similarity between the new image feature representation and the new text feature representation to obtain the image-text semantic consistency score; Determine whether the image-text semantic consistency score is higher than the set threshold. If so, establish a coordinate mapping relationship between each image description paragraph, i.e., text paragraph, and the corresponding image feature. If not, re-detect the annotation box and extract the text features.

4. A method for extracting structured information from medical reports and protecting privacy according to claim 2, characterized in that: Determining the file category to which the first image belongs by using the ResNet target model specifically includes: Acquire a data set containing multiple training samples; each training sample corresponds to a standard template in a medical document structure feature library; the training samples are marked with corresponding file categories; The ResNet-18 model is used as the base model, and a fully connected layer is connected to its last layer to obtain the initial model. The initial model is trained using the dataset and the cross entropy loss function to obtain the ResNet target model. The first image is input into the ResNet target model to obtain the probability values corresponding to various file categories contained in the medical document structure feature library output by the fully connected layer, and the file category with the highest probability value is selected as the classification result.

5. A method for extracting structured information from medical reports and protecting privacy according to claim 1, characterized in that: The dual-channel privacy detection model includes: The explicit privacy channel is used to identify explicit privacy information in the JSON structure using a pre-trained BiLSTM-CRF model and a NER dictionary. It then verifies the ID card number in the explicit privacy information using regular expressions and the Luhn algorithm. After verification, the identified explicit privacy information is removed. An implicit privacy channel is used to construct a semantic graph network based on information in a JSON structure; the semantic graph network includes multiple nodes, each node represents a clinical entity; the clinical entity is a test indicator, a diagnostic conclusion, or patient information; a connection relationship, i.e., an association weight edge, is established between nodes in the semantic graph network based on clinical guidelines and expert knowledge; multiple clinical entity combinations with privacy leakage risks are set; based on the clinical entity combinations, it is determined whether a three-hop node combination in the semantic graph network has a leakage risk. If so, a differential privacy mechanism is applied to perturb the implicit privacy information, i.e., the three-hop node combination with the leakage risk.

6. A method for extracting structured information from medical reports and protecting privacy according to claim 1, characterized in that: The formula expression of the cross-modal alignment loss function is: Where I represents the image feature, T represents the text description, τ represents the temperature coefficient; s(I,T) represents the similarity score between the image feature I and the text description T; K represents the number of negative samples; I k Represents the kth negative sample image feature; the negative sample indicates that the image feature I does not match its corresponding text description T; L align Represents the cross-modal alignment loss value.

7. A method for extracting structured information from medical reports and protecting privacy according to claim 6, characterized in that: The logical reasoning chain is injected into the improved Transformer framework, specifically: Storing the logical reasoning chain in the form of a decision tree results in a clinical decision tree containing multiple diagnostic rules; Define the weight of each diagnostic rule in the clinical decision tree. The definition formula is: w r =σ(w g [h t ;r j ]); Where w r represents the weight of the rth diagnostic rule; σ represents the Sigmoid function; w g represents the parameter matrix in the gating mechanism; h t represents the hidden state of the decoder, r j represents the embedded representation of the rth diagnostic rule; The clinical decision trees with defined weights are dynamically fused through a gating mechanism and injected into the improved Transformer framework.

8. A method for extracting structured information from medical reports and protecting privacy according to claim 7, characterized in that: The case reading model extracts examination information from the processed data and generates a structured summary based on the examination information, including diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations, specifically: Extract inspection information from processed data using the Qwen model; Locate inspection item blocks in the extracted inspection information through a cascade annotation strategy and extract indicator data from them; generate an input sequence using the located inspection items, extracted indicator data, and text information in the table, and input the sequence into a pointer generation network; The attention weight of each element in the input sequence is calculated through a pointer generation network and a multi-head attention mechanism. The key elements are determined based on the attention weights, and the corresponding diagnostic rules in the clinical decision tree are selected based on the key elements. The attention weight of each element in the input sequence is adjusted based on the elements involved in the diagnostic rules and the indicator data corresponding to the elements, and a structured summary is generated that includes diagnostic conclusions, interpretation of abnormal indicators, and clinical recommendations.

9. A method for extracting structured information from medical reports and protecting privacy according to claim 8, characterized in that: The processing method further comprises: A privacy attack simulator is constructed through a generative adversarial network. The generated structured summary is simulated with the privacy attack simulator, and the privacy leakage risk in the structured summary is evaluated. Preprocessing steps are performed based on the evaluation results.

10. A method for extracting structured information and protecting privacy from medical reports according to claim 9, characterized in that: The pre-processing step comprises: Perform data obfuscation or anonymization on the generated structured summaries; Optimize the parameter settings of the pointer generation network.

Images