Artificial Intelligence-Based Method and System for Extracting Medical Image Report Information
By desensitizing, segmenting, keyword matching, and deep learning processing of medical imaging reports, enhanced structured reports are generated, solving the problems of inconsistent report quality and difficulty in structuring data in existing technologies, and achieving automated processing and data interconnection.
Patent Information
- Application Number
- CN202611294180.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-17
AI Technical Summary
Current medical image report processing methods rely on human experience, resulting in inconsistent report quality, omission of key information, and difficulty in achieving structured alignment between image data and reports, which affects the level of intelligence in clinical auxiliary diagnosis and the value of data reuse.
By acquiring medical imaging reports and imaging data, and after de-identifying them, we use text segmentation, keyword matching, the BioBERT model, and deep learning methods to extract structured information. Combined with multimodal feature fusion and consistency verification, we generate an enhanced structured report.
It has automated and improved the accuracy of report processing, reduced the burden on physicians, provided computable structured data assets, supported multi-center studies and intelligent diagnosis, and promoted the interconnection of medical data.
Smart Images

Figure CN123120178A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent technology, specifically to a method and system for extracting medical image report information based on artificial intelligence. Background Technology
[0002] Artificial intelligence-based methods for extracting information from medical image reports play an irreplaceable and crucial role in today's society. Their core value lies in automatically transforming massive amounts of unstructured free text reports into computable and mineable structured medical data assets. This fundamentally solves the structural contradictions caused by radiologists being overloaded with reading images, such as inconsistent report quality and increased risk of missing key information. It transforms medical image data from "dormant archives" into a truly active resource supporting clinical decision-making and scientific research.
[0003] In existing methods, the processing of medical image reports has long relied on human experience, and the generation of report text and the interpretation of image data are disconnected in the process. Essentially, it is a "separate" single-modal processing approach, which leads to a serious deficiency in the lack of objective verification between the subjective descriptions in the report and the actual quantitative characteristics of the image. When key information such as lesion size and location in the report does not match the image itself due to physician fatigue, negligence, or experience differences, downstream medical decisions are still based on this unverified report, thus propagating the error to the entire diagnosis and treatment process. Moreover, these unstructured report data, due to the lack of structured alignment with image features, are difficult to be automatically interpreted and deeply utilized by computer systems, which seriously restricts the level of intelligence in clinical auxiliary diagnosis and the value of data reuse. Summary of the Invention
[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for extracting medical image report information based on artificial intelligence, the method comprising: Step S1: Obtain medical imaging report data and medical imaging data for the same case; perform desensitization processing on the medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data; Step S2: Based on standardized medical image report data, use text segmentation technology and medical keyword matching technology to obtain structured medical image report data; Step S3: Based on the structured medical image report data, use the BioBERT model and information extraction network to obtain the structured medical image report information; Step S4: Based on standardized medical image data, use a deep learning-based medical image feature extraction method to obtain medical image visual feature data; Step S5: Based on the structured medical image report information and the visual feature vector of the medical image, use a multimodal feature fusion method to obtain fused medical feature data; Step S6: Based on the fusion medical feature data, use the consistency verification method based on image visual feature matching to obtain the consistency verification results; Step S7: Based on the structured medical image report information, fused medical feature data, and consistency verification results, use the field-based report generation method to generate an enhanced structured medical image report.
[0005] Furthermore, the acquisition of medical imaging report data and medical imaging data for the same case includes: By interacting with the existing PACS / RIS / HIS systems of partner medical institutions, we can obtain medical imaging data and medical imaging report data generated after patients complete medical imaging examinations. The PACS system acquires DICOM format medical image data generated during the patient's medical imaging examination; the RIS system acquires medical image report data generated based on the medical image data.
[0006] Furthermore, the process of de-identifying the medical image report data and medical image data to obtain standardized medical image report data and standardized medical image data includes: Sensitive information in medical image report data is identified and replaced, and sensitive label fields in DICOM format medical image data are cleared or emptyed to form standardized medical image report data and standardized medical image data.
[0007] Furthermore, the process of obtaining structured medical image report data based on standardized medical image report data, using text segmentation technology and medical keyword matching technology, includes: Standardized medical imaging report data is segmented into examination information area, image description area, and diagnostic conclusion area. For different regions, information is extracted using a pre-established medical imaging keyword database, including examination site, imaging features, disease description, and diagnostic opinions. Using medical terminology standardization technology, the extracted examination sites, imaging findings, disease descriptions, and diagnostic opinions are standardized and converted into unified medical terminology. The data is stored according to preset data fields to form structured medical imaging report data, including the examination site, imaging features, disease description, and diagnostic opinions.
[0008] Furthermore, the process of obtaining structured medical image report information based on structured medical image report data using the BioBERT model and information extraction network includes: The examination site information, imaging findings information, disease description information, and diagnostic opinion information are combined to generate a unified input text; The unified input text is fed into the BioBERT model, where the text encoding layer of the model performs word segmentation to obtain word segments; and the attention mechanism in the semantic encoding layer of the model converts each word segment into a corresponding medical text feature vector. The medical text feature vector corresponding to each word is input into the information extraction network, and the medical text feature vector is labeled with medical information category by the information extraction network; The probability value of each word belonging to different medical information categories is calculated based on the matching relationship between the feature vector of medical text and each medical information category. The highest probability value is taken as the medical information category corresponding to the word. Each word segment is associated with its corresponding medical information category and its position in the same input text. Word segments that belong to the same medical information category and appear consecutively are merged to obtain the complete medical entity information for each medical information category. Structured medical image report information is obtained by using complete medical entity information for each medical information category.
[0009] Furthermore, the step of obtaining medical image visual feature data using a deep learning-based medical image feature extraction method based on standardized medical image data includes: The standardized medical image data is preprocessed to generate standardized image tensor data; Standardized image tensor data are input into a pre-trained medical image feature extraction network to obtain multi-scale image feature maps. The pre-trained medical image feature extraction network employs an attention-based mechanism to weight and enhance the extracted multi-scale image feature maps. The weighted and enhanced multi-scale image feature maps are then subjected to feature dimensionality reduction through a global average pooling layer and a fully connected layer to generate fixed-dimensional medical image visual feature vectors.
[0010] Furthermore, the step of obtaining fused medical feature data using a multimodal feature fusion method based on structured medical image report information and medical image visual feature vectors includes: Key medical information in structured medical image reports is converted into a sequence of text feature vectors using a pre-trained medical text embedding model. The visual feature vectors of medical images are transformed in dimension through a linear mapping layer so that they have the same feature dimension as the text feature vector sequence. A cross-modal attention fusion mechanism is adopted, in which the text feature vector sequence is used as the query vector and the medical image visual feature vector is used as the key vector and value vector. The cross-modal attention weight between the text features and the visual features is calculated to obtain text-guided image enhancement features. Simultaneously, the medical image visual feature vector is used as the query vector, and the text feature vector sequence is used as the key vector and value vector. The cross-modal attention weight between the visual features and the text features is calculated to obtain the image-guided text enhancement features. The text-guided image enhancement features are concatenated and fused with the image-guided text enhancement features, and then integrated through a fully connected layer to generate fused medical feature data.
[0011] Furthermore, the step of obtaining consistency verification results using a consistency verification method based on image visual feature matching based on fused medical feature data includes: Extract image visual quantitative feature data from the fused medical feature data; The image visual quantitative feature data is compared and verified with the image performance information and diagnostic opinion information in the structured medical image report information, and the feature matching degree between the image visual quantitative feature data and the corresponding lesion description parameters in the image performance information is calculated. Based on the relationship between the feature matching degree and the preset threshold, a consistency verification result is generated.
[0012] Furthermore, based on the structured medical image report information, fused medical feature data, and consistency verification results, an enhanced structured medical image report is generated using a field-based report generation method, including: The basic report content is based on the examination site information, imaging manifestation information, disease description information, and diagnostic opinion information in the structured medical imaging report. The key information of the visual features of the images in the fused medical feature data is converted into objective descriptive statements and added to the image performance information after the basic report content; Based on the consistency verification results, the basic report content is annotated; all the above information is then spliced together in the order of examination site, imaging manifestations, disease description, diagnosis opinion, supplementary description, and verification annotation to generate an enhanced structured medical imaging report.
[0013] An artificial intelligence-based medical image report information extraction system, the system comprising: Data preprocessing module: acquires medical imaging report data and medical imaging data of the same case; performs desensitization processing on the medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data; Report structuring module: Based on standardized medical image report data, it uses text segmentation technology and medical keyword matching technology to obtain structured medical image report data; Key information extraction module: Based on structured medical image report data, the BioBERT model and information extraction network are used to obtain structured medical image report information; Image feature extraction module: Based on standardized medical image data, it uses a deep learning-based medical image feature extraction method to obtain visual feature data of medical images; Cross-modal fusion module: Based on structured medical image report information and medical image visual feature vectors, a multimodal feature fusion method is used to obtain fused medical feature data; Consistency verification module: Based on the fused medical feature data, a consistency verification method based on image visual feature matching is used to obtain the consistency verification results; Report generation module: Based on structured medical image report information, fused medical feature data, and consistency verification results, an enhanced structured medical image report is generated using a field-based report generation method.
[0014] This invention provides a method and system for extracting medical image report information based on artificial intelligence. It has the following beneficial effects: Through an end-to-end intelligent processing mechanism, tasks such as reading massive amounts of reports, transcribing information, and quality checking, which previously relied on manual labor, are now automated by the model. This significantly shortens report processing time and effectively alleviates the efficiency bottleneck caused by radiologists being overwhelmed with reading images. In terms of accuracy, the introduction of deep learning models enables refined extraction of text entities. Multimodal feature fusion and bidirectional cross-validation mechanisms allow for cross-verification between image visual features and report text descriptions, automatically identifying potential discrepancies in key information such as lesion size and location between images and reports, providing objective and accurate auxiliary references for clinical practice. Regarding data value, by transforming unstructured free text reports into structured data containing standardized fields, massive historical image reports are transformed from "dormant archives" into computable and mineable structured medical data assets. This provides a high-quality data foundation for multi-center clinical research, early disease screening, and model training for intelligent auxiliary diagnostic systems. Simultaneously, the standardized data format enables interconnection of image report data between different medical institutions, providing practical technical support for the implementation of hierarchical medical services and the balanced allocation of high-quality medical resources. In summary, this method is not only an intelligent tool to improve the efficiency of radiology departments, but also a core technology engine to promote the transformation of medical imaging from experience-driven to data-driven, and from isolated to collaborative and shared processes. Attached Figure Description
[0015] Figure 1This is a flowchart illustrating the steps of the artificial intelligence-based medical image report information extraction method of the present invention; Figure 2 This is an architecture diagram of the AI-based medical image report information extraction system of the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] like Figure 1 As shown, an artificial intelligence-based method for extracting medical image report information includes the following steps: Step S1: Obtain medical imaging report data and medical imaging data for the same case; perform desensitization processing on the medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data; Step S2: Based on standardized medical image report data, use text segmentation technology and medical keyword matching technology to obtain structured medical image report data; Step S3: Based on the structured medical image report data, use the BioBERT model and information extraction network to obtain the structured medical image report information; Step S4: Based on standardized medical image data, use a deep learning-based medical image feature extraction method to obtain medical image visual feature data; Step S5: Based on the structured medical image report information and the visual feature vector of the medical image, use a multimodal feature fusion method to obtain fused medical feature data; Step S6: Based on the fusion medical feature data, use the consistency verification method based on image visual feature matching to obtain the consistency verification results; Step S7: Based on the structured medical image report information, fused medical feature data, and consistency verification results, use the field-based report generation method to generate an enhanced structured medical image report.
[0018] Obtain medical imaging report data and medical imaging data for the same case, including: Based on the data authorization agreement signed with the cooperating medical institutions, and on the premise of fully protecting patient privacy and data security, we will obtain real clinical case data generated by the cooperating medical institutions during the clinical diagnosis and treatment process; specifically, we will obtain medical imaging data and medical imaging report data generated after patients complete medical imaging examinations by interacting with the existing PACS / RIS / HIS systems of the cooperating medical institutions. Specifically, the PACS system acquires DICOM format medical image data generated during patients' medical imaging examinations such as CT, MRI, and X-ray; the RIS system acquires medical image report data generated based on the medical image data, including structured image reports, image description text, and medical text data such as diagnostic conclusions.
[0019] De-identification processing was performed on medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data, including: Identify and replace sensitive information in medical image report data, and clear or empty sensitive label fields in DICOM format medical image data; For medical imaging report data, sensitive personal information such as name, ID number, contact information, and hospital number are identified using pattern matching-based recognition rules, and all identified sensitive personal information is replaced with general placeholders. For DICOM format medical image data, by parsing the label fields containing sensitive personal information such as patient name label, patient ID label, date of birth label, and contact information label in the DICOM file header, the contents of the label fields are deleted or replaced with empty values, while keeping the image pixel data unchanged; After the desensitization process is completed, the data fields related to medical analysis, such as case number, examination type, imaging manifestations, diagnosis conclusion, and imaging characteristics, are retained unchanged to form standardized medical imaging report data and standardized medical imaging data.
[0020] Based on standardized medical image report data, structured medical image report data is obtained using text segmentation and medical keyword matching techniques, including: Based on the fixed writing structure characteristics in standardized medical imaging report data, the standardized medical imaging report data is segmented. By identifying report field keywords such as "examination findings", "imaging manifestations", and "diagnostic opinions", the standardized medical imaging report data is divided into examination information area, image description area and diagnostic conclusion area. For text content in different regions, information is extracted using a pre-established medical imaging keyword database; wherein, the medical imaging keyword database includes a keyword database for examination sites, a keyword database for imaging signs, and a keyword database for disease names; For the examination information area, the examination site information is extracted by performing precise string matching between the keywords in the examination information area of the standardized medical imaging report data and the examination site keyword sub-library; for example, when keywords such as "chest CT examination" and "lung CT plain scan" are detected, the corresponding examination site is extracted as the lung. For the image description area, medical image manifestation information is extracted by performing precise string matching between the image description content in the image description area of the standardized medical image report data and the image sign keyword sub-library. For example, when keywords such as "nodular shadow", "ground-glass shadow", and "pleural effusion" are detected, the corresponding content is extracted as image manifestation information. For the diagnostic conclusion area, the keywords in the standardized medical imaging report data are precisely matched with the disease name keyword sub-database, and the judgment words in the diagnostic statement in the diagnostic conclusion area are analyzed to extract the disease description and diagnostic opinion information. For example, when a description such as "possible pneumonia" is detected, the corresponding disease name and diagnostic tendency are extracted as diagnostic opinion information by identifying diagnostic judgment words such as "possible", "suggest", and "possible". Using medical terminology standardization technology, the extracted examination sites, imaging manifestations, disease descriptions, and diagnostic opinions are standardized, and the medical expressions used by different doctors are converted into unified medical terms; for example, different expressions such as "ground-glass opacity" and "ground-glass appearance" are uniformly converted into "ground-glass opacity". Using text structuring technology, the extracted examination sites, imaging findings, disease descriptions, and diagnostic opinions are stored according to preset data fields to form structured medical imaging report data that includes examination sites, imaging findings, disease descriptions, and diagnostic opinions.
[0021] Based on the fixed writing structure of standardized medical imaging report data, the data is segmented using a segmentation algorithm based on header line detection. First, the standardized medical imaging report data is divided into several lines of text according to line breaks. Keyword matching is then performed on each line. When a line contains report field keywords such as "examination findings," "imaging manifestations," or "diagnostic opinion," it is marked as a segment boundary line. Starting from the marked segment boundary line, subsequent lines of text are assigned to the corresponding area until the next segment boundary line is encountered or the text ends. Through this segmentation process, the standardized medical imaging report data is divided into an examination information area, an image description area, and a diagnostic conclusion area. Specifically, information is extracted from the text content of different regions using a pre-established medical imaging keyword database. This database includes keywords for examination sites, imaging signs, and disease names. The examination site keyword database is constructed based on international standards for examination terminology and commonly used Chinese clinical examination site names. The imaging sign keyword database is constructed based on a radiology vocabulary database and commonly used sign terms in clinical imaging diagnostic reports. The disease name keyword database is constructed based on the International Classification of Diseases (ICD) and commonly used Chinese clinical disease names. Specifically, the medical terminology standardization technology employs a standardization method based on a synonym mapping table: a medical terminology synonym mapping table is pre-constructed, where each row contains a standard medical term and its multiple corresponding synonym variants; for the medical expression to be standardized, string exact matching and prefix matching searches are performed in the synonym mapping table; if a corresponding standard medical term is matched, the expression to be standardized is replaced with that standard term; if no standard term is matched, the original expression is retained and marked as unstandardized.
[0022] Based on structured medical image report data, the BioBERT model and information extraction network were used to obtain structured medical image report information, including: The examination site information, imaging findings information, disease description information, and diagnostic opinion information are combined to generate a unified input text; The unified input text is fed into the BioBERT model, and the unified input text is segmented into words by the text encoding layer in the model (using tokens for word segmentation). And each word segment is converted into a corresponding medical text feature vector through the attention mechanism in the semantic encoding layer of the model; The medical text feature vector corresponding to each word is input into the information extraction network. The information extraction network then labels the medical text feature vector with medical information categories (including examination site labels, lesion location labels, lesion size labels, imaging sign labels, disease name labels, and diagnostic opinion labels, which are set according to the engineering experience method). After receiving the medical text feature vector corresponding to each word, the information extraction network calculates the matching relationship between the medical text feature vector and each medical information category according to the classification parameters corresponding to different medical information categories. This yields the probability value of each word belonging to different medical information categories, and the highest probability value is taken as the medical information category corresponding to that word. For example, for the word "right lung", it is converted into a medical text feature vector and input into the information extraction network. The probability values are 0.95 for lesion location label, 0.02 for disease name label, 0.01 for lesion size label, and 0.02 for imaging sign label. Therefore, the medical information category corresponding to this word is lesion location label. Each word segment, its corresponding medical information category, and its position in the same input text are combined. Word segments that belong to the same medical information category and appear consecutively (such as appearing twice or more consecutively) are merged to obtain the complete medical entity information for each medical information category. By constructing key medical information from complete medical entity information for each medical information category, structured medical image report information is obtained.
[0023] The information extraction network model includes a feature mapping layer, a category prediction layer, and an output layer. The feature mapping layer receives the medical text feature vectors output by BioBERT and performs feature transformation and semantic enhancement on these vectors. The category prediction layer calculates the matching relationship between the input medical text feature vectors and different medical information categories based on preset classification parameters corresponding to medical information category labels, obtaining the probability value corresponding to each medical information category. The output layer determines the medical information category label corresponding to each word segment based on the maximum probability principle and outputs the corresponding medical entity category information. The information extraction network model can employ fully connected neural networks, Transformer networks, lightweight language model classification networks, or other deep learning models capable of medical entity classification.
[0024] The probability value is calculated as follows: the category prediction layer of the information extraction network maps the input medical text feature vector into a real number vector with a dimension equal to the total number of medical information categories through a linear fully connected layer. Each element in the real number vector represents the unnormalized score of the corresponding category. Then, the real number vector is converted into a probability distribution through the Softmax function, so that the sum of the probability values of all categories is 1.
[0025] Based on standardized medical image data, a deep learning-based medical image feature extraction method is used to obtain visual feature data of medical images, including: The standardized medical image data is preprocessed, including window width and window level adjustment, normalization, and image enhancement operations, to generate standardized image tensor data with uniform size and grayscale range; The preprocessed standardized image tensor data is input into a pre-trained medical image feature extraction network. The network then extracts features from the medical images layer by layer through its multi-layer convolutional structure, resulting in multi-scale image feature maps. The feature extraction network employs a backbone network based on a three-dimensional convolutional neural network (3D-CNN), comprising multiple convolutional layers, pooling layers, and activation function layers. This network is used to extract shallow visual features (such as edges, textures, and grayscale distribution) and deep semantic features (such as organ morphology, lesion areas, and abnormal density shadows) from the medical images. The pre-trained medical image feature extraction network uses an attention-based mechanism to weight and enhance the extracted multi-scale image feature maps, highlighting the feature responses of lesion areas and suppressing the interference responses of background areas. The weighted and enhanced multi-scale image feature maps are then subjected to feature dimensionality reduction through a global average pooling layer and a fully connected layer to generate fixed-dimensional medical image visual feature vectors.
[0026] Specifically, the window width and window level adjustment involves: for CT images, reading the window width and window level metadata from the DICOM file header, and mapping the original CT value range to a grayscale range of 0 to 255; the normalization involves: dividing the adjusted pixel value by 255 and mapping it to a floating-point range of 0 to 1; the image enhancement involves: using a histogram equalization method to enhance image contrast; the size unification involves: resampling all images to a uniform spatial size using bilinear interpolation; and the standardized image tensor data is a four-dimensional tensor with a shape of (depth × height × width × number of channels). The pre-trained medical image feature extraction network employs a channel attention-based mechanism. During training, it automatically learns the weight parameters of each channel's features, weighting and enhancing important channel features in the extracted multi-scale image feature map. This results in higher weights for feature channels that contribute significantly to lesion identification, indirectly highlighting the feature response of the lesion region and suppressing interference responses from the background region. Specifically, the channel attention-based mechanism involves: compressing the multi-scale image feature map output from each 3D convolutional block into a scalar value using global average pooling to generate a channel description vector; passing this channel description vector through two fully connected layers for nonlinear transformation. The first fully connected layer has 1 / 16 the number of input channels and uses the ReLU activation function; the second fully connected layer restores the number of input channels and uses the Sigmoid activation function, outputting the attention weight for each channel; and multiplying these attention weights element-wise with each channel of the original multi-scale image feature map to obtain the weighted and enhanced multi-scale image feature map. Based on structured medical image report information and medical image visual feature vectors, a multimodal feature fusion method is used to obtain fused medical feature data, including: Key medical information in structured medical image reports is converted into a sequence of text feature vectors using a pre-trained medical text embedding model. The medical text embedding model uses the same semantic encoding space as the BioBERT model to ensure that text features and visual features are aligned in the same feature space. The visual feature vectors of medical images are transformed in dimension through a linear mapping layer so that they have the same feature dimension as the text feature vector sequence. A cross-modal attention fusion mechanism is adopted, in which the text feature vector sequence is used as the query vector and the medical image visual feature vector is used as the key vector and value vector. The cross-modal attention weight between the text features and the visual features is calculated to obtain text-guided image enhancement features. Simultaneously, the medical image visual feature vector is used as the query vector, and the text feature vector sequence is used as the key vector and value vector. The cross-modal attention weight between the visual features and the text features is calculated to obtain the image-guided text enhancement features. The text-guided image enhancement features are concatenated and fused with the image-guided text enhancement features, and then integrated through a fully connected layer to generate fused medical feature data.
[0027] Specifically, the linear mapping layer is a fully connected layer whose input dimension is equal to the dimension of the medical image visual feature vector and whose output dimension is equal to the dimension of each text feature vector in the text feature vector sequence. The cross-modal attention fusion mechanism is an existing technology and will not be described in detail here. Based on the fusion of medical feature data, a consistency verification method based on image visual feature matching is used to obtain consistency verification results, including: Image visual quantitative feature data is extracted from the fused medical feature data. The image visual quantitative feature data includes lesion size measurement value, lesion density or signal value, lesion morphology parameters and lesion area location coordinate information. The image visual quantitative feature data is compared and verified with the image performance information and diagnostic opinion information in the structured medical image report information, and the feature matching degree between the image visual quantitative feature data and the corresponding lesion description parameters in the image performance information is calculated. Based on the relationship between the feature matching degree and the preset threshold, a consistency verification result is generated. For example, when the feature matching degree is greater than or equal to the preset threshold, the medical image data is determined to be consistent with the medical image report information, and a consistency verification pass identifier is generated. When the feature matching degree is less than the preset threshold, the medical image data is determined to be different from the medical image report information, and a difference prompt message and a corresponding difference description are generated. The difference description includes specific feature items in which there is a deviation between the image visual quantitative feature data and the image performance information. Simultaneously, the image visual quantitative feature data is converted into standardized image objective description statements as supplementary medical inference information.
[0028] The specific steps for extracting image visual quantitative feature data from the fused medical feature data are as follows: The fused medical feature data includes text enhancement features and visual enhancement features. The visual enhancement features are extracted from the fused medical feature data and used as image visual feature representation vectors. These image visual feature representation vectors are input into four parallel task-specific regression branches, each consisting of an independent regression network. The four task-specific regression branches include a size regression branch, a density regression branch, a morphology regression branch, and a location regression branch. The size regression branch outputs lesion size measurements, the density regression branch outputs lesion density or signal values, and the morphology regression branch outputs lesion morphology parameters. The location regression branch is used to output the location coordinate information of the lesion area; each task-specific regression branch contains two fully connected layers. The first fully connected layer has 128 neurons and uses the ReLU activation function, while the second fully connected layer has 1 neuron and uses the linear activation function to output the corresponding single quantized value. The four quantized values output by the four task-specific regression branches are encapsulated according to preset field names to form image visual quantization feature data in structured key-value pair format. The four task-specific regression branches are obtained through multi-task joint training during the model training phase, and the training loss function is the sum of the mean squared error losses of the four regression branches.
[0029] The specific steps for comparing and verifying the image visual quantitative feature data with the image manifestation information and diagnostic opinion information in the structured medical image report, and calculating the feature matching degree between the image visual quantitative feature data and the corresponding lesion description parameters in the image manifestation information, are as follows: First, lesion description parameters are extracted from the image manifestation information in the structured medical image report. These lesion description parameters correspond one-to-one with four quantification parameters in the image visual quantification feature data, including lesion size description parameters, lesion density description parameters, lesion morphology description parameters, and lesion location description parameters. The extraction employs a rule-based matching numerical parameter extraction method: for the lesion size description parameter, regular expressions are used to match the size values in the image manifestation information, and the matched values are used as the lesion size description parameter; for the lesion density description parameter, regular expressions are used to match the CT density values in the image manifestation information. The density value is used as a parameter to describe the lesion density. For the lesion morphology parameter, a pre-built morphological keyword lexicon is used for matching. For the lesion location parameter, a pre-built anatomical location keyword lexicon is used for matching. The anatomical location keyword lexicon includes anatomical location descriptive words such as "right upper lobe of the lung," "left lower lobe of the lung," and "right lobe of the liver." The matched location descriptive words are used as parameters to describe the lesion location. When the same parameter matches multiple values or descriptive words, the first matching result is selected according to their order of appearance in the text. If a parameter does not match any value or descriptive word, the parameter is marked as missing. Then, the four quantification parameters in the image visual quantification feature data are compared and verified item by item with the four extracted lesion description parameters; for numerical parameter pairs, the absolute error is calculated, and whether the parameter is consistent is determined based on whether the absolute error exceeds the preset clinically acceptable error range; for categorical parameter pairs, whether the categories are consistent is determined, and if they are consistent, they are marked as a match, and if they are inconsistent, they are marked as a mismatch. Finally, the feature matching degree is calculated based on the item-by-item comparison and verification results: First, when all four parameter pairs are complete and identical, the feature matching degree is 1. Second, when there are missing or inconsistent parameters, a weighted summation method is used to calculate the feature matching degree. The calculation formula is: the comprehensive feature matching degree is equal to the sum of the size matching degree multiplied by the size weight coefficient, the density matching degree multiplied by the density weight coefficient, the morphology matching degree multiplied by the morphology weight coefficient, and the position matching degree multiplied by the position weight coefficient. The following parameters are used to calculate the size matching degree: Size matching degree equals 1 minus the absolute value of the difference between the quantized size value and the size description value, divided by the size description value. If the result is less than 0, it is set to 0. The following parameters are used to calculate the density matching degree: Density matching degree equals 1 minus the absolute value of the difference between the quantized density value and the density description value, divided by the density description value. If the result is less than 0, it is set to 0. The following parameters are used to calculate the shape matching degree: 1 when the quantized shape value and the shape description value are the same, and 0 when they are different. The following parameters are used to calculate the position matching degree: 1 when the quantized position value and the position description value are the same, and 0 when they are different. The weighting coefficients are as follows: size 0.35, density 0.25, shape 0.20, and position 0.20. When a certain quantitative parameter is missing in the image visual quantitative feature data or a certain parameter is missing in the lesion description parameters, the matching degree of that parameter is 0. The feature matching degree ranges from 0 to 1.
[0030] Based on structured medical image report information, fused medical feature data, and consistency verification results, an enhanced structured medical image report is generated using a field-based report generation method, including: The basic report content is based on the examination site information, imaging manifestation information, disease description information, and diagnostic opinion information in the structured medical imaging report. The key information of the visual features of the images in the fused medical feature data is converted into objective descriptive statements and added to the image performance information after the basic report content; Based on the consistency verification results, the basic report content is annotated: if the consistency verification result is passed, a confirmation annotation is added after the diagnostic opinion information; if the consistency verification result shows a difference, a description of the difference is added to the end of the basic report content. All the above information is pieced together in the order of examination site, imaging manifestations, disease description, diagnosis, supplementary description, and verification annotation to generate an enhanced structured medical imaging report. The enhanced structured medical image report will be exported to the PACS / RIS / HIS system of the cooperating medical institution for clinicians to view and review.
[0031] The specific steps of using a text generation method based on description templates to convert key visual feature information of images into objective descriptive statements are as follows: Key visual information of the images is extracted from the fused medical feature data. The key visual information of the images includes the lesion size measurement value, lesion density or signal value, lesion morphology parameters, and lesion area location coordinate information. The four extracted parameters are directly concatenated into an objective descriptive statement according to the fixed sentence pattern of "lesion location parameter, size approximates size parameter, morphology parameter, density approximates density parameter". The descriptive statement is then input into a pre-trained large language model, which automatically generates the objective descriptive statement.
[0032] like Figure 2 As shown, an artificial intelligence-based medical image report information extraction system includes: Data preprocessing module: acquires medical imaging report data and medical imaging data of the same case; performs desensitization processing on the medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data; Report structuring module: Based on standardized medical image report data, it uses text segmentation technology and medical keyword matching technology to obtain structured medical image report data; Key information extraction module: Based on structured medical image report data, the BioBERT model and information extraction network are used to obtain structured medical image report information; Image feature extraction module: Based on standardized medical image data, it uses a deep learning-based medical image feature extraction method to obtain visual feature data of medical images; Cross-modal fusion module: Based on structured medical image report information and medical image visual feature vectors, a multimodal feature fusion method is used to obtain fused medical feature data; Consistency verification module: Based on the fused medical feature data, a consistency verification method based on image visual feature matching is used to obtain the consistency verification results; Report generation module: Based on structured medical image report information, fused medical feature data, and consistency verification results, an enhanced structured medical image report is generated using a field-based report generation method.
[0033] In this embodiment, an end-to-end intelligent processing mechanism automatically completes tasks such as reading massive amounts of reports, transcribing information, and quality checking that previously relied on manual labor. This significantly shortens report processing time and effectively alleviates the efficiency bottleneck caused by radiologists being overloaded with image reading. Regarding accuracy, a deep learning model is introduced to achieve refined extraction of text entities. Multimodal feature fusion and bidirectional cross-validation mechanisms enable mutual verification between image visual features and report text descriptions. This automatically identifies potential discrepancies between images and reports regarding key information such as lesion size and location, providing objective and accurate auxiliary references for clinical practice. In terms of data value, by transforming unstructured free text reports into structured data containing standardized fields, massive historical image reports are transformed from "dormant archives" into computable and mineable structured medical data assets. This provides a high-quality data foundation for model training in multi-center clinical research, early disease screening, and intelligent auxiliary diagnostic systems. Simultaneously, the standardized data format enables interconnection of image report data between different medical institutions, providing practical technical support for the implementation of hierarchical medical services and the balanced allocation of high-quality medical resources. In summary, this method is not only an intelligent tool to improve the efficiency of radiology departments, but also a core technology engine to promote the transformation of medical imaging from experience-driven to data-driven, and from isolated to collaborative and shared processes.
[0034] This application also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, can perform the artificial intelligence-based medical image report information extraction method and system described above.
[0035] The methods and systems according to the embodiments of this application can also be implemented using the architecture of the electronic device shown in this application. The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output, a hard disk, etc. The storage device in the electronic device, such as a ROM or hard disk, may store the AI-based medical image report information extraction method and system provided in this application. Furthermore, the electronic device may also include a user interface. Of course, the architecture shown in this application is merely exemplary; when implementing different devices, one or more components of the electronic device shown in this application may be omitted according to actual needs.
[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for extracting medical image report information based on artificial intelligence, characterized in that, The method includes: Step S1: Obtain medical imaging report data and medical imaging data for the same case; perform desensitization processing on the medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data; Step S2: Based on standardized medical image report data, use text segmentation technology and medical keyword matching technology to obtain structured medical image report data; Step S3: Based on the structured medical image report data, use the BioBERT model and information extraction network to obtain the structured medical image report information; Step S4: Based on standardized medical image data, use a deep learning-based medical image feature extraction method to obtain medical image visual feature data; Step S5: Based on the structured medical image report information and the visual feature vector of the medical image, use a multimodal feature fusion method to obtain fused medical feature data; Step S6: Based on the fusion medical feature data, use the consistency verification method based on image visual feature matching to obtain the consistency verification results; Step S7: Based on the structured medical image report information, fused medical feature data, and consistency verification results, use the field-based report generation method to generate an enhanced structured medical image report.
2. The method for extracting medical image report information based on artificial intelligence according to claim 1, characterized in that, The acquisition of medical imaging report data and medical imaging data for the same case includes: By interacting with the existing PACS / RIS / HIS systems of partner medical institutions, we can obtain medical imaging data and medical imaging report data generated after patients complete medical imaging examinations. The PACS system acquires DICOM format medical image data generated during the patient's medical imaging examination; the RIS system acquires medical image report data generated based on the medical image data.
3. The method for extracting medical image report information based on artificial intelligence according to claim 2, characterized in that, The process of desensitizing medical image report data and medical image data to obtain standardized medical image report data and standardized medical image data includes: Sensitive information in medical image report data is identified and replaced, and sensitive label fields in DICOM format medical image data are cleared or emptyed to form standardized medical image report data and standardized medical image data.
4. The method for extracting medical image report information based on artificial intelligence according to claim 3, characterized in that, The process of obtaining structured medical image report data based on standardized medical image report data, using text segmentation technology and medical keyword matching technology, includes: Standardized medical imaging report data is segmented into examination information area, image description area, and diagnostic conclusion area. For different regions, information is extracted using a pre-established medical imaging keyword database, including examination site, imaging features, disease description, and diagnostic opinions. Using medical terminology standardization technology, the extracted examination sites, imaging findings, disease descriptions, and diagnostic opinions are standardized and converted into unified medical terminology. The data is stored according to preset data fields to form structured medical imaging report data, including the examination site, imaging features, disease description, and diagnostic opinions.
5. The method for extracting medical image report information based on artificial intelligence according to claim 4, characterized in that, The process of obtaining structured medical image report information based on structured medical image report data, using the BioBERT model and information extraction network, includes: The examination site information, imaging findings information, disease description information, and diagnostic opinion information are combined to generate a unified input text; The unified input text is fed into the BioBERT model, where the text encoding layer of the model performs word segmentation to obtain word segments; and the attention mechanism in the semantic encoding layer of the model converts each word segment into a corresponding medical text feature vector. The medical text feature vector corresponding to each word is input into the information extraction network, and the medical information category label is extracted from the medical text feature vector through the information extraction network. The probability value of each word belonging to different medical information categories is calculated based on the matching relationship between the feature vector of medical text and each medical information category. The highest probability value is taken as the medical information category corresponding to the word. Each word segment is associated with its corresponding medical information category and its position in the same input text. Word segments that belong to the same medical information category and appear consecutively are merged to obtain the complete medical entity information for each medical information category. Structured medical image report information is obtained by using complete medical entity information for each medical information category.
6. The method for extracting medical image report information based on artificial intelligence according to claim 5, characterized in that, The step of obtaining medical image visual feature data based on standardized medical image data using a deep learning-based medical image feature extraction method includes: The standardized medical image data is preprocessed to generate standardized image tensor data; Standardized image tensor data are input into a pre-trained medical image feature extraction network to obtain multi-scale image feature maps. The pre-trained medical image feature extraction network employs an attention-based mechanism to weight and enhance the extracted multi-scale image feature maps. The weighted and enhanced multi-scale image feature maps are then subjected to feature dimensionality reduction through a global average pooling layer and a fully connected layer to generate fixed-dimensional medical image visual feature vectors.
7. The method for extracting medical image report information based on artificial intelligence according to claim 6, characterized in that, The process involves using a multimodal feature fusion method to obtain fused medical feature data based on structured medical image report information and medical image visual feature vectors, including: Key medical information in structured medical image reports is converted into a sequence of text feature vectors using a pre-trained medical text embedding model. The visual feature vectors of medical images are transformed in dimension through a linear mapping layer so that they have the same feature dimension as the text feature vector sequence. A cross-modal attention fusion mechanism is adopted, in which the text feature vector sequence is used as the query vector and the medical image visual feature vector is used as the key vector and value vector. The cross-modal attention weight between the text features and the visual features is calculated to obtain text-guided image enhancement features. Simultaneously, the medical image visual feature vector is used as the query vector, and the text feature vector sequence is used as the key vector and value vector. The cross-modal attention weight between the visual features and the text features is calculated to obtain the image-guided text enhancement features. The text-guided image enhancement features are concatenated and fused with the image-guided text enhancement features, and then integrated through a fully connected layer to generate fused medical feature data.
8. The method for extracting medical image report information based on artificial intelligence according to claim 7, characterized in that, The step of obtaining consistency verification results based on the fusion medical feature data and image visual feature matching method includes: Extract image visual quantitative feature data from the fused medical feature data; The image visual quantitative feature data is compared and verified with the image performance information and diagnostic opinion information in the structured medical image report information, and the feature matching degree between the image visual quantitative feature data and the corresponding lesion description parameters in the image performance information is calculated. Based on the relationship between the feature matching degree and the preset threshold, a consistency verification result is generated.
9. The method for extracting medical image report information based on artificial intelligence according to claim 8, characterized in that, The process involves generating an enhanced structured medical image report based on structured medical image report information, fused medical feature data, and consistency verification results, using a field-based report generation method. This includes: The basic report content is based on the examination site information, imaging manifestation information, disease description information, and diagnostic opinion information in the structured medical imaging report. The key information of the visual features of the images in the fused medical feature data is converted into objective descriptive statements and added to the image performance information after the basic report content; Based on the consistency verification results, the basic report content is annotated; all the above information is then spliced together in the order of examination site, imaging manifestations, disease description, diagnosis opinion, supplementary description, and verification annotation to generate an enhanced structured medical imaging report.
10. A system for extracting medical image report information based on artificial intelligence, which is applied to the method for extracting medical image report information based on artificial intelligence as described in any one of claims 1 to 9, characterized in that, include: Data preprocessing module: Acquires medical imaging report data and medical imaging data for the same case; De-identification processing was performed on medical imaging report data and medical imaging data to obtain standardized medical imaging report data and standardized medical imaging data; Report structuring module: Based on standardized medical image report data, it uses text segmentation technology and medical keyword matching technology to obtain structured medical image report data; Key information extraction module: Based on structured medical image report data, the BioBERT model and information extraction network are used to obtain structured medical image report information; Image feature extraction module: Based on standardized medical image data, it uses a deep learning-based medical image feature extraction method to obtain visual feature data of medical images; Cross-modal fusion module: Based on structured medical image report information and medical image visual feature vectors, a multimodal feature fusion method is used to obtain fused medical feature data; Consistency verification module: Based on the fused medical feature data, a consistency verification method based on image visual feature matching is used to obtain the consistency verification results; Report generation module: Based on structured medical image report information, fused medical feature data, and consistency verification results, an enhanced structured medical image report is generated using a field-based report generation method.
