Structured label generation method fusing medical image and text report

Abstract

The invention relates to medical image analysis, in particular to a medical image and text report fused structured label generation method, which comprises the following steps: reading a to-be-analyzed medical image and a corresponding text report, and preprocessing the medical image and the text report; the preprocessed medical image is converted into a three-channel Tensor type vector, and an image vector is obtained; encoding the preprocessed text report to obtain a text encoding vector; extracting multi-modal fusion features of the image vector and the text coding vector; inputting the multi-modal fusion features into a pre-trained text generation network RoBERTa, and generating a structured tag; according to the technical scheme provided by the invention, the defect that a structured medical image label is difficult to generate more comprehensively and accurately by combining a medical image and a text report in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to medical image analysis, and in particular to a structured label generation method integrating medical images and text reports. Background Art

[0002] With the continuous development of medical imaging technology, medical imaging data has shown explosive growth. Using medical images for intelligent diagnosis is a very meaningful thing, but because the original report may have problems such as colloquialism and non-standardization, such as "no obvious hyperplasia" and "soft tissue injury", it is difficult to directly use it for the training task of intelligent diagnosis AI, and requires standardized structured labels. However, the annotation of medical images is time-consuming and labor-intensive, and relies on the professional knowledge of doctors. The quality of annotation is greatly affected by subjective factors. Traditional medical image analysis methods mainly rely on manual interpretation, which is inefficient and prone to errors.

[0003] In order to solve the above problems, deep learning technology has been widely used in the field of medical image analysis in recent years. Researchers have achieved automatic segmentation and classification of medical images by training deep neural networks. However, most existing methods only focus on the medical images themselves, while ignoring the related text reports, such as medical records and diagnosis reports.

[0004] Text reports contain rich information such as lesion descriptions and diagnostic conclusions, which are highly complementary to medical images. Fusion analysis of medical images and text reports can extract information from medical images more comprehensively and accurately. However, how to effectively fuse medical images and text reports and automatically generate structured medical image labels remains a challenging problem. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for generating structured labels by integrating medical images and text reports, which can effectively overcome the defect of the prior art that it is difficult to combine medical images and text reports to generate structured medical image labels more comprehensively and accurately.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A structured label generation method integrating medical images and text reports comprises the following steps:

[0010] S1, reading the medical images to be analyzed and the corresponding text reports, and preprocessing the medical images and text reports;

[0011] S2, converting the preprocessed medical image into a three-channel Tensor type vector to obtain an image vector;

[0012] S3, encoding the preprocessed text report to obtain a text encoding vector;

[0013] S4, extracting multimodal fusion features of image vector and text encoding vector;

[0014] S5. Input the multimodal fusion features into the pre-trained text generation network RoBERTa to generate structured labels.

[0015] Preferably, S1 reads the medical image to be analyzed and the corresponding text report, and preprocesses the medical image and the text report, including:

[0016] S11, reading the DICOM medical images to be analyzed and the corresponding text reports;

[0017] S12, extract pixel arrays from DICOM medical images and resample them to a 224*224 format to ensure data quality and consistency;

[0018] S13. Combine the inspection findings and inspection impressions in the text report into a combined text.

[0019] Preferably, before extracting pixel arrays from DICOM medical images and resampling to obtain a size of 224*224 format to ensure data quality and consistency in S12, the following steps are included:

[0020] For the DICOM medical image sequences obtained by MRI and CT examination modes, the scanning order and slice position are determined according to the tag values ​​stored in each DICOM medical image, and the DICOM medical image sequences are sorted and spliced ​​in the third dimension to form a 3D array, and then resampled;

[0021] For a single DICOM medical image obtained in the X-ray examination mode, it is directly resampled.

[0022] Preferably, in S2, the preprocessed medical image is converted into a three-channel Tensor type vector to obtain an image vector, including:

[0023] Convert the resampled DICOM medical image into a three-channel Tensor type vector to obtain an image vector.

[0024] Preferably, in S3, the preprocessed text report is encoded to obtain a text encoding vector, including:

[0025] The pre-trained BERT model is used to encode the combined text to obtain the text encoding vector.

[0026] Preferably, the multimodal fusion features of the image vector and the text encoding vector are extracted in S4, including:

[0027] The image vector and text encoding vector are input into the MultimodalBert model to extract the multimodal fusion features of multimodal fusion calculation.

[0028] Preferably, before reading the medical image to be analyzed and the corresponding text report and preprocessing the medical image and the text report in S1, the following steps are included:

[0029] S01. Obtain relevant medical images and corresponding text reports, and manually annotate them to obtain corresponding structured labels to build a historical data set;

[0030] S02, dividing the historical data set into a training set, a validation set, and a test set according to a preset ratio;

[0031] S03. Set the loss function and optimizer of the text generation network RoBERTa;

[0032] S04, input the training set into the text generation network RoBERTa for model training;

[0033] S05. Calculate the loss value based on the loss function, and the optimizer updates the model parameters according to the loss value and network gradient information;

[0034] S06. If the loss value is less than the preset threshold, the model training ends, and the current text generation network RoBERTa is the trained text generation network RoBERTa. Otherwise, return to S04 and continue to use the training set for model training.

[0035] S07. Input the validation set into the trained text generation network RoBERTa, evaluate the generalization ability of the model by observing the performance on the validation set, and tune the hyperparameters and structure of the model;

[0036] S08. Input the test set into the tuned text generation network RoBERTa to evaluate the model performance.

[0037] Among them, the loss function of the text generation network RoBERTa is the BCE loss function.

[0038] Preferably, in S01, relevant medical images and corresponding text reports are obtained, and manually annotated to obtain corresponding structured labels to construct a historical data set, including:

[0039] S011. Obtain relevant medical images and corresponding text reports, and pre-process the medical images and text reports;

[0040] S012. Convert the preprocessed medical image into a three-channel Tensor type vector to obtain an image vector;

[0041] S013, encoding the preprocessed text report to obtain a text encoding vector;

[0042] S014. Extract multimodal fusion features of image vector and text encoding vector;

[0043] S015. Manually annotate each medical image and the corresponding text report to obtain the corresponding structured label and complete the construction of the historical data set.

[0044] Preferably, in S02, the historical data set is divided into a training set, a validation set and a test set according to a preset ratio, including:

[0045] The historical data set is divided into training set, validation set and test set in the ratio of 7:1:2.

[0046] Preferably, the structured label is based on a three-layer structural model for reading films and a knowledge graph, i.e., the lesion location is added as a special type of detail feature to the third-layer architecture of the knowledge graph, and a structured label in the form of [detail feature—lesion feature—lesion area] is obtained, so as to take into account two important information: lesion feature and lesion location.

[0047] (III) Beneficial effects

[0048] Compared with the prior art, the structured label generation method integrating medical images and text reports provided by the present invention has the following beneficial effects:

[0049] 1) By fusing the image vector of the medical image and the text encoding vector of the text report, it is possible to extract the information in the medical image more comprehensively and accurately, and to mine the semantic features of the text report, thereby avoiding the one-sided problem of generating structured labels based only on the text report, and thus generating structured medical image labels more comprehensively and accurately;

[0050] 2) Different from general structured labels, the structured labels in the present invention are based on three-layer structural model reading and knowledge graph, that is, the lesion location is added to the third-layer architecture of the knowledge graph as a special type of detail feature, and a structured label in the form of [detail feature-lesion feature-lesion area] is obtained, which takes into account two important information: lesion feature and lesion location, and uses horizontal lines to connect them, which is more standardized and can be applied to a variety of artificial intelligence model training tasks, such as image classification, image segmentation, target detection, etc., and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a schematic diagram of the process of the present invention;

[0053] Figure 2 Acquiring relevant medical images for the present invention;

[0054] Figure 3 For the present invention Figure 2 Screenshot of the text report. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] A structured label generation method that integrates medical images and text reports, such as Figure 1 As shown, S1, read the medical image to be analyzed and the corresponding text report, and pre-process the medical image and text report, specifically including:

[0057] S11, reading the DICOM medical images to be analyzed and the corresponding text reports;

[0058] S12, extract pixel arrays from DICOM medical images and resample them to a 224*224 format to ensure data quality and consistency;

[0059] S13. Report the findings and impressions of the examination in the text report (e.g. Figure 3 As shown) form a combined text.

[0060] Specifically, S12 extracts pixel arrays from DICOM medical images and resamples them to a 224*224 format to ensure data quality and consistency, including:

[0061] For the DICOM medical image sequences obtained by MRI and CT examination modes, the scanning order and slice position are determined according to the tag values ​​stored in each DICOM medical image, and the DICOM medical image sequences are sorted and spliced ​​in the third dimension to form a 3D array, and then resampled;

[0062] For a single DICOM medical image obtained in the X-ray examination mode, it is directly resampled.

[0063] Will Figure 3 The inspection findings and inspection impressions in the text report shown are combined into a combined text, and the combined text is as follows:

[0064] “

[0065] Examination findings: The chest radiograph showed: the chest was symmetrical on both sides, the trachea was right-deflected, a mass of high-density shadows could be seen in the right upper lung field, nodules were seen in the right lower lung, a curtain-like high-density shadow was seen above the diaphragm surface of the right lower lung field, the texture of both lungs increased, a few scattered fibrous cords could be seen in both lower lungs, the heart shadow was slightly full, the aortic knot was prominent with calcification, the descending aorta was tortuous and widened; the diaphragmatic surface on both sides was smooth, and the bilateral costophrenic angles were unclear. A strip of high-density shadow was seen on the lateral wall pleura outside the right middle lung field. (\nEnter)

[0066] Examination impression: mass shadow in the right upper lung field, nodule in the right lower lung, possible space-occupying mass, CT examination is recommended; chronic inflammation and old lesions in both lower lungs; possible bulging of the right diaphragm; prominent aortic node with calcification, and thickening of the right pleura.

[0067] ”

[0068] S2. Convert the preprocessed medical image into a three-channel Tensor type vector to obtain an image vector, including:

[0069] Convert the resampled DICOM medical image into a three-channel Tensor type vector to obtain an image vector.

[0070] S3. Encode the preprocessed text report to obtain a text encoding vector, which specifically includes:

[0071] The pre-trained BERT model is used to encode the combined text to obtain the text encoding vector.

[0072] S4. Extract multimodal fusion features of image vector and text encoding vector, including:

[0073] The image vector and text encoding vector are input into the MultimodalBert model to extract the multimodal fusion features of multimodal fusion calculation.

[0074] S5. Input the multimodal fusion features into the pre-trained text generation network RoBERTa to generate structured labels.

[0075] In the technical solution of the present application, structured labels are based on three-layer structural model reading and knowledge graph, that is, the lesion location is added to the third-layer architecture of the knowledge graph as a special type of detail feature, and a structured label in the form of [detail feature-lesion feature-lesion area] is obtained to take into account two important information: lesion characteristics and lesion location.

[0076] In the technical solution of the present application, before reading the medical image to be analyzed and the corresponding text report and preprocessing the medical image and text report in S1, the following steps are included:

[0077] S01. Obtain relevant medical images (such as Figure 2 ) and the corresponding text report (such as Figure 3 As shown in the figure), and manually annotate to obtain the corresponding structured labels to build a historical data set;

[0078] S02, dividing the historical data set into a training set, a validation set, and a test set according to a preset ratio;

[0079] S03. Set the loss function and optimizer of the text generation network RoBERTa;

[0080] S04, input the training set into the text generation network RoBERTa for model training;

[0081] S05. Calculate the loss value based on the loss function, and the optimizer updates the model parameters according to the loss value and network gradient information;

[0082] S06. If the loss value is less than the preset threshold, the model training ends, and the current text generation network RoBERTa is the trained text generation network RoBERTa. Otherwise, return to S04 and continue to use the training set for model training.

[0083] S07. Input the validation set into the trained text generation network RoBERTa, evaluate the generalization ability of the model by observing the performance on the validation set, and tune the hyperparameters and structure of the model;

[0084] S08. Input the test set into the tuned text generation network RoBERTa to evaluate the model performance.

[0085] Among them, the loss function of the text generation network RoBERTa is the BCE loss function.

[0086] 1) In S01, relevant medical images and corresponding text reports are obtained, and manually annotated to obtain corresponding structured labels to construct a historical data set, including:

[0087] S011. Obtain relevant medical images and corresponding text reports, and pre-process the medical images and text reports;

[0088] S012. Convert the preprocessed medical image into a three-channel Tensor type vector to obtain an image vector;

[0089] S013, encoding the preprocessed text report to obtain a text encoding vector;

[0090] S014. Extract multimodal fusion features of image vector and text encoding vector;

[0091] S015. Manually annotate each medical image and the corresponding text report to obtain the corresponding structured label and complete the construction of the historical data set.

[0092] right Figure 2 DICOM medical images and Figure 3 The text report shown is manually annotated, and the corresponding structured labels are as follows:

[0093] [Trachea and mediastinum - trachea is not centered, lung parenchyma - patchy shadows - location - lung fields - right lung - upper field of right lung, lung parenchyma - nodular shadows - location - lung fields - right lung - lower field of right lung, diaphragmatic surface - tentorium adhesions - location - diaphragmatic surface - right diaphragmatic surface, lung textures - increased - location - lung fields - left lung, lung textures - increased - location - lung fields - right lung, lung parenchyma - cord-like shadows - location - lung fields - left lung - lower field of left lung, lung parenchyma - cord-like shadows - location - lung fields - right lung - lower field of right lung, heart shadow - full, aorta - protruding, aorta - calcified, aorta - tortuosity, aorta - widened, costophrenic angles - unclear - location - costophrenic angles - left costophrenic angles, costophrenic angles - unclear - location - costophrenic angles - right costophrenic angles, pleura and thorax - pleural thickening].

[0094] 2) In S02, the historical data set is divided into a training set, a validation set, and a test set according to a preset ratio, including:

[0095] The historical data set is divided into training set, validation set and test set in the ratio of 7:1:2.

[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A structured label generation method integrating medical images and text reports, characterized by: The following steps are involved: S1, reading the medical images to be analyzed and the corresponding text reports, and preprocessing the medical images and text reports; S2, converting the preprocessed medical image into a three-channel Tensor type vector to obtain an image vector; S3, encoding the preprocessed text report to obtain a text encoding vector; S4, extracting multimodal fusion features of image vector and text encoding vector; S5. Input the multimodal fusion features into the pre-trained text generation network RoBERTa to generate structured labels.

2. The structured label generation method for integrating medical images and text reports according to claim 1 is characterized in that: S1 reads the medical images to be analyzed and the corresponding text reports, and preprocesses the medical images and text reports, including: S11, reading the DICOM medical images to be analyzed and the corresponding text reports; S12, extract pixel arrays from DICOM medical images and resample them to a 224*224 format to ensure data quality and consistency; S13. Combine the inspection findings and inspection impressions in the text report into a combined text.

3. The structured label generation method for integrating medical images and text reports according to claim 2 is characterized in that: In S12, the pixel array of DICOM medical images is extracted and resampled to a 224*224 format to ensure the quality and consistency of the data, including: For the DICOM medical image sequences obtained by MRI and CT examination modes, the scanning order and slice position are determined according to the tag values ​​stored in each DICOM medical image, and the DICOM medical image sequences are sorted and spliced ​​in the third dimension to form a 3D array, and then resampled; For a single DICOM medical image obtained in the X-ray examination mode, it is directly resampled.

4. The structured label generation method for integrating medical images and text reports according to claim 2 is characterized in that: In S2, the preprocessed medical image is converted into a three-channel Tensor type vector to obtain an image vector, including: Convert the resampled DICOM medical image into a three-channel Tensor type vector to obtain an image vector.

5. The structured label generation method for integrating medical images and text reports according to claim 4 is characterized in that: The preprocessed text report is encoded in S3 to obtain a text encoding vector, including: The pre-trained BERT model is used to encode the combined text to obtain the text encoding vector.

6. The structured label generation method for integrating medical images and text reports according to claim 5 is characterized in that: The multimodal fusion features of image vector and text encoding vector are extracted in S4, including: The image vector and text encoding vector are input into the MultimodalBert model to extract the multimodal fusion features of multimodal fusion calculation.

7. The structured label generation method for integrating medical images and text reports according to claim 6 is characterized in that: S1 reads the medical images to be analyzed and the corresponding text reports, and pre-processes the medical images and text reports, including: S01. Obtain relevant medical images and corresponding text reports, and manually annotate them to obtain corresponding structured labels to build a historical data set; S02, dividing the historical data set into a training set, a validation set, and a test set according to a preset ratio; S03. Set the loss function and optimizer of the text generation network RoBERTa; S04, input the training set into the text generation network RoBERTa for model training; S05. Calculate the loss value based on the loss function, and the optimizer updates the model parameters according to the loss value and network gradient information; S06. If the loss value is less than the preset threshold, the model training ends, and the current text generation network RoBERTa is the trained text generation network RoBERTa. Otherwise, return to S04 and continue to use the training set for model training. S07. Input the validation set into the trained text generation network RoBERTa, evaluate the generalization ability of the model by observing the performance on the validation set, and tune the hyperparameters and structure of the model; S08. Input the test set into the tuned text generation network RoBERTa to evaluate the model performance. Among them, the loss function of the text generation network RoBERTa is the BCE loss function.

8. The method for generating structured labels by integrating medical images and text reports according to claim 7, characterized in that: In S01, relevant medical images and corresponding text reports are obtained, and manually annotated to obtain corresponding structured labels to construct a historical data set, including: S011. Obtain relevant medical images and corresponding text reports, and pre-process the medical images and text reports; S012. Convert the preprocessed medical image into a three-channel Tensor type vector to obtain an image vector; S013, encoding the preprocessed text report to obtain a text encoding vector; S014. Extract multimodal fusion features of image vector and text encoding vector; S015. Manually annotate each medical image and the corresponding text report to obtain the corresponding structured label and complete the construction of the historical data set.

9. The method for generating structured labels by integrating medical images and text reports according to claim 7, characterized in that: In S02, the historical data set is divided into a training set, a validation set, and a test set according to a preset ratio, including: The historical data set is divided into training set, validation set and test set in the ratio of 7:1:

2.

10. The method for generating structured labels by integrating medical images and text reports according to claim 1 or 8, characterized in that: The structured label is based on a three-layer structural model for reading films and a knowledge graph, that is, the lesion location is added to the third-layer architecture of the knowledge graph as a special type of detail feature, and a structured label in the form of [detail feature—lesion feature—lesion area] is obtained, taking into account two important information: lesion feature and lesion location.

Images