A method for automatic generation of medical imaging reports based on deep learning

Abstract

The present invention discloses a method for automatically generating medical image reports based on deep learning, comprising the following steps: step 1, preparing a data set and preprocessing the data set; step 2, extracting fine-grained features of medical images through an Attention model; step 3, training through an image processing module using a text-image correspondence method, using keywords and image features for correspondence, and improving a transformer module to improve the accuracy of machine recognition of image features; step 4, generating report paragraphs through a Decoder module. The present invention extracts fine-grained medical image features and improves the training model, so that the generated diagnostic report is more professional and accurate.

Description

Technical Field

[0001] The present invention belongs to the intersection field of computer vision and natural language processing, and in particular relates to a method for automatically generating a medical imaging report. Background Art

[0002] With the rapid development of medical technology, medical imaging technology has continued to improve and perfect, forming a medical imaging system including X-ray, CT, PET-CT, MRI, etc. Medical imaging technology provides great help for doctors to diagnose diseases and becomes an important basis for doctors to diagnose diseases. The generation of medical imaging reports can help doctors to diagnose, reduce doctors' workload and the probability of missed diagnosis and misdiagnosis. At present, the data sets of medical imaging reports include ROCO, IU-Xray, MIMIC_CXR, CX-CHR, etc.

[0003] In 2016, Shin HC et al. [7] applied the encoder-decoder model to the generation of medical imaging reports. The encoder part used the GoogleNet convolutional network model for image analysis, and the decoder part used LSTM and GRU to generate reports. However, the generated results were not ideal and could not generate a complete report, but only a few medical terminology words.

[0004] In 2018, Baoyu Jing et al. [9] proposed a co-attention mechanism and introduced a Hierarchical LSTM structure. The article constructed a multi-task learning framework that jointly predicts labels and generates diagnostic reports. They proposed a co-attention mechanism to focus on abnormal information, taking into account both image features and semantic features, and then used a multi-layer LSTM model to generate paragraphs to improve the accuracy of prediction.

[0005] In 2020, Zhihong Chen et al.

[10] took advantage of the characteristics of medical imaging reports and added a memory module to the decoder part of the Transformer, which allows the model to better understand the local structure in the imaging report. The encoder and decoder of this model both use the transformer model. The Relation Memory module is introduced in the transformer of the decoder, and the memory is used to control the mean and variance of the transformer output features, thereby improving the quality of the generated medical imaging reports.

[0006] In 2021, Fenglin Liu et al.

[11] proposed a posterior-prior knowledge exploration and distillation (PPKED) framework, which combines prior and posterior knowledge to generate reports. This model mainly consists of three modules: the posterior knowledge exploration PoKE module, the prior knowledge exploration PrKE module, and the multi-domain knowledge distiller MKD module. The prior knowledge exploration module can reduce the bias of text generation by encoding prior knowledge, prior medical imaging reports and prior medical knowledge retrieved from the corpus. The posterior knowledge exploration module can classify the labels of the current image, determine the approximate abnormal area, and then perform identification analysis on this area. After the prior and posterior knowledge are comprehensively filtered by the multi-domain knowledge distillation module, the final report is obtained.

[0007] With the deepening of the research on medical image report generation based on deep learning, the generation of reports has become more and more fluent and standardized, but it is still far behind the traditional image caption in several evaluation indicators. This shows that although the generated reports are fluent and standardized, there are still many deficiencies, which will also cause many missed diagnoses and misdiagnoses. The feature extraction of medical images is more difficult than that of traditional RGB images. Traditional RGB images have significant features and objects, while the feature differences of medical images are very small, which is a test for the model. The sentences generated by traditional image captions are relatively simple and only need to describe the pictures, while medical image reports require professional terms for analysis and strong logic. In the process of report generation, it is necessary not only to describe the image information, but also to analyze the specific diagnosis. This is also a difficulty in the research of medical image report generation.

[0008] References:

[0009] [1] Vinyals O, Toshev A, Bengio S, et al.Show and tell: A neural imagecaption generator[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2015:3156-3164.

[0010] [2]Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need[C] / / Advances in neural information processing systems.2017:5998-6008.

[0011] [3]Feng Y,Ma L,Liu W,et al.Unsupervised image captioning[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2019:4125-4134.

[0012] [4]Cornia M,Stefanini M,Baraldi L,et al.Meshed-memory transformer forimage captioning[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2020:10578-10587.

[0013] [5]Kisilev P,Walach E,Barkan E,et al.From medical image to automaticmedical report generation[J].IBM Journal of Research and Development,2015,59(2 / 3):2:1-2:7.

[0014] [6]Kisilev P,Sason E,Barkan E,et al.Medical image captioning:Learningto describe medical image findings using multi-task-loss CNN[J].Deep Learningfor Precision Medicine,Riva del Garda,Italy,2016.

[0015] [7]Shin H C,Roberts K,Lu L,et al.Learning to read chest x-rays:Recurrent neural cascade model for automated image annotation[C] / / Proceedingsof the IEEE conference on computer vision and pattern recognition.2016:2497-2506.

[0016] [8]Xue Y,Xu T,Long L R,et al.Multimodal recurrent model withattention for automated radiology report generation[C] / / InternationalConference on Medical Image Computing and Computer-AssistedIntervention.Springer,Cham,2018:457-466.

[0017] [9]Jing B,Xie P,Xing E.On the Automatic Generation of Medical ImagingReports[J].arXiv:1711.08195.

[0018]

[10] Chen Z,Song Y,Chang T H,et al.Generating Radiology Reports viaMemory-driven Transformer[C] / / Proceedings of the 2020Conference on EmpiricalMethods in Natural Language Processing(EMNLP).2020.

[0019]

[11] Liu F,Wu X,Ge S,et al.Exploring and Distilling Posterior andPrior Knowledge for Radiology Report Generation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:13753-13762.

[0020]

[12] Pavlopoulos J,Kougia V,Androutsopoulos I.A survey on biomedicalimage captioning[C] / / Proceedings of the Second Workshop on Shortcomings inVision and Language.2019:26-36.

[0021]

[13] Li CY,Liang X,Hu Z,Xing EP.Hybrid Retrieval-Generation ReinforcedAgent for Medical Image Report Generation[J].arXiv:1805.08298.

[0022]

[14] Irvin J,Rajpurkar P,Ko M,et al.CheXpert:A Large Chest RadiographDataset with Uncertainty Labels and Expert Comparison[C] / / AAAI Conference onArtificial Intelligence,2019.

[0023]

[15] Li M,Wang F,X Chang,et al.Auxiliary Signal-Guided KnowledgeEncoder-Decoder for Medical Report Generation[J].arXiv:2006.03744.

[0024]

[16] Subramanian S, Wang LL, Bogin B, et al. MedICaT: A Dataset of MedicalImages, Captions, and Textual References[J].arXiv:2010.06000.

[0025]

[17] Wang F,

[0026]

[18] Huang JH, Wu TW, Worring M.Contextualized keyword representations for multi-modal retinal image captioning[C] / / International Conference onMultimedia Retrieval, 2021:242-250.

[0027]

[19] Li S, Sui X, Luo X, et al. Medical Image Segmentation using Squeeze-and-Expansion Transformers[C] / / International Joint Conference on ArtificialIntelligence, 2021.

[0028]

[20] Moon JH, Lee H, Shin W, et al.Multi-modal Understanding andGeneration for Medical Images and Text via Vision-Language Pre-Training.arXiv:2105.11333. Summary of the invention

[0029] The purpose of the present invention is to provide a method for automatically generating medical imaging reports based on deep learning, so as to achieve the purpose of generating more professional and accurate diagnostic reports.

[0030] To achieve the above object, the technical solution adopted by the present invention is:

[0031] A method for automatically generating medical imaging reports based on deep learning, comprising the following steps:

[0032] Step 1, prepare the data set and preprocess the data set;

[0033] Step 2: Extract fine-grained features of medical images through the Attention model;

[0034] Step 3: Through the image processing (Encoder) module, the image-text correspondence method is used for training, keywords and image features are matched, and the transformer module is improved to improve the accuracy of the machine's recognition of image features;

[0035] Step 4: Generate report paragraphs through the Decoder module.

[0036] In step 1, the data set is a medical imaging data set, which includes medical images and corresponding diagnostic reports. Through data preprocessing, the diagnostic reports are segmented and stored in a json file corresponding to the medical images, and Resnet101 is used to extract image features.

[0037] In step 1, the medical image datasets include the IU-Xray dataset, the ROCO dataset, and the MIMIC-CXR dataset. The IU-Xray dataset contains 3,955 reports and 7,470 images; the ROCO dataset contains more than 81,000 images and reports; and the MIMIC-CXR dataset contains 377,110 chest X-ray images of 65,739 patients, with a total of 227,835 imaging examinations.

[0038] In step 2, Resnet is first used to extract features, and then these features are input into the Attention model for pre-training. After processing, a more fine-grained feature input model is obtained for training.

[0039] In step 2, the overall structure of the Attention model is as follows:

[0040] input→Resnet→SAT model→Attention matrix→fine-grained features

[0041] Among them, the input is a medical image; Resnet is a deep residual network, and Resnet101 is used to extract the basic features of medical images;

[0042] The Attention model adopts the SAT (Show, Attend and Tell) model, which combines the attention mechanism with the traditional encoder-decoder framework. The role of the attention mechanism is to focus on different image areas and give different weights to different areas. The Attention matrix is ​​obtained through training, and then multiplied with the features to obtain fine-grained features.

[0043] In step 3, the overall structure of the image processing (Encoder) module is as follows:

[0044] input→image processing module→Encoder output

[0045] The input is fine-grained features and text keyword information; the standard transformer encoder module (encoder layer) processes a single image feature input, and the image processing module of the present invention is improved on the transformer encoder layer, so that attention is paid to both image features and text keyword features, so that the fine-grained features correspond to text keywords; the encoder output is the context vector context that combines vision and semantics.

[0046] In step 4, two LSTM modules and one Attention module are used in the Decoder module. The first LSTM module determines whether to continue generating sentences, the second LSTM module generates sentences, and the Attention module assists in generation.

[0047] In step 4, the overall structure of the Decoder module is as follows:

[0048] input→LSTM1→Attention→LSTM2→Decoder output

[0049] Among them, input is the output of the image processing (Encoder) module, which is a context vector; LSTM1 has two functions: the first function is to process the context vector input to the Attention module, and the second function is to control the end of paragraph generation. If it sends a Finish signal, LSTM2 stops generating sentences; the function of the Attention module is to receive the context vector processed by LSTM1 and generate an attention matrix to make the direct connection between each word in each sentence closer; LSTM2 generates a sentence word by word and obtains a complete diagnostic report through a loop.

[0050] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0051] (1) The present invention extracts fine-grained medical image features. Experiments show that when Resnet is used to extract features, the features are very similar and the model cannot learn anything. Therefore, the present invention uses the Attention model for pre-training to extract fine-grained features, so that the model can learn more and more useful features;

[0052] (2) The present invention uses a method that matches medical image features with text keywords for training. Traditional training methods only use image features for training, and the model can only learn superficial information and cannot generate professional medical diagnoses. The present invention uses image features and keywords for joint training, allowing the model to learn professional knowledge and prepare for subsequent report generation;

[0053] (3) The present invention uses two LSTM modules and one Attention module to generate report paragraphs. The two LSTM modules solve the problem of paragraph generation, and the Attention module solves the problem of ultra-long distance dependency in paragraph generation, making the generated report more complete, professional and accurate. DETAILED DESCRIPTION

[0054] The present invention will be further explained below.

[0055] A method for automatically generating a medical imaging report based on deep learning of the present invention comprises the following steps:

[0056] Step 1, prepare the dataset and preprocess it: 3 available datasets were obtained through reading papers and searching literature: IU-Xray, ROCO, and MIMIC-CXR. The IU-Xray dataset contains 3955 reports and 7470 images, of which each patient generally has two chest X-ray images, divided into front and side; the ROCO dataset contains more than 81,000 images and reports. The ROCO dataset is not a medical image of a single part, it contains CT and X-ray images of different parts such as the brain, chest, and hands; the MIMIC-CXR dataset contains 377,110 chest X-ray images of 65,739 patients, with a total of 227,835 imaging examinations. The ROCO dataset gives the official division, and the division of the IU-Xray and MIMIC-CXR datasets is processed according to the division in other papers.

[0057] First, the dataset is divided into training set, validation set and test set according to the conditions; secondly, the medical images are assigned image ids and correspond one-to-one with the reports; then the report is segmented using the word segmenter punkt and stored in a json file together with the report and images, and Resnet101 is used to extract image features.

[0058] Step 2: Extract fine-grained features of medical images based on SAT. The overall structure of the fine-grained feature extraction model is as follows:

[0059] input→Resnet→SAT model→Attention matrix→fine-grained features

[0060] Among them, the input is a medical image; Resnet is a trained deep residual network.

[0061] First, Resnet101 is used to extract basic medical image features and obtain a series of feature vectors a i :

[0062] a i ={a1,...,a L},a i ∈R D

[0063] Where L is the number of vectors, L = H*W, H represents the height of the feature map, W represents the width of the feature map, D is the dimension of the extracted feature map, and each vector corresponds to an area of ​​the medical image.

[0064] Then these feature vectors are input into the SAT model for pre-training. The present invention mainly uses the Encoder part of the SAT model, and obtains the required Attention matrix through the Attention module of the Encoder part. The function of the Attention module is to generate a set of weight matrices, using α t ={α t0 ,...,α ti ,...,α tL}, where α t It is the weight matrix, ti represents a at time t i The corresponding weights are calculated as follows:

[0065] e ti =f att (a i ,h t-1 )

[0066]

[0067] where h t-1 is the implicit vector of the decoder module at time t-1, f att It is a layer of neural network.

[0068] The present invention adopts the Soft Attention mechanism, that is, each area a i and its corresponding weight α at time t tiTake the weighted sum and get the context vector context vector:

[0069]

[0070] Finally, the obtained set of context vectors is multiplied by the features extracted later to obtain the required fine-grained features.

[0071] Step 3: In the image processing (Encoder) module, we use a text-image correspondence method for training, using keywords and image features to correspond, improving the attention module of the transformer so that it can pay attention to both image features and keyword features, and fuse the two attention matrices to improve the accuracy of the machine's recognition of image features. The overall structure of the Encoder module is as follows:

[0072] input→image processing module→Encoder output

[0073] The input is fine-grained features and text keyword information; the standard transformer encoder module (encoder layer) processes a single image feature input, and the image processing module of the present invention is improved on the transformer encoder layer, so that attention is paid to both image features and text keyword features, so that the fine-grained features correspond to text keywords; the encoder output is a context vector context that combines vision and semantics.

[0074] The processed fine-grained features and corresponding keyword features are input into the attention module of the transformer, and a single-layer feedforward network is used to calculate the visual attention weight matrix α respectively. v,n and the semantic attention weight matrix α a,m , the calculation process is as follows:

[0075]

[0076]

[0077] Among them, Wv, Wv ,h , is the parameter matrix of the visual attention network; Wa, Wa ,h , is the parameter matrix of the semantic attention network; h is the hidden state.

[0078] This step also uses the Soft Attention mechanism, which is to perform a weighted summation of the feature vectors of each region and its corresponding weight at a certain moment to obtain the context vector context vector. Therefore, the calculation process of the visual context vector and the semantic context vector is as follows:

[0079]

[0080]

[0081] Finally, the visual context vector and the semantic context vector are linked as [v att ; a att ], and then use a fully connected layer FC to integrate the two attention matrices to get the final context vector context vector:

[0082] c=FC[v att ; a att ]

[0083] After processing, the image processing (Encoder) module finally obtains the context vector c, which contains visual features and semantic features. Finally, the context vector c is input into the Decoder module for report generation.

[0084] Step 4: Generate report paragraphs through the Decoder module. The Decoder module uses two LSTM modules. The first LSTM module determines whether to continue generating sentences, and the second LSTM module generates sentences. Since paragraph generation is a long-distance dependency problem, using only LSTM is not effective, so an Attention module is introduced to assist generation.

[0085] The overall structure of the Decoder module is as follows:

[0086] input→LSTM1→Attention→LSTM2→Decoder output

[0087] Among them, input is the output of the Encoder module, which is a context vector. LSTM1 has two functions: the first function is to process the context vector input to the Attention module; the second function is to control the end of paragraph generation. If it sends a Finish signal, LSTM2 stops generating sentences. The function of the Attention module is to receive the context vector processed by LSTM1 and generate an attention matrix to make the direct connection between each word in each sentence closer. LSTM2 generates a sentence word by word and obtains a complete diagnostic report through a loop.

[0088] First, the input passes through LSTM1. The main function of LSTM1 is to determine whether to continue generating sentences. A deep output layer is applied to control the generation of sentences. s-1 and the hidden state h at time s s As input, the calculation process is as follows:

[0089] p(Finish|h s-1 ,h s )∝exp{W finish tanh(W finish,s-1 h s-1 +W finish,s h s )}

[0090] Where W finish , W finish,s-1 , W finish,s is the parameter matrix, h s-1 is the hidden state at time s-1, h s is the hidden state at time s.

[0091] If p(Finish|h s-1 ,h s ) is greater than the set threshold, it means that sentence generation stops. After LSTM2 receives the signal, it stops generating words and report generation ends.

[0092] The p(Finish|h s-1 ,h s ) and the corresponding input is sent to the Attention module, which performs s-1 ,h s ) is not processed, the Attention module processes the corresponding context vector so that it can focus on more distant information, and then inputs it into the LSTM2 module for sentence generation.

[0093] First, use the context vector obtained by the Encoder module that combines perception and semantics and the hidden state at that moment to get a vector t:

[0094] t=tanh(W t,h h+W t,c c)

[0095] Where W t,h , W t,c is the weight matrix, h is the hidden state, and c is the context vector combining vision and semantics obtained by the Encoder module.

[0096] LSTM2 uses a special start flag START and a vector t as input, and uses the hidden state h of the word word To predict the distribution of words:

[0097] p(word|h word )∝exp(W out h word )

[0098] Where W out is the parameter matrix, h word is the hidden state of the word.

[0099] LSTM2 generates one word at a time, then concatenates these words to form a sentence, and then concatenates each sentence to get the final diagnostic report.

[0100] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for automatically generating medical imaging reports based on deep learning, characterized in that: The steps include: Step 1, prepare the data set and preprocess the data set; Step 2: Extract fine-grained features of medical images through the Attention model; The overall structure of the fine-grained feature extraction model is as follows: input→Resnet→SAT model→Attention matrix→fine-grained features Among them, the input is a medical image; Resnet is a trained deep residual network; First, Resnet101 is used to extract basic medical image features and obtain a series of feature vectors a i : a i ={a1,...,a L },a i ∈R D Where L is the number of vectors, L = H*W, H represents the height of the feature map, W represents the width of the feature map, D is the dimension of the extracted feature map, and each vector corresponds to an area of ​​the medical image; Then, these feature vectors are input into the SAT model for pre-training. The Encoder part of the SAT model is used to obtain the required Attention matrix through the Attention module of the Encoder part. The function of the Attention module is to generate a set of weight matrices, using α t ={α t0 ,...,α ti ,...,α tL }, where α t It is the weight matrix, ti represents α at time t i The corresponding weights are calculated as follows: e ti =f att (a i ,h t-1 ) where h t-1 is the implicit vector of the decoder module at time t-1, f att It is a layer of neural network; Adopt Soft Attention mechanism, that is, each area a i and its corresponding weight α at time t ti Take the weighted sum and get the context vector context vector: Finally, the obtained set of context vectors is multiplied by the features extracted later to obtain the required fine-grained features; Step 3: Through the image processing module, the image-text correspondence method is used for training, keywords and image features are matched, and the transformer module is improved to improve the accuracy of the machine's recognition of image features; Step 4: Generate report paragraphs through the Decoder module.

2. The method for automatically generating medical imaging reports based on deep learning according to claim 1, characterized in that: In step 1, the data set is a medical imaging data set, which includes medical images and corresponding diagnosis reports. Through data preprocessing, the diagnosis reports are segmented and stored in a json file corresponding to the medical images, and Resnet101 is used to extract image features.

3. The method for automatically generating medical imaging reports based on deep learning according to claim 2, characterized in that: In step 1, the medical imaging dataset includes an IU-Xray dataset, a ROCO dataset, and a MIMIC-CXR dataset.

4. The method for automatically generating medical imaging reports based on deep learning according to claim 1, characterized in that: In step 3, the overall structure of the image processing module is as follows: input→image processing module→Encoder output The input is fine-grained features and text keyword information; the Encoder output is the context vector context that combines vision and semantics.

5. The method for automatically generating medical imaging reports based on deep learning according to claim 1, characterized in that: In step 4, two LSTM modules and one Attention module are used in the Decoder module. The first LSTM module determines whether to continue generating sentences, the second LSTM module generates sentences, and the Attention module assists in generation.

6. The method for automatically generating medical imaging reports based on deep learning according to claim 1 or 5, characterized in that: In step 4, the overall structure of the Decoder module is as follows: input→LSTM1→Attention→LSTM2→Decoder output Among them, input is the output of the image processing module, which is a context vector; LSTM1 has two functions: the first function is to process the context vector input to the Attention module, and the second function is to control the end of paragraph generation. If it sends a Finish signal, LSTM2 stops generating sentences; the function of the Attention module is to receive the context vector processed by LSTM1 and generate an attention matrix to make the direct connection between each word in each sentence closer; LSTM2 generates a sentence word by word and obtains a complete diagnostic report through a loop.

Images