A method for automatic generation of brain CT medical reports based on weakly supervised attention

By introducing weak supervision attention mechanism and Grad-CAM algorithm into brain CT images, combined with keyword-driven interactive circulation network, the problem of identifying micro-lesion areas in brain CT images is solved, and efficient and accurate medical report generation is achieved, which improves the Diagnostic efficiency and reporting accuracy.

CN113313199BActive Publication Date: 2024-11-12BEIJING UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110685415.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-21
Publication Date
2024-11-12
Estimated Expiration
2041-06-21

AI Technical Summary

Technical Problem

Existing automatic generation of medical reports is difficult to accurately capture micro-lesion areas in brain CT images, and the failure to effectively utilize prior knowledge, resulting in inaccuracy and inefficiency in generating reports.

Method used

Using a weak supervision-based attention mechanism, combined with Grad-CAM algorithm and keyword-driven interactive circulation network, important lesion characteristics of brain CT images are extracted and accurate medical reports are generated. This method uses the spatial attention and sequence attention mechanism to modify spatial attention using gradient weighted class activation map (Grad-CAM), improves the recognition ability of lesion areas, and activates the language generation module through the keyword driving mechanism to improve the accuracy of the report sex.

Benefits of technology

It significantly improves the accuracy and efficiency of brain CT medical reports, can accurately capture lesion areas, and generate logical medical reports, improving the efficiency and accuracy of doctors' diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113313199B_ABST
    Figure CN113313199B_ABST
Patent Text Reader

Abstract

The present invention proposes a method for automatically generating brain CT medical reports based on weakly supervised attention, which involves three fields: medical images, computer vision, and natural language processing. A weakly supervised attention mechanism WGAM is designed to explicitly guide the attention model to focus on the lesion area, thereby improving the accuracy of medical report generation. WGAM is a hierarchical structure, including two attention mechanisms: spatial attention and sequence attention. The spatial attention is weakly supervised by the gradient-weighted class activation mapping algorithm to obtain better attention effects. A keyword-driven interactive recurrent network KIRN is designed as a language generation module to generate brain CT medical reports. The hidden layer state of the language generation module is activated by keyword information containing lesion location information, and the LSTM is used to generate the brain CT medical report. word and LSTM sen The dynamic interaction improves the accuracy of generating brain CT image reports. The present invention explores the work of automatically generating brain CT medical reports for the first time and achieves effectiveness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to three fields of medical imaging, computer vision and natural language processing. Aiming at the goal of automatically generating brain CT medical reports, an automatic generation method for brain CT medical reports based on weakly supervised attention is designed. Background Art

[0002] Nowadays, medical imaging technology is indispensable in actual treatment. Many diseases require patients to take medical images and have them diagnosed by doctors before they can be diagnosed and treated. In many countries, radiologists may need to read hundreds of radiological images every day and write reports on their diagnosis results. It takes doctors 5-10 minutes to diagnose medical images and input the analysis results of each image into the computer, which takes up most of their energy and time.

[0003] Among many medical imaging technologies, cranial imaging (brain CT) can be used to diagnose many cerebrovascular diseases, such as cerebral hemorrhage, cerebral infarction, subarachnoid hemorrhage, intracerebral hematoma, etc. The common feature of these diseases is that they cause ischemia or hemorrhagic accidents in brain tissue. Rapid and accurate diagnosis of diseases is very important for clinical treatment. Therefore, assisting doctors to more efficiently discover lesions and write medical reports has high research and application value. Among them, the automatic generation of medical imaging reports is a key link in diagnosis, which is of great significance for improving the efficiency and accuracy of doctors' diagnosis.

[0004] Medical report generation is an emerging research direction in the intersection of medicine and computer science in recent years. Its goal is to automatically generate a coherent and logical medical report for a medical image. At present, the data carriers of this work are mainly chest X-ray images and their corresponding diagnostic reports, and the task of generating medical reports for brain CT has not yet been explored. Compared with other image data, brain CT images are multiple sequences with three-dimensionality; lesions are continuous and sparse; and the report content is complex and has stronger unstructured nature.

[0005] Most existing methods for automatic medical report generation use the encoder-decoder framework in the field of natural image description. This framework uses neural network structures such as CNN as encoders to extract visual features of input images, and uses recurrent neural networks such as RNN as decoders to generate natural and fluent sentences. In order to further improve the accuracy of image description and better align image and text features, the attention mechanism is introduced into the encoder-decoder framework. The attention mechanism originates from the study of the human brain's visual system. Cognitive science research shows that due to the bottleneck of information processing in the nervous system, the human brain selectively focuses on the important part of all information while ignoring other useless information. At present, the attention mechanism is widely used in many image processing related fields. This mechanism assigns different weights to different features, so that the model can selectively focus on the important part of the features while ignoring other visible information. The introduction of the attention mechanism in medical report generation can imitate the doctor's experience in reading films, focusing on the visual information of medical images, so that the generated words are more accurate.

[0006] Existing methods for automatically generating medical reports based on attention mechanisms do not consider prior knowledge, such as the possible location of lesion areas. Brain CT medical report generation needs to focus more on tiny lesion areas. Therefore, it is extremely difficult to let the model learn to give larger attention weights to these tiny lesion areas through gradient descent methods without introducing any prior knowledge. Inspired by the guided attention mechanism and gradient-weighted class activation mapping (Grad-CAM), the present invention attempts to use the Grad-CAM method in the automatic generation method of brain CT medical reports to guide the attention mechanism to accurately capture the important lesion areas and generate accurate brain CT reports. Summary of the invention

[0007] At present, the work of automatic generation of brain CT reports has not been explored, and research in this field at home and abroad is still blank. Figure 1 As shown, compared with other image data, brain CT medical images are multiple sequences and three-dimensional; the lesions are continuous and sparse; the report content is complex and has stronger unstructured nature. Existing methods are not suitable for the automatic generation of brain CT reports. The introduction of the attention mechanism can solve the above problems, but the existing attention mechanism only relies on the gradient descent method to automatically learn to assign different weights to different positions without considering any prior knowledge such as the possible location of bleeding lesions. In order to improve the effect of the attention mechanism, the present invention attempts to use the Grad-CAM method in the automatic generation method of brain CT medical reports to guide the attention mechanism to accurately capture the area where important lesions are located, thereby generating an accurate brain CT report.

[0008] To achieve the above objectives, the present invention proposes a method for automatically generating brain CT medical reports based on weakly supervised attention.

[0009] It includes two stages: training and prediction.

[0010] The training phase includes:

[0011] (1) Create and preprocess a training dataset for brain CT medical report generation to obtain standardized three-dimensional brain CT images and corresponding reports;

[0012] (2) Using the brain CT image feature extractor to extract the features of the brain CT image F = [f1, f2, ... f N ], where N represents the number of brain CT sequences in each case, and f i Represents the image features of the i-th brain CT sequence;

[0013] (3) Constructing a weakly supervised attention module, which is used to extract important lesion features a from the image features of N brain CT images in sequence, including a spatial attention mechanism, a sequence attention mechanism, and a modified spatial attention mechanism: wherein the spatial attention mechanism is used to extract important lesion features in each brain CT sequence, and the mechanism has the ability to capture important features in a single sequence by assigning different weights to different positions of a single sequence and assigning higher weights to key features; the sequence attention mechanism is used to extract important sequence features in N brain CT images in sequence, and the key sequence features are assigned higher weights to participate in feature addition, so as to obtain the image features a that finally contain important lesions in the entire CT image; the modified spatial attention mechanism is used to correct inaccurate spatial attention and enhance the ability of spatial attention to extract lesion features, specifically using the gradient weighted class activation mapping Grad-CAM algorithm to generate supervision information for spatial attention, and then calculating the cross entropy loss function Loss between the spatial attention and the supervision information. attn , participates in the overall model training as part of the overall loss;

[0014] (4) Construct a language model for generating a brain CT medical report. The input of the language model is the important lesion feature a extracted in step (3), and the output is the brain CT medical report. The language model is a keyword-driven interactive recurrent network. The keyword-driven interactive recurrent network consists of two parts. The first part is the keyword-driven mechanism, which is used to activate the hidden layer vector h at the initial moment of the LSTM in the interactive recurrent network. t , t = 0, the input of this mechanism is the keyword K of the CT image predicted by the keyword classification network and the image feature a of the important lesion of the CT image obtained by the weakly supervised attention module. The keyword refers to the bleeding location of the brain CT image. The keyword driving mechanism is a single-layer LSTM. The calculation process is as follows:

[0015] h k =LSTM k (K, a)

[0016] h k Represents the hidden layer vector of LSTM in the keyword mechanism;

[0017] The second part is the interactive recurrent network, which is used to generate medical reports. It contains two layers of LSTM. The first layer is the word LSTM, represented by LSTM word , which is used to iteratively generate a word for each moment, and multiple words are finally combined into a sentence s m ,s m represents the mth sentence generated; the second layer is the sentence LSTM, represented by LSTM sen , which encodes the relationship between different sentences and guides the word LSTM to generate the words of the next sentence. The hidden layer vector of the two-layer LSTM is defined as follows:

[0018]

[0019] in Represents the hidden layer vector of the word LSTM at time t, where time t refers to the number of words generated. Represents the hidden layer vector of the word LSTM at the previous moment, y t-1 Represents the last moment LSTM wor d generated words, a t is the image feature input at time t, The hidden layer vector of the previous sentence generated by the sentence LSTM, S is the hidden layer vector of the current sentence generated by the sentence LSTM, S m-1 is the previous sentence predicted; among them, That is the output of the keyword-driven mechanism;

[0020] The process of word LSTM generating words is as follows:

[0021]

[0022] Here t , p t Represent the word generated by LSTM at the tth moment and the probability of the word, W h Represents the hidden layer vector The parameter matrix of Represents the hidden layer vector of the word LSTM at time t; finally, the word LSTM and sentence LSTM interact to generate the entire long paragraph of the medical report.

[0023] (5) Construct the overall loss function and train it. The overall loss function consists of two parts:

[0024] Loss=LOSS lang +Loss attn

[0025] in, Refers to y t The real label.

[0026] The forecasting phase includes the following steps:

[0027] (6) preprocessing the brain CT to be predicted to obtain a standardized three-dimensional brain CT image;

[0028] (7) using the trained brain CT image feature extractor to extract features of the brain CT image to be predicted;

[0029] (8) Using the trained weakly supervised attention module, important lesion features are extracted from the features of the brain CT images to be predicted;

[0030] (9) Using the trained language model, a medical report of the brain CT image to be predicted is generated.

[0031] Furthermore, the specific steps of step (1) are as follows:

[0032] Step (1.1): Collect brain CT images to construct a data set. Each patient data contains the RGB matrix I = {I1…, I N}, Brain CT report Where N represents the number of brain CT sequences in each case, I i Represents the RGB matrix of the i-th sequence, W and H represent the width and height of the sequence respectively, represents the i-th word of a report, Represents the number of words in a report;

[0033] Step (1.2) divides all patient data into training set, validation set and test set. The training set is used to learn the parameters of the neural network; the validation set is used to determine the hyperparameters; and the test set is used to verify the classification effect of the neural network.

[0034] Step (1.3) Data preprocessing: For brain CT images with a sequence number not exceeding 24, an interpolation algorithm is used to complete the sequence, and for brain CT images with a sequence number exceeding 24, a uniform sampling method is used to select the sequence.

[0035] Furthermore, the feature extractor in step (2) uses the Resnet101 network, and the calculation process of the feature F extracted from the brain CT image is as follows:

[0036] F = Resnet101 (I).

[0037] Furthermore, the working process of the weakly supervised attention module in step (3) is as follows:

[0038] Step (3.1) constructs the spatial attention mechanism in the weakly supervised attention module to extract important lesion features in each CT sequence. Specifically, this mechanism extracts the important lesion features in each brain CT sequence by i Calculate its spatial attention matrix The calculation process is as follows:

[0039] e i =φ(h,f i )

[0040]

[0041] φ is a feedforward neural network used to automatically learn the relationship matrix between its two internal parameters, h is the hidden layer vector inside the LSTM used to generate medical reports, and finally the spatial attention matrix is ​​obtained is a two-dimensional matrix, whose matrix values ​​correspond to the weights assigned by the attention mechanism to different spatial positions in a single sequence. Finally, the brain CT image feature is obtained by weighted sum of weight features: V = [v1, v2, ...v N ], where v i Represents the sequence feature of the i-th image containing important lesion features, and its calculation formula is:

[0042]

[0043] Step (3.2) constructs the sequence attention mechanism in the weakly supervised attention module to extract important sequence features from N sequences of brain CT images. Specifically, the mechanism calculates the sequence attention weight a for all sequence features of each brain CT image. Frame , where a Frame is a one-dimensional vector, whose vector value corresponds to the weight given by the attention mechanism to different sequences. Finally, the important sequence feature a in the N sequences of brain CT images is obtained by weighted sum of weight features. The calculation formula is:

[0044]

[0045] Step (3.3) uses weakly supervised attention to correct spatial attention. The specific steps are as follows:

[0046] (I) For a single sequence, the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm is used to generate the corresponding class activation map. The color of the important lesion area in the class activation map is marked, and then the image segmentation tool is used to draw the detection box R of the lesion in the single image. i ; (II) The generated detection box R i Transformed into supervision information β for spatial attention i , supervision information β i is a two-dimensional matrix, and the transformation process of any element is as follows:

[0047]

[0048] Where (l, c) represents the lth row and cth column of the supervision matrix. When the position is in the lesion detection frame, the matrix value of the position is set to 1, otherwise it is set to 0;

[0049] (III) After generating the supervision information of spatial attention, the cross entropy loss function of spatial attention and supervision information is calculated and used as part of the overall loss to participate in the overall model training. At this time, the incorrect spatial attention matrix generated by spatial attention Supervised information β i The correction can extract important lesion features of a sequence, and the loss function is defined as follows:

[0050]

[0051] Furthermore, the method for generating keywords for the keyword-driven mechanism is as follows:

[0052] (I) Two types of keywords are defined: one is the three directional keywords, left side, right side, and bilateral, which are used to indicate which side of the brain midline the lesion is located on; the other is the high-frequency location keywords that appear in brain CT medical reports, including 11 locations: frontal lobe, temporal lobe, parietal lobe, occipital lobe, semiovale, lateral ventricle, basal ganglia, thalamus, brainstem, cerebellum, and cerebrum;

[0053] (II) For each brain CT image, two types of keywords are extracted from the corresponding report. A keyword classification network is trained based on the obtained keywords to automatically learn the keywords of the image. Finally, the trained keyword classification network is used to predict the keywords of the brain CT image. The network predicts that the keywords are K = {k1, k2, ..., k n}, where n is the number of keywords, k i is the ith keyword, and the keyword classification network adopts Resnet34.

[0054] Compared with the existing methods, the present invention has the following obvious advantages and innovations:

[0055] The present invention proposes a method for automatically generating brain CT medical reports based on weakly supervised attention, and the method has the following characteristics: 1) A weakly supervised attention mechanism (WGAM) is designed to explicitly guide the attention model to focus on the lesion area, thereby improving the accuracy of medical report generation. WGAM is a hierarchical structure, including two attention mechanisms: spatial attention and sequential attention, in which the spatial attention is weakly supervised by the gradient-weighted class activation mapping (Grad-CAM) algorithm to obtain better attention effect. 2) A keyword-driven interactive recurrent network is designed as a language generation module to generate brain CT medical reports, and the hidden layer state of the language generation module is activated by keyword information containing lesion location information, and the LSTM is used to generate the brain CT medical report. word and LSTM sen The dynamic interaction improves the accuracy of generating brain CT image reports. 3) This paper is the first to explore the automatic generation of brain CT medical reports. The sufficient experiments in this paper demonstrate the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 : Schematic diagram of brain CT images and reports.

[0057] Figure 2 : Framework diagram of a method for automatic generation of brain CT medical reports based on weakly supervised attention.

[0058] Figure 3 : Schematic diagram of a keyword-driven interactive recurrent network.

[0059] Figure 4 : Generate reports and attention visualization presentations. DETAILED DESCRIPTION

[0060] In this embodiment, the brain CT images and corresponding reports provided by the cooperative hospital are used as the research objects, and the implementation steps of this method are specifically described:

[0061] Step (1) Obtain brain CT images and corresponding medical report data and preprocess them:

[0062] Step (1.1) Data: Collect brain CT images to construct a data set. The data set contains a total of 2048 brain CT images and corresponding reports from which patient information has been deleted. The image data of each patient contains multiple CT sequences, and the brain CT report is in the form of long paragraphs: represents the i-th word of the report, Represents the number of words in the report. After obtaining the data provided by the hospital, we first use a python script to convert the CT images in dicom format into PNG images in RGB format: I = {I1…, I N}, Where N represents the number of CT slices for each case, and W and H represent the width and height of each sequence, respectively.

[0063] Step (1.2) randomly divides the data into three parts: training set, validation set and test set, with the number of data being 1848 / 100 / 100.

[0064] Step (1.3) Data preprocessing: First, the number of different brain CT image sequences was unified to 24, and the number of brain CT sequences in the acquired data set ranged from 12 to 64. For brain CT images with a sequence number not exceeding 24, an interpolation algorithm was used to complete them, and for brain CT images with a sequence number exceeding 24, a uniform sampling method was used to select them. Then, the width and height of the brain CT image sequence images were unified to 512×512, and finally, the brain CT image sequence was standardized and homogenized.

[0065] Step (2) Constructing a feature extractor: First, a binary classification dataset is constructed based on whether there are hemorrhagic lesions in the brain CT images. Then, the pre-trained ResNet101 model is used to fine-tune the binary classification task based on the dataset. Finally, the feature map of the last convolutional neural network layer of the ResNet101 network is extracted. As the image features of brain CT images, 2048 represents the number of channels, 24 represents the number of sequences in a brain CT image, and 196 represents the number of spatial attention units in the feature map. The formula for extracting features is as follows:

[0066] F=Re et101(I)

[0067] Step (3) constructs the weakly supervised attention module proposed in the present invention to extract important lesion features a from the image features of the brain CT images containing N sequences, which includes a spatial attention mechanism, a sequence attention mechanism, and a modified spatial attention mechanism:

[0068] Step (3.1) constructs the spatial attention mechanism module in the weakly supervised attention module to capture the lesion area features in each CT image sequence, that is, for each brain CT sequence I i The spatial characteristics F i Calculate its spatial attention weight The calculation process is as follows:

[0069] e i =φ(h,f i )

[0070]

[0071] φ is a feedforward neural network that can automatically learn the relationship matrix between its two internal parameters. h is the hidden layer vector inside the LSTM used to generate medical reports. Finally, the spatial attention matrix is ​​obtained. in is a two-dimensional matrix, whose matrix values ​​correspond to the weights assigned by the attention mechanism to different spatial positions in a single sequence. Finally, the brain CT image feature is obtained by weighted sum of weight features: V = [v1, v2, ...v N ], where v i Represents the sequence feature of the i-th image containing important lesion features, and its calculation formula is:

[0072]

[0073] Step (3.2) constructs the sequence attention mechanism in the weakly supervised attention module to extract important sequence features from N sequences of brain CT images. The mechanism calculates the sequence attention weight a for all sequence features of each brain CT image. Frame , where a Frame is a one-dimensional vector, whose vector value corresponds to the weight given by the attention mechanism to different sequences. Finally, the important sequence feature a in the N sequences of brain CT images is obtained by weighted sum of weight features. The calculation formula is:

[0074]

[0075] Step (3.3) uses weakly supervised attention to correct inaccurate spatial attention. The specific approach is:

[0076] (I) For a single sequence, the Gradient Weighted Class Activation Map (Grad-CAM) algorithm is used to generate the corresponding class activation map. The color of the important lesion area in the class activation map is marked, and then the image segmentation tool is used to draw the detection box R of the lesion in the single image. i ;

[0077] (II) The generated detection box R i Transformed into supervision information β for spatial attention i , supervision information β i is a two-dimensional matrix, and the transformation process of any element is as follows:

[0078]

[0079] Where (l, c) represents the lth row and cth column of the supervision matrix. If the position is in the lesion detection frame, the matrix value of the position is set to 1, otherwise it is set to 0

[0080] (III) After generating the supervision information of spatial attention, the cross entropy loss function of spatial attention and supervision information is calculated and used as part of the overall loss to participate in the overall model training. At this time, the incorrect spatial attention matrix generated by spatial attention Supervised information β iCorrection can be made to extract important lesion features of a sequence. The loss function is defined as follows:

[0081]

[0082] Step (4) constructs a language model for generating medical reports. The input of the language model is the image feature a with important lesion features extracted by the weakly supervised attention module in step 3, and the output is the brain CT medical report predicted by the model. Since the paragraphs of medical reports are long, the traditional single-layer LSTM driven by image features alone is not enough to generate accurate brain CT medical reports. Therefore, we propose a keyword-driven interactive recurrent network as the language model in the present invention. The keyword-driven interactive recurrent network mainly includes two parts. The first part is the keyword-driven mechanism, which is used to activate the hidden layer vector h at the initial moment of the LSTM in the interactive recurrent network. t , t=0, the second part is the interactive recurrent network, which is used to generate medical reports. The specific steps are as follows:

[0083] Step (4.1) The keyword-driven mechanism is used to activate the hidden layer vector h0 at the initial moment of the interactive recurrent network. The input of this mechanism is the keyword corresponding to a brain CT image and the image feature a corrected by the supervised attention mechanism in step 3. The keyword refers to the bleeding location of a brain CT image. For example, the keywords of the patient's brain CT image are left and thalamus, indicating that the patient's left brain is bleeding in the thalamus. Since the brain CT data set only contains images and reports, we use the following steps to extract the keywords corresponding to each image from the report. The keywords obtained at this time are the real labels and are not used as the input of the keyword-driven mechanism. Then, a classification network is trained based on each brain CT image and the corresponding keywords extracted to automatically learn the keywords of the image. At this time, the keywords predicted by the network are used as the input information in the keyword-driven mechanism. The specific steps are as follows:

[0084] (I) First, with the assistance of radiologists, two types of keywords were defined in advance. One type was the three directional keywords of left and right and bilateral, indicating which side of the brain midline the lesion was located on. The other type was the high-frequency location words that appeared in brain CT medical reports, including 11 locations: frontal lobe, temporal lobe, parietal lobe, occipital lobe, semiovale, lateral ventricle, basal ganglia, thalamus, brainstem, cerebellum, and cerebrum.

[0085] (II) Then, for each brain CT image, the corresponding keywords are extracted from the corresponding report. For example, if the report is "high-density shadow appears in the left basal ganglia", where "left side" and "basal ganglia" appear in the orientation keywords and high-frequency position words in step I respectively, then the keywords corresponding to the brain CT image are extracted as "left side" and "basal ganglia". Keywords are extracted for each brain CT image according to the above method, and then a keyword classification network is trained based on the obtained keywords for automatically learning the keywords of the image. The keyword classification network uses Resnet34. Finally, a brain CT image is input, and the network predicts its keywords as K={k1, k2, ..., k n}, where n is the number of keywords, k i is the ith keyword.

[0086] (III) The keywords predicted by the network and the image feature a corrected by the supervised attention mechanism in step 3 are used as the input of the keyword-driven mechanism. The keyword-driven mechanism is actually a single-layer LSTM. The calculation process is as follows:

[0087] h k =LSTM k (K, a)

[0088] In order to distinguish it from the interactive recurrent network, the hidden layer vector of LSTM in the keyword mechanism is represented by h k Indicates that the LSTM in the keyword mechanism is LSTM k Indicates that K represents the keyword of the CT image predicted by the keyword classification network.

[0089] (IV) The output of the keyword-driven mechanism is used to activate the hidden layer vector h at the initial moment of the LSTM in the interactive recurrent network. t , t = 0, specifically, the output of the keyword-driven mechanism is used as the hidden layer vector h at the initial moment of the LSTM in the interactive recurrent network t , t=0, the formula is as follows:

[0090] h0=h k =LSTM k (K, a)

[0091] Step (4.2) uses the interactive recurrent network proposed in this paper to generate the final brain CT medical report. The network contains two layers of LSTM. The first layer is a word LSTM, represented by LSTM word , which is used to iteratively generate a word for each moment, and multiple words are finally combined into a sentence s m ,s m represents the mth sentence generated; the second layer is the sentence LSTM, represented by LSTM sen, which encodes the relationship between different sentences and guides the word LSTM to generate the words of the next sentence. The hidden layer vector of the two-layer LSTM is defined as follows:

[0092]

[0093] in Represents the hidden layer vector of the word LSTM at time t, where time t refers to the number of words generated. Represents the hidden layer vector of the word LSTM at the previous moment, y t-1 Represents the last moment LSTM word The generated word, a t is the image feature input at time t, The hidden layer vector of the previous sentence generated by the sentence LSTM, The hidden layer vector of the current sentence generated by the sentence LSTM; where, That is the output of the keyword-driven mechanism;

[0094] The process of word LSTM generating words is as follows:

[0095]

[0096] Here t , p t Represent the word generated at the tth moment by word LSTM and the probability of the word, W h Represents the hidden layer vector The parameter matrix of The hidden layer vector representing the word LSTM at time t. Finally, the word LSTM and sentence LSTM interact to generate the entire long paragraph of the medical report.

[0097] Step (5) defines the overall loss function and trains the brain CT image report automatic generation network:

[0098] Step (5.1) defines the loss function of the language generation module. The data of the brain CT medical report automatic generation method based on weakly supervised attention described in the present invention is the patient's brain CT image I = {I1, ..., I N} and brain CT medical report The image I is used as the input for network training and testing, and the medical report As the true label for network training, the output of the network is the predicted report Y = [y1, y2, ...y T ].in, yi represents the i-th word of a real report and a predicted report respectively, T represents the number of words in a real report and a predicted report respectively. The objective function of the final language generation module of the present invention is to maximize the similarity between the real report and the predicted report, and the calculation formula is as follows:

[0099] θ * =argmax∑logp(Y|I;θ)

[0100] where θ * represents all the network parameters after optimization, and θ represents all the network parameters to be trained in the model. The loss function of the language generation module uses cross entropy loss, which is defined as follows:

[0101]

[0102] where p t Represents the predicted probability value of the tth word generated by the model, Refers to the tth word in the real report.

[0103] Step (5.2) defines the final loss function of the method for automatically generating a brain CT medical report based on weakly supervised attention described in the present invention, which includes two parts:

[0104] Loss=LOSSlang+Lossattn

[0105] Finally, under the Adam optimizer algorithm, the network can adaptively optimize the loss values ​​of the real report and the predicted report. After training, by inputting a brain CT image, the model can automatically generate a coherent and accurate brain CT medical report. At the same time, the guided attention mechanism can accurately capture the area where the lesion is located, and its visualization is highly interpretable.

[0106] In order to verify the effectiveness of the method described in the present invention, we experimented with the proposed method on the brain CT medical report dataset and conducted a comprehensive comparative analysis with other existing methods. In the specific implementation process, we compared with three mainstream frameworks for image description, namely CNN-RNN, Soft-ATT, and Up-Down method, and reproduced two methods used for automatic generation of chest X-ray image reports, namely HRNN and MRMA as comparison methods. The experiment uses automatic evaluation indicators for natural language processing tasks such as text generation, including BLEU, METEOR, ROUGE, and CIDEr. These evaluation indicators evaluate the effectiveness of the model by calculating the similarity between the model-generated sentences and the reference sentences.

[0107] Table 1 Comparative experiment between the method proposed in the present invention and different methods

[0108]

[0109] Table 1 shows the comparison of the performance results of the proposed method and other methods on 7 language indicators. It can be seen that the performance of the early encoder-decoder framework method without attention mechanism, namely CNN-RNN, is significantly lower than that of other methods using attention mechanism such as Soft-ATT, indicating that the introduction of attention model is conducive to the model to explore key lesion areas. Our method can flexibly combine the encoder-decoder framework based on visual attention. In this paper, we use Up-Down, one of the most advanced frameworks, as the baseline method. As shown in Table 1, the results of our method are significantly better than the baseline model. Among all indicators, the CIDEr indicator has the most obvious improvement, with an improvement of 5.5, reflecting the effectiveness of the proposed method in the automatic generation of brain CT medical reports. HRNN and the method in this paper have similar language generation structures, that is, both contain recurrent networks at the sentence level and recurrent networks at the word level. The difference is that the sentence RNN of HRNN is used to generate topic vectors and stop vectors, and the word RNN generates sentences around the topic vector. However, in the task of medical report generation, due to the abstractness and ambiguity of the topic vector, the sentence RNN cannot generate long medical paragraphs well. Compared with the HRNN method, the method in this invention abandons the stop vector in the sentence RNN, uses word LSTM to control the number of generated sentences, increases the interaction between word LSTM and sentence LSTM in the language generation process, and uses a keyword-driven mechanism to activate the hidden layer states of both at the initial moment. In addition, the spatial attention mechanism in the model is weakly supervised by Grad-CAM, so our model has significant advantages over HRNN. MRMA uses a multimodal recursive model based on the attention mechanism, which performs better than HRNN, but except for the METEOR indicator, which is the same as ours, other metrics are weaker than our method.

[0110] In addition, we also conducted ablation experiments to prove the effectiveness of the two mechanisms in the method proposed in this invention, one is the weak supervision module of attention, and the other is the interactive recurrent network driven by keywords. The ablation experiment still uses Up-Down as the baseline. It is worth noting that the image input in the original Up-Down paper is two-dimensional data, which is not suitable for three-dimensional brain CT images. Therefore, we adjusted the attention mechanism in Up-Down to a hierarchical structure that includes spatial attention mechanism and sequence attention mechanism. The difference between our model and the baseline model Up-Down is that the guidance module of spatial attention and the interactive recurrent network driven by keywords are added.

[0111] Table 2 Ablation experiments of different modules of the method proposed in this invention

[0112]

[0113] As shown in Table 2, Up-Down+WGAM and Up-Down+KIRN outperform Up-Down in BLEU@1 and CIDEr, and they perform comparable to Up-Down in other metrics. Up-Down+WGAM outperforms Up-Down+KIRN in most metrics, which demonstrates the effectiveness of the weakly supervised attention mechanism. The Up-Down+WGAM+KIRN model proposed for automatic generation of brain CT medical reports takes advantage of the weakly supervised attention mechanism and keyword-driven interactive recurrent networks, achieving the best performance in all metrics.

[0114] In summary, the present invention verifies the effectiveness of the method proposed in this paper in the automatic generation of brain CT medical reports through comparative experiments and ablation experiments. This is because the hierarchical attention structure in the weakly supervised attention can capture important lesion features and important brain CT sequences in the brain CT sequence, and the weakly supervised information generated by the Grad-CAM method can effectively improve the accuracy of the spatial attention mechanism in capturing important lesion features. In addition, the keyword-driven interactive recurrent network in the language generation module also plays its unique advantages in the task of generating long paragraphs.

[0115] Figure 4 The qualitative results of our proposed method are shown. It can be seen that our method can not only improve the accuracy of the generated brain CT report generation, but also improve the interpretability of the attention mechanism. The figure shows two forms of attention maps, namely spatial attention visualization and sequence attention visualization. The first and second rows show the spatial attention visualizations generated by Up-Down and our model, where the brightness refers to the confidence of the spatial attention module for different regions of the sequence. The third row shows the sequence attention visualization, where the brightness indicates the confidence of the sequence attention module for different sequences. It can be observed that the WGAM module can correctly guide the attention model to focus on the lesion area. Compared with our method, the attention visualization generated by the spatial attention module of the baseline model Up-Down is scattered and inaccurate.

[0116] The last three rows show the real medical reports corresponding to the brain CT images, the reports generated by the Up-Down model, and the reports generated by the method of the present invention. In the first example, our model accurately generated the sentence "Large patches of high-density shadows can be seen in the right frontal lobe, surrounded by low-density edema bands... The left lateral ventricle posterior horn is compressed... Spotty low-density shadows can be seen in the left basal ganglia, and some boundaries are clear. The density of white matter around the bilateral lateral ventricles is reduced", in which the position word "frontal lobe" is consistent with the real report. In addition, our model also generated sentences such as "the left lateral ventricle posterior horn is compressed" and "Spotty low-density shadows can be seen in the left basal ganglia", which did not appear in the real report. By consulting radiologists, it was verified that the abnormalities described by the model existed in the corresponding brain CT images. This phenomenon verified the effectiveness of the weakly supervised attention mechanism and the keyword-driven interactive recurrent network module. In other examples, our model can accurately capture the lesion area and generate correct position words such as "parietal occipital lobe", "right side" and "pons". Compared with our method, the baseline model Up-Down has a poor ability to capture the lesion area due to its attention mechanism, and the generated words such as "basal ganglia", "right side" and "thalamus" are different from the real reports.

[0117] The attention visualization of the spatial attention mechanism and the sequence attention mechanism in the proposed method shows that the WGAM module can first focus on the lesion area in the sequence, and then adaptively capture the corresponding brain CT sequence with significant lesions. It is worth noting that the sequence distribution corresponding to the more significant sequence attention visualization is continuous, which is due to the continuity of the distribution of brain CT lesions in three-dimensional space.

Claims

1. A method for automatic generation of brain CT medical reports based on weakly supervised attention, Features: It includes two stages: training and prediction. The training phase includes: (1) Create and preprocess a training dataset for brain CT medical report generation to obtain standardized three-dimensional brain CT images and corresponding reports; (2) Using the brain CT image feature extractor to extract the features of the brain CT image F = [f 1 ,f 2 ,…f N ], where N represents the number of brain CT sequences in each case, and f i Represents the image features of the i-th brain CT sequence; (3) Constructing a weakly supervised attention module, which is used to extract important lesion features a from the image features of N brain CT images in sequence, including a spatial attention mechanism, a sequence attention mechanism, and a modified spatial attention mechanism: wherein the spatial attention mechanism is used to extract important lesion features in each brain CT sequence, and the mechanism has the ability to capture important features in a single sequence by assigning different weights to different positions of a single sequence and assigning higher weights to key features; the sequence attention mechanism is used to extract important sequence features in N brain CT images in sequence, and the key sequence features are assigned higher weights to participate in feature addition, so as to obtain the image features a that finally contain important lesions in the entire CT image; the modified spatial attention mechanism is used to correct inaccurate spatial attention and enhance the ability of spatial attention to extract lesion features, specifically using the gradient weighted class activation mapping Grad-CAM algorithm to generate supervision information for spatial attention, and then calculating the cross entropy loss function Loss between the spatial attention and the supervision information. attn , participates in the overall model training as part of the overall loss; (4) Construct a language model for generating a brain CT medical report. The input of the language model is the important lesion feature a extracted in step (3), and the output is the brain CT medical report. The language model is a keyword-driven interactive recurrent network. The keyword-driven interactive recurrent network consists of two parts. The first part is the keyword-driven mechanism, which is used to activate the hidden layer vector h at the initial moment of the LSTM in the interactive recurrent network. t , t=0, the input of this mechanism is the keyword K of the CT image predicted by the keyword classification network and the image feature a of the important lesion of the CT image obtained by the weakly supervised attention module. The keyword refers to the bleeding location of the brain CT image. The keyword driving mechanism is a single-layer LSTM. The calculation process is as follows: h k =LSTM k (K,a) h k Represents the hidden layer vector of LSTM in the keyword mechanism; The second part is the interactive recurrent network, which is used to generate medical reports. It contains two layers of LSTM. The first layer is the word LSTM, represented by LSTM word , which is used to iteratively generate a word for each moment, and multiple words are finally combined into a sentence s m ,s m represents the mth sentence generated; the second layer is the sentence LSTM, represented by LSTM sen , which encodes the relationship between different sentences and guides the word LSTM to generate the words of the next sentence. The hidden layer vector of the two-layer LSTM is defined as follows: in Represents the hidden layer vector of the word LSTM at time t, where time t refers to the number of words generated. Represents the hidden layer vector of the word LSTM at the previous moment, y t-1 Represents the last moment LSTM word The generated word, a t is the image feature input at time t, The hidden layer vector of the previous sentence generated by the sentence LSTM, The hidden layer vector of the current sentence generated by the sentence LSTM, s m-1 is the previous sentence predicted; among them, That is the output of the keyword-driven mechanism; The process of word LSTM generating words is as follows: Here t ,p t Represent the word generated by LSTM at the tth moment and the probability of the word, W h Represents the hidden layer vector The parameter matrix of The hidden layer vector representing the word LSTM at time t; finally, the word LSTM and sentence LSTM interactively generate the entire long paragraph of the medical report; (5) Construct the overall loss function and train it. The overall loss function consists of two parts: Loss=Loss lang +Loss attn in, Refers to y t The true label is the word in the real report; the prediction stage includes the following steps: (6) preprocessing the brain CT to be predicted to obtain a standardized three-dimensional brain CT image; (7) using the trained brain CT image feature extractor to extract features of the brain CT image to be predicted; (8) Using the trained weakly supervised attention module, important lesion features are extracted from the features of the brain CT images to be predicted; (9) Using the trained language model, a medical report of the brain CT image to be predicted is generated.

2. According to claim 1, a method for automatically generating brain CT medical reports based on weakly supervised attention, Features: The specific steps of step (1) are as follows: Step (1.1): Collect brain CT images to construct a data set. Each patient data contains the RGB matrix I generated by the brain CT image. 1 …,I N }, Brain CT report Where N represents the number of brain CT sequences in each case, I i Represents the RGB matrix of the i-th sequence, W and H represent the width and height of the sequence respectively, represents the i-th word of a report, Represents the number of words in a report; Step (1.2) divides all patient data into training set, validation set and test set; the training set is used to learn the parameters of the neural network; the validation set is used to determine the hyperparameters; and the test set is used to verify the classification effect of the neural network; Step (1.3) Data preprocessing: For brain CT images with a sequence number not exceeding 24, an interpolation algorithm is used to complete the sequence, and for brain CT images with a sequence number exceeding 24, a uniform sampling method is used to select the sequence.

3. According to claim 1, a method for automatically generating brain CT medical reports based on weakly supervised attention, Features: The feature extractor in step (2) uses the Resnet101 network, and the calculation process of the feature F extracted from the brain CT image is as follows: F = Resnet101 (I).

4. According to claim 1, a method for automatically generating brain CT medical reports based on weakly supervised attention, Features: The working process of the weakly supervised attention module in step (3) is as follows: Step (3.1) constructs the spatial attention mechanism in the weakly supervised attention module to extract important lesion features in each CT sequence. Specifically, the mechanism extracts the important lesion features in each brain CT sequence by i Calculate its spatial attention matrix The calculation process is as follows: e i =φ(h,f i ) φ is a feedforward neural network used to automatically learn the relationship matrix between its two internal parameters, h is the hidden layer vector inside the LSTM used to generate medical reports, and finally the spatial attention matrix is ​​obtained is a two-dimensional matrix, whose matrix values ​​correspond to the weights assigned by the attention mechanism to different spatial positions in a single sequence. Finally, the brain CT image feature is obtained by weighted sum of weight features: V = [v 1 ,v 2 ,…v N ], where v i Represents the sequence feature of the i-th image containing important lesion features, and its calculation formula is: Step (3.2) constructs the sequence attention mechanism in the weakly supervised attention module to extract important sequence features from N sequences of brain CT images. Specifically, the mechanism calculates the sequence attention weight a for all sequence features of each brain CT image. Frame , where a Frame is a one-dimensional vector, whose vector value corresponds to the weight given by the attention mechanism to different sequences. Finally, the important sequence feature a in the N sequences of brain CT images is obtained by weighted sum of weight features. The calculation formula is: Step (3.3) uses weakly supervised attention to correct spatial attention. The specific steps are as follows: (I) For a single sequence, the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm is used to generate the corresponding class activation map. The color of the important lesion area in the class activation map is marked, and then the image segmentation tool is used to draw the detection box R of the lesion in the single image. i ; (II) The generated detection box R i Transformed into supervision information β for spatial attention i , supervision information β i is a two-dimensional matrix, and the transformation process of any element is as follows: Where (l,c) represents the lth row and cth column of the supervision matrix. When the position is in the lesion detection frame, the matrix value of the position is set to 1, otherwise it is set to 0; (III) After generating the supervision information of spatial attention, the cross entropy loss function of spatial attention and supervision information is calculated and used as part of the overall loss to participate in the overall model training. At this time, the incorrect spatial attention matrix generated by spatial attention Supervised information β i Correction can be made to extract important lesion features of a sequence. The loss function is defined as follows:

5. According to claim 1, a method for automatically generating brain CT medical reports based on weakly supervised attention, Features: The method for generating keywords of the keyword-driven mechanism is as follows: (I) define two types of keywords: one is the three directional keywords of left side, right side and bilateral side, which are used to indicate which side of the brain midline the lesion is located on; the other is the high-frequency location words that appear in brain CT medical reports, including 11 locations: frontal lobe, temporal lobe, parietal lobe, occipital lobe, semiovale, lateral ventricle, basal ganglia, thalamus, brainstem, cerebellum, and cerebrum; (II) For each brain CT image, two types of keywords are extracted from the corresponding report. A keyword classification network is trained based on the obtained keywords to automatically learn the keywords of the image. Finally, the trained keyword classification network is used to predict the keywords of the brain CT image. The network predicts that the keywords are K = {k 1 ,k 2 ,…,k n }, where n is the number of keywords, k i is the ith keyword, and the keyword classification network adopts Resnet34.