A method for generating brain CT medical reports based on hierarchical recurrent neural network decoding

Abstract

The present invention discloses a brain CT medical report generation method based on hierarchical recurrent neural network decoding. First, a brain CT image and corresponding medical report data are obtained and preprocessed; a feature extractor is constructed to complete the encoding of the brain CT image data to obtain the encoding feature #imgabs0# and the fault block visual feature #imgabs1#; an orientation keyword predictor is constructed to extract the orientation keyword semantic feature F of the brain CT image data I. s ; Construct a hierarchical recurrent neural network language model, which uses #imgabs2# and F s Perform hierarchical decoding and generate medical reports sentence by sentence; train and optimize the model; preprocess the brain CT to be predicted; use location keywords to extract the coding features and fault block visual features of the brain CT to be predicted; use location keywords to extract semantic features; the language model uses coding features, fault block visual features and semantic features to generate the predicted medical report sentence by sentence.

Description

Technical Field

[0001] The present invention uses relevant technical means in the two fields of computer vision and natural language processing, and designs a medical report generation method based on hierarchical recurrent neural network decoding for the task of automatic report generation in the field of medical image analysis. Background Art

[0002] The task goal of automatic medical report generation is to input a set of brain CT images containing spatial sequence relationships, and the computer can automatically generate several sentences describing the content of the images as the corresponding medical report. This requires the computer to have the ability to understand brain CT images and the ability to organize the language to express the image content. It is currently a research hotspot in medical image analysis.

[0003] With the rise of various technologies in the field of artificial intelligence, computer-assisted doctors in medical image analysis has become a focus of increasing attention. Machine learning technology has played an important role in tasks such as classification and segmentation. However, traditional machine learning technology has shown limited capabilities when processing large-scale annotated or unlabeled data. Deep learning is one of the most important breakthroughs in the field of artificial intelligence in the past decade. It has achieved great success in different tasks of computer vision and image processing. Methods and models for medical image analysis have also benefited from the powerful representation learning capabilities of deep learning technology. Not only has the relevant research literature continued to grow, but substantial progress has also been made in practical application areas such as chest X-ray image diagnostic report generation.

[0004] Early work on automatic generation of medical reports was mainly based on models in the field of image description. At present, there are three main research methods in the field of image description: template-based methods, retrieval-based methods, and neural network encoding and decoding methods. Among them, the neural network encoding and decoding method is the most commonly used method in the field of image description and is widely used in fields such as intelligent medical care. The basic idea of ​​the encoder-decoder based image description method is to first use the encoder to extract the visual features of the image, and then use the decoder to establish a mapping relationship from vision to language to generate the image description. Compared with the first two methods, this method does not require detailed image annotation or a large data set. The generated image description is natural and has become the mainstream model framework in the field of image description. Compared with image description, the brain CT medical report generation task has certain uniqueness in the decoding stage: the corresponding medical report is a long text sequence with a fixed structure and more description subjects. Summary of the invention

[0005] In order to combine the data characteristics of medical reports and fully mine the detailed feature information of brain CT coding, the present invention proposes a brain CT medical report generation method based on hierarchical recurrent neural network decoding (Hierarchy Long Short-Term Memory Decoder, HLSTMD) based on the relevant deep learning model in the field of image description, which effectively mines the visual features matching different keywords in brain CT coding, improves the language performance index of the final generated medical report, and generates a higher quality medical report. The medical report generation method based on hierarchical recurrent neural network decoding mainly consists of an encoder and a decoder, wherein the encoder part consists of a feature extractor, and the decoder consists of a key word predictor (Key Word Predictor, KWP) and a hierarchical recurrent neural network language decoding model (Topic To Sentence Model, TTSM).

[0006] The technical solution adopted by the present invention is a method for generating a brain CT medical report based on hierarchical recurrent neural network decoding, which comprises the following steps:

[0007] Step (1) prepares and preprocesses a training data set for generating a brain CT medical report to obtain a standardized three-dimensional brain CT image and corresponding text information, the text information including keyword text Report text

[0008] Step (1.1) collects brain CT images to construct a data set. Each patient data includes an RGB image I = {I1…,I n}, Keyword text Brain CT report text Where n represents the number of brain CT sequences in each case, I i represents the RGB image of the i-th sequence, W and H represent the width and height of the sequence respectively, represents the i-th word in the keyword text, Represents the number of keywords in a report. represents the i-th word of the report text, Represents the number of words in a report.

[0009] Step (1.2) divides all patient data into training set, validation set and test set. The training set is used to learn the parameters of the neural network; the validation set is used to determine the hyperparameters; and the test set is used to verify the classification effect of the neural network.

[0010] Step (1.3) Data preprocessing: For brain CT images with a sequence number not exceeding 24, an interpolation algorithm is used to complete the sequence, and for brain CT images with a sequence number exceeding 24, a uniform sampling method is used to select the sequence.

[0011] Step (2) uses a brain CT image feature extractor to complete the encoding of the brain CT image data I. The feature extractor uses the encoder in the patent "A method for generating a brain CT medical report based on hierarchical self-attention sequence encoding" to extract the three-dimensional encoding features of the brain CT image. And the visual characteristics of the fault block

[0012] Step (3) constructs a location keyword predictor for binary classification of left and right sides of brain CT images, and then uses the binary classification information to assist in generating the final medical report. The location keyword predictor is implemented by a multi-layer perception with only one hidden layer. The number of neurons in its output layer is 2, that is, a binary classification of left and right semantic labels is performed. The left and right semantic information F s The value of is the value of the hidden layer neuron of the multilayer perceptron. 3D brain CT coding features The semantic feature containing the left and right keyword labels is obtained through the positional keyword predictor Where n represents the number of brain CT images of a case after preprocessing.

[0013] Step (4) constructs a hierarchical recurrent neural network language decoding model, hereinafter referred to as the language model, which is used to generate brain CT medical reports. The language model is divided into two parts: the fusion attention model (FAM) and the hierarchical decoding module. Among them, the fusion attention module is used to fuse the semantic features with the visual features. The fused semantic features are used to activate the LSTM initial hidden layer vector in the hierarchical decoding module, and the fused visual features guide the LSTM to generate words with emphasis. In the training stage, the input of the language model is the visual features. Semantic feature F s , keyword text And the report text The output value is the loss function value in step (5), which is used to optimize the performance of the model.

[0014] Step (4.1) guides the construction of the attention model, which is used in step (4.2). The attention model mines the associated information of a feature tensor in another related feature tensor space. The attention model can be further divided into the visual guided attention model (VGA) and the semantic guided attention model (SGA). The input of the query tensor in the visual guided attention model VGA is The input of keyword tensor and value tensor is F s , the new features calculated at this time are guided by visual information, so that the semantic features are reconstructed in the visual feature space; the input of the query tensor in the semantic guided attention model SGA is F s , the input of keyword tensor and value tensor is At this time, the calculated new features are guided by semantic information, so that the visual features are reconstructed in the semantic feature space.

[0015] Step (4.2) Construction of the fused attention module (FAM). The fused attention module is implemented by stacking two guided attention models and linear transformation, residual connection, and normalization operations. The fused attention module is used in step (4.3). When the input is the semantic feature F s With visual features The output is the visual fusion feature F v′ And the semantic fusion feature F s′ ; When the input is a semantic feature (obtained by T-LSTM in step (4.3)) and visual features F v′ , the output is the visual fusion feature F v″ And the semantic fusion feature F s″ .

[0016] Step (4.3) Construction of hierarchical decoding module. The hierarchical decoding module consists of a keyword recurrent neural network T-LSTM and a description sentence recurrent neural network S-LSTM.

[0017] The characteristics of T-LSTM are:

[0018] 1) First, the semantic feature F is integrated by the fusion attention model FAM s With visual features Fusion is performed to obtain the reconstructed feature F v′ 、F s′ .

[0019] 2) The T-LSTM hidden layer and the first recurrent unit use the semantic fusion feature F s′ Initialize, visual fusion feature Fv′ Perform visual attention calculation with the input word value so that T-LSTM pays attention to the corresponding visual features when generating the corresponding word value.

[0020] 3) Hidden layer state of T-LSTM at time t It consists of two parts. One is the hidden layer state output by T-LSTM at time t-1. The other is the hidden layer state returned by the description sentence generation recurrent neural network S-LSTM after generating the tth description sentence

[0021] 4) The overall input of T-LSTM is the semantic feature F s , visual features and keyword text words The output is the predicted value of the keyword text word X = {x1,…,x t}, x t Represent the input word value and output word prediction value at time t respectively.

[0022] The characteristics of S-LSTM are:

[0023] 1) First, the keyword x t The keyword semantic features are obtained through linear changes and dimension expansion Then the fusion attention model transforms the semantic features With visual features Fusion is performed to obtain the reconstructed feature F v″ 、F s″ .

[0024] 2) The S-LSTM hidden layer and the first recurrent unit use the semantic fusion feature F s″ Initialize, visual fusion feature F v″ Visual attention calculation is performed with the input word value to achieve visual semantic association matching between brain CT tomographic block features and short sentences describing detailed lesions.

[0025] 3) Hidden layer state of S-LSTM at time t The hidden state returned by the previous description sentence And the semantic reconstruction feature F at time t s″ constitute.

[0026] 4) The overall input of S-LSTM is semantic features Visual features And describe the short words The output is the predicted value Y={y1,…,y t}, yt Represent the input word value and output word prediction value at time t respectively.

[0027] Step (5) constructs the overall loss function and trains it. The overall loss function consists of two parts:

[0028] Loss=Loss sentence +Loss topic

[0029] in, Point to x t The true label of is the word in the keyword text, Refers to y t The true label of the word in the report text, p t Refers to the corresponding word x t or t The predicted probability of .

[0030] The forecasting phase includes the following steps:

[0031] Step (6) preprocesses the brain CT to be predicted to obtain a standardized three-dimensional brain CT image and corresponding keyword text information and a medical report.

[0032] Step (7) uses the trained feature extractor to extract the three-dimensional coding features and the visual features of the to-be-predicted brain CT image.

[0033] Step (8) uses the trained directional keyword predictor to generate left and right semantic features.

[0034] Step (9) utilizes the trained language model to fuse the semantic features and visual features and generate a medical report of the brain CT image to be predicted.

[0035] Beneficial Effects

[0036] Compared with image description, the brain CT medical report generation task has certain uniqueness in the decoding stage: the corresponding medical report is a long text sequence with fixed structure and more description subjects. The model proposed in this method can generate corresponding medical report description sentences from different visual spaces while combining the text data characteristics of medical reports, and generate higher-quality brain CT medical reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 : Feature Extractor

[0038] Figure 2(a): Self-attention network structure

[0039] Figure 2(b): Guided attention network structure

[0040] Figure 3 : Fusion attention model

[0041] Figure 4 :Sentence decoding model based on hierarchical recurrent neural network

[0042] Figure 5 : Generated report comparison example DETAILED DESCRIPTION

[0043] The following is an example of 492 cases of data provided by Peking University Third Medical College to illustrate the specific implementation steps of the present invention:

[0044] Step (1) Obtain brain CT images and corresponding keywords and report data and preprocess them:

[0045] Step (1.1) collects brain CT images to construct a data set, which contains a total of 492 brain CT images and corresponding reports from which sensitive patient information has been deleted. The image data of each patient includes multiple CT sequences and a corresponding report text. The original brain CT medical image is in dicom format, which is converted into three scales of observation views commonly used by doctors as three channel values ​​of RGB three-channel color images, and the boundary noise CT value -2000 of the image is removed, and finally the brain CT image data in PNG format is obtained. I={I1…,I N}, Where N represents the number of CT slices in each case, and W and H represent the width and height of each sequence, respectively. Then the corresponding text is preprocessed: remove redundant punctuation marks in the report text data; unify the professional terms; split it into several descriptive sentences with "," as the separator and perform Chinese word segmentation through the jieba word segmentation tool to obtain a description dictionary with a size of 244, and finally obtain the medical report text Finally, the most frequent words in each short sentence in the report text are combined into keyword text represents the i-th word in the keyword text, Represents the number of keywords in a report. represents the i-th word of the report text, Represents the number of words in a report.

[0046] Step (1.2) randomly divides the data set into training set, validation set and test set in a ratio of 10:1:1. The data ratio is 410:41:41.

[0047] Step (1.3) Data preprocessing: 24 brain CT images were used for each case. Images less than 24 were supplemented by interpolation algorithm, and images more than 24 were selected by uniform sampling method. Each image was denoised and normalized, and then resized to 512*512.

[0048] Step (2) uses a brain CT image feature extractor to complete the encoding of 492 brain CT image data I. The feature extractor uses the encoder in the patent "A method for generating a brain CT medical report based on hierarchical self-attention sequence encoding" to extract the three-dimensional encoding features of the brain CT image. And the visual characteristics of the fault block The specific network structure is as follows Figure 1 As shown, Corresponding to Figure 1 The SA Feature and NSA Feature in the feature extractor are expressed as follows:

[0049]

[0050] Step (3) constructs a location keyword predictor for binary classification of left and right sides of brain CT images, and then uses the binary classification information to assist in generating the final medical report. The location keyword predictor is implemented by a multi-layer perception with only one hidden layer. The number of neurons in its output layer is 2, that is, a binary classification of left and right semantic labels is performed. The left and right semantic information F s The value of is the value of the hidden layer neuron of the multilayer perceptron. 3D brain CT coding features The semantic feature containing the left and right keyword labels is obtained through the positional keyword predictor Where n represents the number of brain CT images of a case after preprocessing. Here we use 24 brain CT images, so n=24.

[0051] Step (4) constructs a hierarchical recurrent neural network language decoding model, hereinafter referred to as the language model, which is used to generate brain CT medical reports. The language model is divided into two parts: the fusion attention model (FAM) and the hierarchical decoding module. Among them, the fusion attention module is used to fuse the semantic features with the visual features. The fused semantic features are used to activate the LSTM initial hidden layer vector in the hierarchical decoding module, and the fused visual features guide the LSTM to generate words with emphasis. In the training stage, the input of the language model is the visual features. Semantic feature F s , keyword text And the report text The output value is the loss function value in step (5), which is used to optimize the performance of the model.

[0052] Step (4.1) guides the construction of the attention model. The guided attention model can mine the associated information of a feature tensor in another related feature tensor space. The present invention constructs a fusion attention model by guiding the attention model, so that the language model pays attention to the associated visual detail features in the brain CT when generating each descriptive short sentence. The specific structure of the self-attention model is shown in Figure 2 (a). First, the same feature vector undergoes a linear change to obtain the query tensor Q, the keyword tensor K and the value tensor V, and then the matrix multiplication, scaling, Softmax activation function and matrix multiplication of Q and K and V are successively performed to finally calculate the attention feature that takes into account the vector's own dependency. The guided attention is improved from the self-attention, as shown in Figure 2 (b). The guided attention replaces the query tensor with other related tensors. When the replaced tensor is a visual feature, and the value tensor and the keyword tensor are semantic features, the guided attention at this time is a visual guided attention model VGA. On the contrary, when the query tensor is a semantic feature, and the value tensor and the keyword tensor are visual features, the attention model is a semantic guided attention model SGA.

[0053] Step (4.2) constructs the fusion attention model FAM. The fusion attention model FAM is mainly implemented by stacking two guided attention models. For example, FAM is used to calculate the fusion semantic feature F s′ The specific process is as follows Figure 3 As shown. The input of the model includes visual features Semantic feature F s , the feature dimensions are and Where N v is the number of fault blocks, N s is the number of labels on the left and right sides. First, the visual features are mined through the SGA model The semantic feature F s The enhanced features are then used as the visual information of VGA to guide the semantic features F s Finally, the semantic features reconstructed by the VGA and SGA models are calculated through linear transformation, residual connection, and normalization operations to obtain the final semantic fusion feature F s′ , the forward propagation formula of the FAM model of the whole process is as follows:

[0054]

[0055]

[0056] Similar to the above fusion process, the visual features are used as the value tensor body to generate visual fusion features. The forward propagation formula of the FAM model is as follows:

[0057]

[0058]

[0059] Where W is the parameter matrix of the neural network linear transformation, Norm is the layer normalization function (LayerNormalization), is the candidate value of semantic fusion feature, F s′ This is the final semantic fusion feature, which will then be used to initialize our keyword generation recurrent neural network. is the candidate value of visual fusion feature, F v′ The final visual fusion feature will be used to calculate the visual attention mechanism in the keyword generation recurrent neural network.

[0060] Step (4.3) Construction of hierarchical decoding module. The hierarchical decoding module is composed of a keyword recurrent neural network T-LSTM and a description short sentence recurrent neural network S-LSTM. The specific structure is as follows: Figure 4 As shown, T-LSTM is used to generate keywords in the description sentences, and L-LSTM generates corresponding description sentences according to the corresponding keywords.

[0061] The characteristics of T-LSTM are:

[0062] 1) First, the semantic feature F is integrated by the fusion attention model FAM s With visual features Fusion is performed to obtain the reconstructed feature F v′ 、F s′ .

[0063] 2) The T-LSTM hidden layer and the first recurrent unit use the semantic fusion feature F s′ Initialize, visual fusion feature F v′ Perform visual attention calculation with the input word value so that T-LSTM pays attention to the corresponding visual features when generating the corresponding word value.

[0064] 3) Hidden layer state of T-LSTM at time t It consists of two parts. One is the hidden layer state output by T-LSTM at time t-1. The other is the hidden layer state returned by the description sentence generation recurrent neural network S-LSTM after generating the tth description sentence (like Figure 4 The specific formula is as follows:

[0065]

[0066] Where α is a hyperparameter with a value of [0,1]. We can adjust the value of the hyperparameter α to make T-LSTM retain or discard the description sentence information and keyword information of the previous moment to varying degrees when generating each keyword. In the actual experimental process, we added a bias to the forget gate of the recurrent neural network and initialized it to 1 to reduce the forgetting ability of the model in the early stage of training, avoiding the problem of output explosion or disappearance at the beginning of training. The specific formula is as follows, where W is the parameter matrix f of the linear transformation of the neural network t is the forget gate vector, h t-1 The hidden layer state vector at the previous moment, x t is the input vector at time t, and b is the bias:

[0067] f t =(W fx x t +W fh h t-1 +b)

[0068] 4) The overall input of T-LSTM is the semantic feature F s , visual features and keyword text words The output is the predicted value of the keyword text word X = {x1,…,x t}, x t Represent the input word value and output word prediction value at time t respectively.

[0069] The characteristics of S-LSTM are:

[0070] 1) First, the keyword x t The keyword semantic features are obtained through linear changes and dimension expansion Then the fusion attention model transforms the semantic features With visual features Fusion is performed to obtain the reconstructed feature F v″ 、F s″ ;

[0071] 2) The S-LSTM hidden layer and the first recurrent unit use the semantic fusion feature F s″ Initialize, visual fusion feature F v″ Perform visual attention calculation with the input word value to achieve visual semantic association matching between brain CT tomographic block features and short sentences describing detailed lesions;

[0072] 3) Hidden layer state of S-LSTM at time t The hidden state returned by the previous description sentence And the semantic reconstruction features at time t Composition (such as Figure 4 Considering that the medical report is to concatenate each short sentence into the final medical report, during the initialization process, we also introduce the memory information of S-LSTM when generating the previous sentence. The specific formula is as follows:

[0073]

[0074] β is a hyperparameter with a value of [0, 1]. We can adjust the value of the hyperparameter β to make S-LSTM retain or discard the description short sentence and keyword information generated at the previous moment to varying degrees when generating each short sentence.

[0075] 4) The overall input of S-LSTM is semantic features Visual features And describe the short words The output is the predicted value Y={y1,…,y t}, y t Represent the input word value and output word prediction value at time t respectively.

[0076] Step (5) defines the overall loss function and trains the brain CT image report automatic generation network. The overall loss function includes two parts:

[0077] loss = Loss sentence +Loss topic

[0078] in, Point to x t The true label of is the word in the keyword text, Refers to y t The true label of the word in the report text, p t Refers to the corresponding word x t or t Finally, under the Adam optimizer algorithm, the network can adaptively optimize the loss values ​​of the real report and the predicted report. After training, a brain CT image is input, and the model can automatically generate a coherent and accurate brain CT medical report. At the same time, using the features reconstructed by the fusion attention mechanism, the model can pay attention to different visual focuses when generating each descriptive sentence, which reflects the advantage of the present invention in generating long text sequences.

[0079] In order to verify the feasibility of this method, we conducted quantitative experimental analysis and qualitative result analysis on the proposed method. The generated report results were evaluated by four evaluation indicators: BLEU, METEOR, ROUGE-L and CIDEr. These four evaluation indicators have been widely used in the quantitative evaluation of generated sentences in the fields of image description and natural language processing. Figure 5 Examples of the present invention are shown, including brain CT images, sample reports generated by a comparative baseline method, and sample reports generated by the method.

[0080] Table 1 Module ablation experiment

[0081]

[0082] First, we conducted an ablation experiment between modules. The first row of Table 1 uses a single-layer LSTM to decode medical reports, which is the baseline model. The LSTM_S method in the second row uses the binary semantic information F s′ Initialize the memory unit of the decoder part and use it as the start word information of the medical report in the test phase. In the LSTM_FAM method in the third row, the semantic information of the binary classification is replaced by the fusion feature of itself and the visual features of the fault block through the FAM module. The HLSTM method below the dividing line uses the hierarchical T-LSTM and S-LSTM as the decoder of the medical report. The fifth and sixth rows are HLSTM using the semantic information F of the directional keyword respectively. s′ and F s″ Initialize the loop unit

[0083] From Table 1, we can see that simply increasing the number of layers of the recurrent neural network does not necessarily improve the performance of the final medical report generation. Since the training sample is too small, adding a layer of neural network to train the main description keywords will bring more training parameters. The language evaluation index value in the final test is lower, indicating that increasing the parameters will bring further overfitting problems in this data set. However, through the newly added recurrent neural network, we can perform a task decomposition on the generation of the final medical report, so as to obtain the generation process of each descriptive sentence and the capture of the corresponding visual and semantic information, and through the fusion of multimodal information, the language evaluation index value of the model has achieved results that are not inferior to or even greater than that of a single-layer recurrent neural network. Specifically, whether it is a single-layer LSTM baseline method or an HLSTM method, the language model is initialized using binary semantic information, and the left and right labels generated by the directional keyword predictor are used as the start words of the first medical report short sentence in the test phase. The performance of the final model has been slightly improved, among which the BLEU-1 and CIDEr index values ​​have been significantly improved. In addition, the FAM module is used to convert the left and right keywords F in HLSTM into the starting words of the first medical report short sentence. s′ and position keyword F s″ They are fused with visual coding features and used to enhance T-LSTM and S-LSTM, which greatly improves the performance of the final medical report generation. Finally, the HLSTM_GT experiment no longer uses the semantic information on the left and right sides and the semantic information output by T-LSTM, but directly uses the real labels of keywords for the generation of FAM fusion information. The results show that when the quality of keyword label generation is very high, the method proposed in the present invention can effectively improve the performance indicators of report description sentences.

[0084] Table 2 Ablation experiment results of FAM model

[0085]

[0086] Then, the experiments of different stacking methods of FAM model were carried out to explore the effectiveness of FAM model in a certain hypothesis space. v′ (F v″ ), the semantic fusion (reconstruction) feature is F s′ (F s″ ), SA_* represents the calculation of self-attention for the value tensor of the corresponding guided attention. Above the dividing line, the guided attention is used once as the corresponding fusion feature, and below it, the guided attention is used twice as the corresponding fusion feature. "——" indicates that the fusion feature of this model is not used.

[0087] As can be seen from the top of the dividing line in Table 2, using the same semantic guided attention model, that is, the same SGA and SA_SGA models, the final model has a similar ability to fit the data set, and its evaluation index value fluctuates slightly, which shows that the model's discriminability for visual fusion features is significantly stronger than that for semantic fusion features. In addition, the use of guided attention can effectively improve the quality of medical report generation. More specifically, when the number of guided attention stacking is the same, the calculation of self-attention on SGA can improve the performance of the model. There is a spatial sequence relationship between visual information. Using self-attention to reconstruct the visual feature space before using SGA can effectively play the characteristic advantages of semantic information. At the same time, through the cross-fusion of fault block information, the label detail features contained in the semantic information can be further learned, so that the model can enhance the degree of association with relevant visual information when generating a description sentence with the corresponding label as the description subject, thereby improving the quality of the medical report finally generated by the model.

[0088] Finally, we conduct a qualitative analysis of the model’s descriptive capabilities. Figure 5 The following is a comparison example of the medical reports generated by this method and the baseline method. From the generated medical report text, the single-layer LSTM method generates fewer description subjects in the medical report, and the described subjects are generally adjacent brain tissue structures (near the location of the tomographic image); the HLSTMD method generates a significantly larger number of description subjects in the medical report, describes more symptoms, and generates more diverse reports. In addition, the detailed information of the description subject can be analyzed to a certain extent. For example, a detailed description of "right maxillary sinus submucosal cyst" is generated for the case of sinusitis.

[0089] In general, whether it is the quantitative evaluation of language performance or the medical report examples presented in the qualitative analysis, the automatic generation framework of medical reports proposed in the present invention has achieved good results and has good application prospects in future practical applications.

Claims

1. A method for generating a brain CT medical report based on hierarchical recurrent neural network decoding, characterized in that: It includes two stages: training and prediction. The training phase includes: Step (1) prepares and preprocesses a training data set for generating a brain CT medical report to obtain a standardized three-dimensional brain CT image and corresponding text information, the text information including keyword text Report text Step (2) uses a brain CT image feature extractor to complete the encoding of the brain CT image data I to obtain a three-dimensional brain CT encoding feature and visual features of brain CT tomographic blocks Step (3) constructs a directional keyword predictor for left and right side binary classification of brain CT images, and then uses the binary classification information to assist in generating the final medical report; 3D brain CT coding features The semantic feature containing the left and right keyword labels is obtained through the positional keyword predictor Where n represents the number of brain CT images of a case after preprocessing; Step (4) constructs a hierarchical recurrent neural network language decoding model, hereinafter referred to as the language model, which is used to generate brain CT medical reports; the language model is divided into two parts: a fusion attention module and a hierarchical decoding module, wherein the fusion attention module is used to fuse semantic features with visual features, the fused semantic features are used to activate the LSTM initial hidden layer vector in the hierarchical decoding module, and the fused visual features guide the LSTM to generate words with emphasis; in the training stage, the input of the language model is the visual features Semantic feature F s , keyword text And the report text The output value is the loss function value in step (5), which is used to optimize the performance of the model; Step (5) constructs the overall loss function and trains it. The overall loss function consists of two parts: Loss=Loss sentence +Loss topic in, p t Refers to the corresponding x t or t The predicted probability of The prediction phase includes the following steps: Step (6) preprocessing the brain CT to be predicted to obtain a standardized three-dimensional brain CT image and corresponding keyword text information and a medical report; Step (7) using the trained feature extractor to extract the three-dimensional coding features and the visual features of the to-be-predicted brain CT image; Step (8) using the trained directional keyword predictor to generate left and right semantic features; Step (9) using the trained language model, fusing the semantic features and the visual features and generating a medical report of the brain CT image to be predicted; The location keyword predictor in step (3) is implemented by a multi-layer perception with only one hidden layer. The number of neurons in its output layer is 2, that is, a binary classification of the left and right semantic labels is performed. The semantic feature F s The value of is the value of the hidden layer neuron of the multilayer perceptron; The specific steps in step (4) are: Step (4.1) guides the construction of the attention model, which is used in step (4.2); the attention model is used to mine the associated information of a feature tensor in another related feature tensor space; the attention model can be further divided into a visual guided attention model VGA and a semantic guided attention model SGA. The input of the query tensor in the visual guided attention model VGA is The input of keyword tensor and value tensor is F s , the new features calculated at this time are guided by visual information, so that the semantic features are reconstructed in the visual feature space; the input of the query tensor in the semantic guided attention model SGA is F s , the input of keyword tensor and value tensor is At this time, the calculated new features are guided by semantic information, so that the visual features are reconstructed in the semantic feature space; Step (4.2) Construction of the fused attention module (FAM): The fused attention module is implemented by stacking the guided attention model twice and linear transformation, residual connection, and normalization operations; the fused attention module is used in step (4.3) when the input is the semantic feature F s With visual features The output is the visual fusion feature F v′ And the semantic fusion feature F s′ , the specific formula is as follows: Among them, W is the parameter matrix of the linear transformation of the neural network, Norm is the layer normalization function, and Respectively represent the candidate values ​​of visual fusion features and the candidate values ​​of semantic fusion features; Step (4.3) Construction of hierarchical decoding module: The hierarchical decoding module is composed of a keyword recurrent neural network T-LSTM and a description short sentence recurrent neural network S-LSTM; The characteristics of T-LSTM are: 1) First, the semantic feature F is integrated by the fusion attention model FAM s With visual features Fusion is performed to obtain the visual fusion feature F v′ , semantic fusion feature F s′ ; 2) The T-LSTM hidden layer and the first recurrent unit use the semantic fusion feature F s′ Initialize, visual fusion feature F v′ Perform visual attention calculation with the input word value, so that T-LSTM pays attention to the corresponding visual features when generating the corresponding word value; 3) Hidden layer state of T-LSTM at time t It consists of two parts. One is the hidden layer state output by T-LSTM at time t-1. The other is the hidden layer state returned by the description statement generation recurrent neural network S-LSTM after the description statement at time t The specific formula is as follows: Where α is a hyperparameter, and its value is [0,1]; 4) The overall input of T-LSTM is the semantic feature F s , visual features and keyword text words The output is the predicted value of the keyword text word X = {x1,…,x t }, x t Respectively represent the input word value and output word prediction value at time t; The characteristics of S-LSTM are: 1) First, x t The keyword semantic features are obtained through linear changes and dimension expansion Then the fusion attention model transforms the semantic features With visual features Fusion is performed to obtain the reconstructed visual fusion feature F v″ , reconstructed semantic fusion feature F s″ ; 2) The S-LSTM hidden layer and the first recurrent unit use the reconstructed semantic fusion feature F s″ Initialize and reconstruct the visual fusion feature F v″ Perform visual attention calculation with the input word value to achieve visual semantic association matching between brain CT tomographic block features and short sentences describing detailed lesions; 3) Hidden layer state of S-LSTM at time t The hidden state returned by the previous description sentence and time t The specific formula is as follows: Where β is a hyperparameter, and its value is [0, 1]; 4) The overall input of S-LSTM is keyword semantic features Visual Features And describe the short words The output is the predicted value Y={y1,…,y t }, y t Represent the input word value and output word prediction value at time t respectively.

Images