An Automatic Method for Generating Brain CT Medical Reports Based on Hierarchical Attention Based on Co-occurrence Relationships

Abstract

This invention discloses an automatic generation method for brain CT medical reports based on co-occurrence relation hierarchical attention. The method preprocesses the brain CT dataset and establishes a vocabulary; constructs a feature extractor for brain CT images to extract visual features; and builds a co-occurrence relation semantic attention module to extract semantic attention features of common medical terms in brain CT images, which includes a word embedding layer and a semantic attention mechanism. A topic vector-guided visual attention module is also constructed, where topic vectors fuse semantic information from common and rare medical terms to fully express sentence-level medical terminology topics. These topics then guide the visual attention mechanism to capture important lesion region features. This method combines the co-occurrence relationships between common medical terms to infer missing semantic information, thereby extracting richer semantic attention features. This hierarchical collaboration improves the accuracy and diversity of the generated brain CT medical reports.

Description

Technical Field

[0001] This invention relates to three fields: medical imaging, computer vision, and natural language processing. Targeting the goal of automatically generating brain CT medical reports, it designs a method for automatically generating brain CT medical reports based on hierarchical attention according to co-occurrence relationships. Background Technology

[0002] Medical imaging technology plays a crucial role in clinical diagnosis today, with radiologists analyzing images to make clinical diagnoses and write reports. In densely populated countries and regions, radiologists may need to read a large number of medical images daily and write reports based on their findings. This process consumes a significant portion of their energy and time, severely impacting their work efficiency.

[0003] Among numerous medical imaging techniques, cranial imaging (brain CT) is used to diagnose various cerebrovascular diseases, such as cerebral hemorrhage, cerebral infarction, subarachnoid hemorrhage, and intracranial hematoma. Rapid and accurate diagnosis is crucial for clinical treatment. Therefore, assisting doctors in more efficiently identifying lesions and writing medical reports has significant research and application value. The automatic generation of brain CT medical reports is of great importance in improving doctors' image interpretation efficiency and assisting them in providing more comprehensive and accurate clinical diagnoses.

[0004] Medical report generation is an emerging research area in the fields of medical imaging and computer technology in recent years. Its goal is to automatically generate coherent and logically consistent medical reports from a set of medical images. Currently, most research in this field focuses on methods for automatically generating medical reports from chest X-ray images, with limited research on the automatic generation of medical reports from brain CT images. Compared to other imaging data, brain CT images are sequential data, meaning they contain multiple consecutive images. Brain CT images are three-dimensional, and lesions exhibit both continuity and sparsity; the report content is complex, containing a large number of specialized medical terms and exhibiting greater unstructured nature.

[0005] Most existing methods for automatically generating medical reports employ the encoder-decoder framework from the field of natural image description. This framework uses convolutional neural networks such as ResNet as the encoder to extract visual features from the input image and recurrent neural networks such as RNNs and LSTMs as the decoder to generate corresponding sentences. To further extract key visual features, an attention mechanism has been introduced into the encoder-decoder framework. The attention mechanism originates from research on the human brain's visual system. Cognitive science research shows that due to the bottleneck of information processing in the nervous system, the human brain selectively focuses on the important parts of all information while ignoring other useless information. Currently, the attention mechanism is widely used in many image processing-related fields. This mechanism assigns different weights to different features, allowing the model to selectively focus on the important parts of the features while ignoring other visible information. Introducing an attention mechanism into medical report generation can mimic a doctor's experience in reading images, focusing on the lesion areas in medical images, thereby making the generated medical reports more accurate.

[0006] Existing attention-based methods for automatically generating medical reports neglect the semantic correlations between medical terms in brain CT scans. For example, some common medical terms may co-occur; low-density lesions in the basal ganglia are often accompanied by low-density edema. Brain CT images typically contain rich medical terminology, some of which may be closely related. However, existing work has ignored the potential co-occurrence dependencies between common medical terms and lacks attention to rare medical terms, such as orbital hollowing and arachnoid cysts. This severely limits the accuracy and diversity of the generated brain CT medical reports. To address these issues, this invention attempts to mine potential pathological information and extract richer semantic information by analyzing the co-occurrence relationships between common medical terms in brain CT scans. Simultaneously, it focuses on rarer medical terms and combines this with a hierarchical attention mechanism to capture important lesion region features, thereby generating accurate brain CT medical reports. Summary of the Invention

[0007] Currently, there is limited research on the automatic generation of brain CT medical reports. The task of generating brain CT medical reports involves automatically generating a corresponding brain CT medical report from a set of brain CT images. For example... Figure 1 As shown, a set of brain CT images consists of multiple slices, in which the lesions exhibit both continuity and sparseness; the corresponding medical report is as follows. Figure 1 As shown in (a), its content is complex, consisting of a description of symptoms and a diagnostic conclusion. The description of symptoms details the symptoms present in the brain CT images, while the diagnostic conclusion is a brief summary of the abnormal symptoms. Figure 1(b) shows multiple medical terms corresponding to this set of brain CT images. Because brain CT images contain rich pathological information, this invention attempts to explore the co-occurrence relationships between common medical terms in brain CT scans in order to mine the semantic relevance of common medical terms, extract richer semantic information, and simultaneously focus on rare medical terms to extract important lesion region features. This is combined with visual and semantic hierarchical attention mechanisms to generate accurate and diverse brain CT medical reports.

[0008] To achieve the above objectives, this invention proposes an automatic generation method for brain CT medical reports based on co-occurrence relation hierarchical attention, characterized by: (1) preprocessing the brain CT dataset to obtain standardized three-dimensional brain CT images and their corresponding brain CT medical report texts, and establishing a vocabulary; (2) constructing a feature extractor for brain CT images to extract visual features F = [f1, f2, ... f2] of the brain CT images. i ,…f N ], where N represents the number of slices in the brain CT image sequence, f i(3) Construct a co-occurrence relation semantic attention module, which is used to extract semantic attention features of common medical terms in brain CT images. It includes a word embedding layer and a semantic attention mechanism. The word embedding layer is used to embed common medical terms into semantic features, and the semantic attention mechanism calculates the weights corresponding to each medical term to assign different attention to different medical terms, extract important semantic information, and ignore irrelevant medical terms. In order to infer potential semantic information and enrich semantic expression, this invention proposes co-occurrence relation semantic attention to explore the co-occurrence relationship between medical terms and mine the semantic correlation between medical terms. The main approach is to extract common medical terms in the dataset, obtain the semantic features of medical terms through the word embedding layer, use them for semantic attention calculation and obtain the weights corresponding to each medical term, embed the co-occurrence relation matrix between common medical terms into the relation weights through linear transformation, and fuse the relation weights with the weights corresponding to each medical term calculated before to obtain a new weight. The new weights combine the co-occurrence relationships among common medical terms, which can better express the importance of each medical term and give some attention to those potentially related medical terms, enriching the expression of semantic attention features and containing more pathological information. (4) Construct a topic vector-guided visual attention module. The topic vector in this module integrates the semantic information of common and rare medical terms, fully expressing the topic of medical terms at the sentence level. The topic of medical terms guides the visual attention mechanism to capture important lesion area features. The main approach is to first organize the rare medical term database in the dataset, retrieve the corresponding rare medical terms through visual global features, and obtain the rare medical term topic vector through word embedding. Embed the semantic attention features of common medical terms into the common medical term topic vector and fuse them with the rare medical term topic vector. Then guide the visual attention mechanism to capture important lesion areas to obtain visual attention features. The visual attention mechanism includes a sequence attention and a spatial attention. The sequence attention is used to select important slice sequences, and the spatial attention is used to select lesion areas in the slices. The two are combined to capture the key lesion area features in important slices, and then the medical report is generated through the language generation model. The specific steps of this invention are as follows:

[0009] Step (1) Preprocess the brain CT images and their corresponding medical report datasets:

[0010] Step (1.1): Acquire brain CT images and construct slice sequences. Each patient's data includes an RGB image sequence I = {I1, I2, ... I...} generated from the brain CT images. i ,…I N}, The corresponding medical report Y = [y1, y2, ... y i ,…y T ], where N represents the number of brain CT slice sequences, I i Let W represent the RGB image matrix of the i-th slice, where W and H represent the width and height of the image, respectively. i The word represents the i-th word in the report, and T represents the number of words in the brain CT medical report. This means the dimensions of the image are 3×W×H.

[0011] Step (1.2) divides all patient data into training, validation, and test sets in a 7:1:2 ratio. The training set is used to learn the model parameters; the validation set is used to determine hyperparameters; and the test set is used to validate model performance.

[0012] Step (1.3) Data preprocessing: For brain CT image sequences with no more than 24 slices, interpolation algorithm is used to complete them; for brain CT image sequences with more than 24 slices, uniform sampling method is used to select them.

[0013] Step (2) Construct a feature extractor to extract visual feature maps F = [f1, f2, ... f1] from brain CT images. i ,…f N ] and global feature FC = [fc1, fc2, ... fc i ,…fc N ], where N represents the number of brain CT slice sequences, f i fc represents the visual feature map of the i-th brain CT slice. i This represents the global visual features of the i-th brain CT slice. A ResNet101 network is trained on the brain CT dataset, and the feature map of the last layer of the ResNet101 network is used as the final image visual feature F. The calculation process for extracting the image visual feature F is as follows:

[0014] F, FC = Resnet101(I)

[0015] Step (3) Constructs a co-occurrence-driven semantic attention module to extract semantic information from common medical terms. It then combines the co-occurrence relationships between these terms with the semantic attention mechanism to mine semantic relevance and infer missing semantic information. The specific process is as follows:

[0016] Step (3.1) constructs a semantic attention mechanism in the co-occurrence relation-driven semantic attention module to extract semantic features of common medical terms in brain CT images. This mechanism first processes m common medical term keywords K = {k1,k2,…k}. i ,…k mWord embedding is performed to obtain semantic features S of common medical terms. Then, a semantic attention mechanism is combined, using visual global features to guide the calculation of attention, thereby assigning different weights to the semantic features of different medical terms and automatically capturing important semantic information. First, the semantic attention weight α of the semantic features S of common medical terms is calculated. init The calculation process is as follows:

[0017] S = Embedding(K)

[0018] e=φ(h m ,FC,S)

[0019] α init =Softmax(e)

[0020] Embedding is a semantic embedding layer that embeds common medical terms into semantic features S. φ is a feedforward neural network that automatically learns the correlation matrix between its three internal parameters, h. m LSTM for encoding and memorizing already generated sentences m The internal hidden layer states are: FC represents the global visual features, S represents the semantic features of common medical terms, e represents the output of the feedforward neural network, and α represents the output of the feedforward neural network. init These are the initial semantic attention weights for the output. Where LSTM... m and its hidden layer state h m The definition is as follows:

[0021] h m =LSTM m (sen pre )

[0022] sen pre This is the previously generated sentence. The semantic attention loss function is defined as follows:

[0023] Loss sem =BCE(σ(e),label)

[0024] Where BCE represents the binary cross-entropy loss function, σ represents the sigmoid activation function, and label represents the correct label of common medical terms for each sample. The initial semantic attention weights α are obtained. init Subsequently, in order to infer missing semantic information by utilizing the co-occurrence relationships among common medical terms, mine the semantic relevance of common medical terms, and enhance the expression of semantic attention features, this invention introduces a co-occurrence relationship matrix among common medical terms and embeds it into the relationship weights, and calculates the final semantic attention features by combining the initial semantic attention weights with semantic features.

[0025] Step (3.2) Constructs a co-occurrence matrix R among common medical terms. Based on the brain CT dataset, the frequency of common medical terms in brain CT medical reports and the co-occurrence frequency among these terms are statistically analyzed, and then the co-occurrence relationship values ​​among the medical terms are calculated. The specific definitions are as follows:

[0026]

[0027] Where f(a,b) represents the frequency of both medical terms a and b appearing in the report, and f(a) and f(b) represent the frequencies of each medical term appearing individually in the report. N represents the total number of samples. max represents taking the maximum value, and log represents the logarithmic function.

[0028] Step (3.3) combines the initial semantic attention weights with the relation matrix to obtain the final attention weights, thereby extracting richer semantic information. The specific steps are as follows:

[0029] (I) Embed the co-occurrence relation matrix into relation weights α R :

[0030] α R =Relu(W R R)

[0031] Among them, W R R represents the parameter matrix of the linear embedding layer, and R is the static co-occurrence relation matrix. ReLU represents the activation function.

[0032] (II) Combine the initial attention weights with the relation weights to obtain the final attention weights, and then calculate the final semantic attention features S. att :

[0033] S att =α init *α R *S

[0034] Where, α init Let α be the initial attention weight. R Represents the weight of the relationship.

[0035] Step (4) Construct a visual attention module guided by common and rare topics. This module integrates the semantic information of common and rare medical terms as sentence-level topic vectors to guide the calculation of visual attention, in order to capture key lesion region features. A hierarchical recurrent network is proposed as a language model to generate brain CT medical reports. The specific steps are as follows:

[0036] Step (4.1) First, based on the semantic attention features S of common medical terms obtained in step (3), att Obtain topic vectors of common medical terms at the sentence level.

[0037]

[0038] Among them, W c , W S These are the learnable parameters in a feedforward neural network. For LSTM m The hidden layer states were determined. To model the semantic information of rare medical terms, 40 rare medical terms corresponding to brain CT images were identified based on doctors' clinical experience and brain CT medical reports in the dataset, such as orbital protrusion, sphenoid sinus mucosal thickening, and shallowing of gyri. Corresponding rare medical terms were retrieved from the rare medical terminology database based on global visual features, and topic vectors of the rare medical terms were obtained through word embedding. The specific steps are as follows:

[0039]

[0040]

[0041] Among them, W rare W fc , For learnable parameters in a feedforward neural network, rare index The index is the corresponding subscript in the retrieved rare medical terminology database. Embedding is the embedding layer. Then, we fuse the topic vectors of common medical terms with those of rare medical terms to obtain the final sentence-level topic vector to guide the computation of the visual attention mechanism.

[0042] Step (4.2) first merges the topic vectors of common medical terms with those of rare medical terms:

[0043]

[0044] Among them W topic W c W r λ represents the learnable parameters in a feedforward neural network. common ,λ rare , representing the weights of the topic vectors for common and rare medical terms, respectively. To guide the retrieval of rare medical terms, a rare medical term classification loss is introduced, defined as follows:

[0045] Loss rare =CE(rare) index ,rare label )

[0046] Where CE represents cross-entropy loss, rare label The labels are real. The topic vectors at the sentence level are obtained. s Subsequently, sequence attention and spatial attention are calculated at the sentence level and word level, respectively, to obtain the visual attention feature V. att :

[0047]

[0048]

[0049] V att =α frame *α spatial *F

[0050] Among them, Attention spatial Attention frame These represent the computation of spatial attention and sequence attention, respectively. Sequence attention selects the most important slices from a sequence of slices by assigning different weights to each slice, while spatial attention captures important lesion region features by assigning different weights to scale features within a slice. Representing LSTM word LSTM sent Hidden state, LSTM word LSTM sent The definition is as follows:

[0051]

[0052]

[0053] Among them, sen pre Represents the previously generated sentence, [,] represents the concatenation operation, V att Represents the visual attention features corresponding to each time step. Representing LSTM sent LSTM word In the hidden state at time step t-1, x t This represents the words input at each time step; during training, they come from real reports, and during testing, they come from words generated at the previous time step. (This is achieved through the `sen` function.) pre and LSTM word LSTM sent The final medical report is generated collaboratively in an interactive manner. The probability distribution p of the predicted words at each time step is then determined. t It can be defined as:

[0054]

[0055] in, For the learnable parameters of the linear embedding layer, For the LSTM at time step t word The hidden state of y. t For the predicted word, p t The predicted probabilities of the words are used to assemble a brain CT scan report. The language model training process is optimized using the following loss function:

[0056]

[0057] Finally, the overall loss function is defined as follows:

[0058] Loss = λ1 * Loss word +λ2*Loss sem +λ3*Loss rare

[0059] Here, λ1, λ2, and λ3 represent the weights of each loss, which are used to balance the magnitude of each loss in order to better train the model.

[0060] Compared with existing methods, the present invention has the following significant advantages and innovations:

[0061] This invention proposes an automatic brain CT medical report generation method based on co-occurrence relation-driven hierarchical attention. The method has the following characteristics: 1) A co-occurrence relation-driven semantic attention mechanism (CRSA) is designed to mine the semantic information of common medical terms in brain CT images, and combines the co-occurrence relations between common medical terms to infer missing semantic information, thereby extracting richer semantic attention features. 2) A common-rare topic-guided visual attention mechanism (CRVA) is designed. By fusing the topic vectors of common and rare medical terms as sentence-level topic vectors, it guides the visual attention mechanism to capture key lesion region features for brain CT medical report generation, and utilizes LSTM... word and LSTM sent The hierarchical collaboration improves the accuracy and diversity of the generated brain CT medical reports. 3) This invention is the first to explore the semantic correlation between common medical terms in brain CT images, while also taking into account the semantic information of rare medical terms. Sufficient experiments have demonstrated the effectiveness of the method proposed in this invention. Attached Figure Description

[0062] Figure 1 A schematic diagram of a set of brain CT images and their corresponding medical reports and medical terminology.

[0063] Figure 2A framework diagram of an automatic brain CT medical report generation method based on co-occurrence relation hierarchical attention.

[0064] Figure 3 A schematic diagram of the co-occurrence matrix of common medical terms.

[0065] Figure 4 : A diagram showing the medical report generated by this invention. Detailed Implementation

[0066] This embodiment uses brain CT images and corresponding medical reports provided by a cooperating hospital as the research object, and specifically describes the implementation steps of this method:

[0067] Step (1) Preprocess the brain CT images and their corresponding medical report datasets:

[0068] Step (1.1) Data: Collect brain CT images and construct slice sequences. Each patient's data includes an RGB image sequence I = {I1, I2, ... I...} generated from the brain CT images. i ,…I N}, The corresponding brain CT medical report Y=[y1,y2,…y i ,…y T ], where N represents the number of brain CT slice sequences, I i Let W represent the RGB image matrix of the i-th slice, where W and H represent the width and height of the image, respectively. i The word represents the i-th word in the brain CT medical report, and T represents the number of words in the brain CT medical report. This means the dimensions of the image are 3×W×H.

[0069] Step (1.2) randomly divide the dataset into three parts: training set, validation set and test set in a ratio of 7:1:2, with 1436 / 204 / 408 samples in each part.

[0070] Step (1.3) Data preprocessing: For brain CT image sequences with no more than 24 slices, interpolation algorithm is used for completion; for brain CT image sequences with more than 24 slices, uniform sampling method is used for selection. Then, the width and height of the brain CT images are uniformly set to 512×512, and finally, the brain CT image slice sequences are standardized and homogenized.

[0071] Step (2) Constructing the feature extractor: First, a binary classification dataset was constructed based on whether there are hemorrhagic lesions in brain CT images. Then, a pre-trained ResNet101 model was used to fine-tune this binary classification task based on the dataset. Finally, the last convolutional layer of the ResNet101 network was used as the final visual feature map. The representative dimension is used as the final global visual feature after average pooling. Where 2048 represents the number of channels, 24 represents the number of slices in the brain CT image slice sequence, and 196 represents the number of spatial attention units in the feature map. The formula for feature extraction is as follows:

[0072] F, FC = Resnet101(I)

[0073] Step (3) Constructs the co-occurrence relation-driven semantic attention module proposed in this invention to extract semantic information of common medical terms, and combines the co-occurrence relations between common medical terms and the semantic attention mechanism to mine the semantic relevance between medical terms, infer missing semantic information, and obtain semantic attention features with rich semantic information. The specific process is as follows:

[0074] Step (3.1) constructs a semantic attention mechanism in the co-occurrence relation-driven semantic attention module to extract semantic features of common medical terms in brain CT images. This mechanism first processes m common medical term keywords K = {k1,k2,…k}. i ,…k m Word embedding is performed to obtain semantic features S of common medical terms. Then, a semantic attention mechanism is combined, using global visual features to guide the calculation of attention, thereby assigning different weights to the semantic features of different medical terms and automatically capturing important semantic information. Specifically, this mechanism first calculates the semantic attention weight α of the semantic features S of common medical terms. init The calculation process is as follows:

[0075] S = Embedding(K)

[0076] e=φ(h m ,FC,S)

[0077] α init =Softmax(e)

[0078] φ is a feedforward neural network that can automatically learn the correlation matrix between its three internal parameters. Embedding is a semantic embedding layer that embeds common medical terms into semantic features S and h. m LSTM for encoding and memorizing already generated sentences m The internal hidden layer states are: FC represents the global visual features, S represents the semantic features of common medical terms, e represents the output of the feedforward neural network, and α represents the output of the feedforward neural network. init These are the initial semantic attention weights for the output. Where LSTM... m and its hidden layer state h m The definition is as follows:

[0079] h m =LSTM m (sen pre )

[0080] sen pre This is the previously generated sentence. To guide the initial semantic attention calculation process, this invention designs a semantic attention loss function defined as follows:

[0081] Loss sem =BCE(σ(e),label)

[0082] Where BCE represents the binary cross-entropy loss function, σ represents the sigmoid activation function, and label represents the correct label of common medical terms for each sample. The initial semantic attention weights α are obtained through the above process. init Subsequently, in order to infer missing semantic information by utilizing the co-occurrence relationships among common medical terms, mine the semantic relevance of common medical terms, and enhance the expression of semantic attention features, this invention introduces a co-occurrence relationship matrix among common medical terms and embeds it into the relationship weights, and calculates the final semantic attention features by combining the initial semantic attention weights with semantic features.

[0083] Step (3.2) Constructs a co-occurrence matrix R among common medical terms. Based on the brain CT dataset, the frequency of common medical terms in medical reports and the co-occurrence frequency among medical terms are statistically analyzed, and then the co-occurrence relationship values ​​among medical terms are calculated. The specific definitions are as follows:

[0084]

[0085] Where f(a,b) represents the frequency of both medical terms a and b appearing in the report, and f(a) and f(b) represent the frequency of each medical term a and b appearing individually in the report, respectively. N represents the total number of samples.

[0086] Step (3.3) combines the initial semantic attention weights with the relation matrix to obtain the final attention weights, thereby extracting richer semantic information. The specific steps are as follows:

[0087] (III) Embed the co-occurrence relation matrix into relation weight α R :

[0088] α R =Relu(W R R)

[0089] Among them, W R R represents the parameter matrix of the linear embedding layer, and R is the static co-occurrence relation matrix.

[0090] (IV) Combine the initial attention weights with the relation weights to obtain the final attention weights, and then calculate the final semantic attention feature S. att :

[0091] S att =α init *α R *S

[0092] Step (4) Construct a visual attention module guided by common and rare topics. This module integrates the semantic information of common and rare medical terms as sentence-level topic vectors to guide the calculation of visual attention, in order to capture key lesion region features. It is then combined with a language model to generate medical reports. Since medical reports are often long, a traditional single-layer LSTM is insufficient to generate accurate and diverse brain CT medical reports; therefore, we propose a hierarchical recurrent network as the language model in this invention. The specific steps are as follows:

[0093] Step (4.1) First, based on the semantic attention features S of common medical terms obtained in step (3), att Obtain topic vectors of common medical terms at the sentence level.

[0094]

[0095] Among them, W c , W S These are the learnable parameters in a feedforward neural network. For LSTM m The hidden layer states. To model the semantic information of rare medical terms, this invention, based on doctors' clinical experience and medical reports in the dataset, statistically analyzed 40 rare medical terms corresponding to brain CT images, such as orbital protrusion, sphenoid sinus mucosal thickening, and shallowing of gyri. Then, we retrieved the corresponding rare medical terms from the rare medical terminology database based on global visual features and obtained the topic vector of the rare medical terms through word embedding. The specific steps are as follows:

[0096]

[0097]

[0098] Among them, W rare W fc , For learnable parameters in a feedforward neural network, rare indexThe index is the corresponding subscript in the retrieved rare medical terminology database. Then, we fuse the topic vectors of common medical terms with those of rare medical terms to obtain the final sentence-level topic vector to guide the computation of the visual attention mechanism.

[0099] Step (4.2) first merges the topic vectors of common medical terms with those of rare medical terms:

[0100]

[0101] Among them W topic W c W r λ represents the learnable parameters in a feedforward neural network. common ,λ rare These represent the weights of the topic vectors for common and rare medical terms, respectively. To guide the retrieval of rare medical terms, this invention introduces a rare medical term classification loss, defined as follows:

[0102] Loss rare =CE(rare) index ,rare label )

[0103] Where CE represents cross-entropy loss, rare label The labels are real. The topic vectors at the sentence level are obtained. s Subsequently, sequence attention and spatial attention are calculated at the sentence level and word level, respectively, to obtain the visual attention feature V. att :

[0104]

[0105]

[0106] V att =α frame * αspatial *F

[0107] Among them, Attention spatial Attention frame These represent the computation of spatial attention and sequence attention, respectively. Sequence attention selects the most important slices from a sequence of slices by assigning different weights to each slice, while spatial attention captures important lesion region features by assigning different weights to scale features within a slice. Representing LSTM word LSTM sent Hidden state, LSTM word LSTM sentThe definition is as follows:

[0108]

[0109]

[0110] Among them, sen pre Represents the previously generated sentence, [,] represents the concatenation operation, V att x represents the visual attention feature corresponding to each time step. t This represents the words input at each time step; during training, they come from real reports, and during testing, they come from words generated at the previous time step. (This is achieved through the `sen` function.) pre and LSTM word LSTM sent The final medical report is generated collaboratively in an interactive manner. The probability distribution p of the predicted words at each time step is then determined. t It can be defined as:

[0111]

[0112] in, For the learnable parameters of the linear embedding layer, For the LSTM at time step t word The hidden state of y. t The predicted words are ultimately used to assemble a medical report. The language model training process is optimized using the following loss function:

[0113]

[0114] Step (5) Define the overall loss function and the automatic generation network for brain CT medical reports:

[0115] Step (5.1) defines the loss function for the language generation module. The automatic generation method for brain CT medical reports based on co-occurrence relation hierarchical attention described in this invention generates brain CT images I = {I1, L2, ... I...} for the patient. i ,…I N Generate the corresponding brain CT medical report Y = {y1, ..., y} T In this context, I serves as the input for network training and testing, Y (the medical report) serves as the true label for network training, and the network output is the predicted report. Among them, y i , T represents the i-th word in a real report and a predictive report, respectively. These represent the number of words in a real brain CT medical report and a predicted report, respectively. The objective function of the final language generation module in this invention is to maximize the similarity between the real report and the predicted report, calculated as follows:

[0116]

[0117] Where θ * Here, θ represents all the network parameters after training, and θ represents all the network parameters to be trained in the model. The loss function for the language generation module uses cross-entropy loss, specifically defined as follows:

[0118]

[0119] in This represents the predicted probability value of the t-th word generated by the model.

[0120] Step (5.2) defines the final loss function of the automatic generation method for brain CT medical reports based on co-occurrence relation hierarchical attention described in this invention, which includes three parts:

[0121] Loss = λ1 * Loss word +λ2*Loss sem +λ3*Loss rare

[0122] Ultimately, under the Adam optimizer, the network can adaptively optimize the loss values ​​of the real report and the predicted report. After training, when a set of brain CT images is input, the model can automatically generate a coherent and accurate brain CT medical report.

[0123] To verify the effectiveness of the method described in this invention, we conducted experiments on a brain CT medical report dataset and performed a thorough comparative analysis with other existing methods. In the specific implementation, we compared the method with three mainstream methods in the field of image description: CNN-RNN, Soft-ATT, and Up-Down methods. We also reproduced two methods for automatically generating medical reports for chest X-ray images, including HRNN and MRMA, as comparison methods. Furthermore, we conducted a thorough comparison with WGAM, the only work in the field of brain CT medical report generation. The experiments not only employed automatic evaluation metrics for natural language processing tasks such as text generation, including BLEU, METEOR, ROUGE, and CIDEr (which evaluate the model's effectiveness by calculating the similarity between the generated and reference sentences), but also extracted n keywords as clinical evaluation metrics to assess the accuracy, precision, recall, and F1 score of the keywords in the generated medical reports.

[0124] Table 1. Comparison of the method proposed in this invention with different methods (%)

[0125]

[0126]

[0127] Table 1 compares the performance of our method with other methods on seven language metrics and four clinical metrics. It can be seen that early encoder-decoder based methods, such as CNN-RNN and HRNN, which did not use attention mechanisms, performed significantly worse than other methods using attention mechanisms, such as Soft-ATT and Up-Down. This indicates that the introduction of attention mechanisms is beneficial for the model to capture key lesion region features. Our method's baseline is a hierarchical recurrent network based on an encoder-decoder framework. As shown in Table 1, our method achieved good performance on all metrics, especially showing the most significant improvement on the CIDEr metric, demonstrating the effectiveness of our proposed method in the field of automatic generation of brain CT medical reports. HRNN and our method have similar language models, both containing sentence-level and word-level recurrent networks. Compared to the HRNN method, our method uses word LSTM to control the number of generated sentences, increasing the collaboration between word LSTM and sentence LSTM in the language generation process. Furthermore, our method combines semantic attention mechanisms and visual attention mechanisms in a hierarchical manner, enhancing the synergistic effect between visual and semantic information. MRMA employs a multimodal recursive model based on an attention mechanism, which outperforms HRNN, but its performance is still inferior to our method across various metrics.

[0128] Furthermore, we conducted ablation experiments to demonstrate the effectiveness of the two attention modules in the proposed method: a co-occurrence relation-driven semantic attention module (CRSA) and a common-rare topic-guided visual attention module (CRVA). The baseline of this method is a hierarchical recurrent network based on an encoder-decoder framework.

[0129] Table 2 Ablation experiments (%) of different modules in the method proposed in this invention

[0130]

[0131] As shown in Table 2, Baseline+CRSA and Baseline+CRVA significantly outperform Baseline across all metrics, with Baseline+CRVA exhibiting superior performance. This demonstrates the effectiveness of common and rare medical terminology topics in guiding visual attention mechanisms. Our proposed method, Baseline+CRSA+CRVA, achieves optimal performance across all metrics. This indicates that mining co-occurrence relationships among medical terms helps the model extract richer and more complete semantic information.

[0132] In summary, through comparative and ablation experiments, this invention verifies the effectiveness of the proposed method in the automatic generation of brain CT medical reports. This is because the CRSA module mines the semantic correlations between common medical terms, enabling it to infer potentially missing semantic information and facilitating the model to extract richer semantic attention features. Furthermore, the CRVA module integrates topic vectors of both common and rare medical terms, guiding the visual attention mechanism to capture key and accurate lesion region features. Additionally, the hierarchical recurrent network in the language generation module also demonstrates unique advantages in generating long paragraph text tasks.

[0133] Figure 4 The qualitative results of our proposed method are presented. It can be seen that our method can not only capture the complete semantic information of common medical terms, but also generate descriptions of relatively rare symptoms. However, existing works have poor performance in generating rare medical terms. Figure 4 As shown in (a), our model fully predicts the pathology in the "parietal lobe" and "occipital lobe," a point previously overlooked. Our model also predicts pathology such as "low-density lesions in the basal ganglia" and "well-defined borders." This indicates that our method extracts richer semantic information than previous work, thanks to the introduction of co-occurrence relationships between common medical terms in our approach. Figure 4 As shown in (b), "cerebral gyri swelling" is a relatively rare medical term, which our model correctly predicted. However, existing work has neglected descriptions of rare medical terms like "cerebral gyri swelling." Furthermore, our model outperforms baseline and existing methods in terms of sentence coherence. For example, reports generated by the WGAM model contain repetitive sentences such as "shallow sulci, shallow sulci," while our model generates medical reports with better coherence and diversity.

Claims

1. A method for automatically generating brain CT medical reports based on hierarchical attention according to co-occurrence relationships, characterized in that: The method includes the following steps: (1) Preprocess the brain CT dataset to obtain standardized three-dimensional brain CT images and their corresponding brain CT medical report texts, and establish a vocabulary; (2) Construct a feature extractor for brain CT images to extract visual features F and global features FC from brain CT images; (3) Construct a co-occurrence relation semantic attention module. Obtain the semantic features of common medical terms through word embedding. Combined with the semantic attention mechanism, use visual global features to guide the calculation of attention and obtain the initial semantic attention weight α. init Construct a co-occurrence matrix R among common medical terms, and assign α init Combined with R as the final attention weight, the semantic attention features S of common medical terms are calculated. att ; (4) Construct a topic vector-guided visual attention module, based on S att Obtain topic vectors of common medical terms at the sentence level. ; Based on global visual features, corresponding rare medical terms are retrieved from a rare medical terminology database, and topic vectors of rare medical terms are obtained through word embedding. ;Will and The fusion process yields sentence-level topic vectors. s Then, the visual attention mechanism is guided to capture the lesion area, thereby obtaining the visual attention feature V. att Using a hierarchical recurrent network as a language model, brain CT medical reports are generated, including: obtaining the topic. s Subsequently, sequence attention and spatial attention are calculated at the sentence level and word level, respectively, to obtain the visual attention feature V. att : ； ； V att =a frame *a spatial *F; Among them, Attention spatial Attention frame These represent the computation of spatial attention and sequence attention, respectively. Sequence attention selects the most important slice from the slice sequence by assigning different weights to each slice, while spatial attention captures important lesion region features by assigning different weights to the scale features within the slice. Representing LSTM word LSTM sent In the hidden state at time step t-1, LSTM word LSTM sent The definition is as follows: ； ； Among them, sen pre Represents the previously generated sentence, [,] represents the concatenation operation, V att x represents the visual attention feature corresponding to each time step. t This represents the words entered at each time step; via sen pre and LSTM word LSTM sent The final brain CT medical report is generated collaboratively in an interactive manner; the probability distribution p of the predicted words at each time step is ultimately determined. t Defined as: ； in, For the learnable parameters of the linear embedding layer, For the LSTM at time step t word The hidden state of y; t For the predicted word, p t The predicted probabilities of the words are used to compose a brain CT medical report.

2. The method for automatically generating brain CT medical reports based on co-occurrence relationship hierarchical attention according to claim 1, characterized in that: The preprocessing procedure for brain CT images and their corresponding brain CT medical report datasets is as follows. Step (1.1): Acquire brain CT images and construct slice sequences. Each patient's data includes an RGB image sequence I = {I1, I2, ... I...} generated from the brain CT images. i ,…I N }, The corresponding brain CT medical report Y=[y1,y2,…y i ,…y T ], where N represents the number of brain CT slice sequences, I i Let W represent the RGB image matrix of the i-th slice, where W and H represent the width and height of the image, respectively. i The word represents the i-th word in the brain CT medical report, and T represents the number of words in the brain CT medical report; The dimensions of this image are 3×W×H; Step (1.2) divides all patient data into training, validation, and test sets in a 7:1:2 ratio; the training set is used to learn the model parameters; the validation set is used to determine hyperparameters; and the test set is used to validate model performance. Step (1.3) Data preprocessing: For brain CT image sequences with no more than 24 slices, interpolation algorithm is used to complete them; for brain CT image sequences with more than 24 slices, uniform sampling method is used to select them.

3. The method for automatically generating brain CT medical reports based on co-occurrence relationship hierarchical attention according to claim 2, characterized in that: Construct a feature extractor to extract visual feature maps F = [f1, f2, ... f] from brain CT images. i ,…f N ] and global feature FC = [fc1, fc2, ... fc i ,…fc N ], where N represents the number of brain CT slice sequences, f i fc represents the visual feature map of the i-th brain CT slice. i Global visual features representing the i-th brain CT slice; The ResNet101 network was trained on a brain CT dataset. The feature map of the last layer of the ResNet101 network was then used as the final visual feature F of the image. The calculation process for extracting the visual feature F is as follows: F, FC = Resnet101(I).

4. The method for automatically generating brain CT medical reports based on co-occurrence relationship hierarchical attention according to claim 1, characterized in that: A co-occurrence-driven semantic attention module is constructed to extract semantic information from common medical terms. This module, combined with co-occurrence relationships and a semantic attention mechanism, is used to mine semantic relevance among common medical terms and infer missing semantic information. The specific process is as follows: Step (3.1) constructs a semantic attention mechanism in the co-occurrence relation-driven semantic attention module to extract semantic features of common medical terms in brain CT images. This semantic attention mechanism focuses on m common medical term keywords K = {k1, k2, ... k}. i ,…k m Word embedding is performed to obtain semantic features S of common medical terms. Combined with a semantic attention mechanism, the calculation of attention is guided by visual global features. Different weights are assigned to the semantic features of different medical terms to automatically capture important semantic information. First, the semantic attention weight α of common medical terms is calculated based on their semantic features S. init The calculation process is as follows: S = Embedding(K); e=φ(h m ,FC,S); a init =Softmax(e); Embedding is a semantic embedding layer that automatically learns the correlation matrix between three internal parameters, h, of the semantic features S of common medical terms through a feedforward neural network φ. m LSTM for encoding and memorizing already generated sentences m The internal hidden layer states are: FC represents the global visual features, S represents the semantic features of common medical terms, e represents the output of the feedforward neural network, and α represents the output of the feedforward neural network. init The initial semantic attention weights are the output; where LSTM... m and its hidden layer state h m The definition is as follows: h m =LSTM m (sen prev ); sen prev Given the previously generated sentence; the semantic attention loss function is defined as follows: Loss sem = BCE(σ(e),label); Where BCE represents the binary cross-entropy loss function, σ represents the sigmoid activation function, and label represents the correct label of common medical terms for each sample; the initial semantic attention weight α is obtained. init Subsequently, a co-occurrence relation matrix among common medical terms is introduced and embedded into relation weights. The final semantic attention features are then calculated by combining the initial semantic attention weights with semantic features. Step (3.2) Construct a co-occurrence matrix R among common medical terms; based on the brain CT dataset, statistically analyze the frequency of occurrence of common medical terms in brain CT medical reports and their co-occurrence frequencies, and then calculate the co-occurrence relationship values ​​among common medical terms; the specific definitions are as follows: ； Where f(a,b) represents the frequency of the two common medical terms a and b appearing simultaneously in a brain CT medical report, and f(a) and f(b) represent the frequency of the two common medical terms a and b appearing individually in a brain CT medical report, respectively; N n Represents the total number of samples; max represents taking the maximum value, and log represents the logarithmic function; Step (3.3) combines the initial semantic attention weights with the relation matrix to obtain the final attention weights, thereby extracting richer semantic information; the specific steps are as follows: (I) Embed the co-occurrence relation matrix into relation weights α R : α R =Relu(W R R); Among them, W R R represents the parameter matrix of the linear embedding layer, and R is the static co-occurrence relation matrix; ReLU stands for activation function; (II) Combine the initial attention weights with the relation weights to obtain the final attention weights, and then calculate the final semantic attention features S. att : S att =a init *a R *S; Where, α init Let α be the initial attention weight. R Represents the weight of the relationship.

5. The method for automatically generating brain CT medical reports based on co-occurrence relationship hierarchical attention according to claim 4, characterized in that: A visual attention module guided by common and rare topics is constructed. This module integrates the semantic information of common and rare medical terms as sentence-level topic vectors to guide the calculation of visual attention, in order to capture key lesion region features. Common medical terms include low-density shadows in the basal ganglia, thalamic hemorrhage, lateral ventricle compression and narrowing, and rightward shift of the brain midline; rare medical terms include sphenoid sinus mucosal thickening, localized orbital depression, and arachnoid cysts. A hierarchical recurrent network is proposed as a language model to generate brain CT medical reports. The specific steps are as follows: Step (4.1) First, based on the semantic attention features S of common medical terms obtained in step (3), att Obtain topic vectors of common medical terms at the sentence level. ； Among them, W c , W S These are the learnable parameters in a feedforward neural network. For LSTM m The hidden layer states; to model the semantic information of rare medical terms, corresponding rare medical terms are retrieved from the rare medical term database based on global visual features, and the topic vectors of rare medical terms are obtained through word embedding. The specific steps are as follows: ； ； Among them, W rare W fc , For learnable parameters in a feedforward neural network, rare index The subscripts of the medical terms in the retrieved rare medical terminology database are used; the embedding layer is used; then, the topic vectors of common medical terms and rare medical terms are fused to obtain the final sentence-level topic vectors to guide the calculation of the visual attention mechanism. Step (4.2) first merges the topic vectors of common medical terms with those of rare medical terms: ； Among them W topic W c W r λ is a learnable parameter in a feedforward neural network. common ,λ rare , representing the weights of the topic vectors for common and rare medical terms, respectively; to guide the retrieval of rare medical terms, a rare medical term classification loss is introduced, defined as follows: Loss rare =CE(rare index ,rare label ); Where CE represents cross-entropy loss, rare label For real tags, rare index This is the predicted value; after obtaining the topic vector at the sentence level, the Topic... s Subsequently, sequence attention and spatial attention are calculated at the sentence level and word level, respectively, to obtain the visual attention feature V. att : ； ； V att =a frame *a spatial *F; Among them, Attention spatial Attention frame These represent the computation of spatial attention and sequence attention, respectively. Sequence attention selects the most important slice from the slice sequence by assigning different weights to each slice, while spatial attention captures important lesion region features by assigning different weights to the scale features within the slice. Representing LSTM word LSTM sent In the hidden state at time step t-1, LSTM word LSTM sent The definition is as follows: ； ； Among them, sen pre Represents the previously generated sentence, [,] represents the concatenation operation, V att x represents the visual attention feature corresponding to each time step. t This represents the words input at each time step; during training, they are derived from real brain CT medical reports, and during testing, they are derived from words generated at the previous time step; through sen pre and LSTM word LSTM sent The final brain CT medical report is generated collaboratively in an interactive manner; the probability distribution p of the predicted words at each time step is ultimately determined. t Defined as: ； in, For the learnable parameters of the linear embedding layer, For the LSTM at time step t word The hidden state of y; t For the predicted word, p t The predicted probabilities of the words are used to assemble a brain CT scan report; the language model training process is optimized using the following loss function: ； Finally, the loss function is defined as follows: Loss=λ1*Loss word +λ2*Loss sem +λ3*Loss rare ; Here, λ1, λ2, and λ3 represent the weights of each loss, which are used to balance the magnitude of each loss in order to better train the model.

Images