Medical image report generation method based on double-branch feature fusion

Abstract

The application discloses a medical report generation method based on dual-branch feature fusion. Traditional medical image report automatic generation methods usually only use convolutional neural networks to extract image features, focus on local details, and cannot capture global relationships well. Therefore, the application first proposes a medical image report generation method based on dual-branch feature fusion (Dual-Branch Feature Fusion Network for Medical Report Generation, DBFFN), which combines the advantages of Transformers and convolutional neural networks to extract global features and local features of medical images, captures global relationships, and focuses on local subtle semantic information. A multi-scale feature fusion module (Multi-Scale Feature Fusion Module, MSFFM) is designed to align image features and fuse semantic information. A large number of experiments on the IU-X-Ray dataset prove the effectiveness of the DBFFN model and the MSFFM fusion module proposed by the application.

Description

Technical Field

[0001] This invention belongs to the field of automatic generation method for medical image reports, and relates to the technical fields of computer vision and natural language processing. Background Technology

[0002] As an interdisciplinary field of computer vision and natural language processing, the task of automatic medical image report generation models is to generate textual descriptions of given medical images. However, unlike traditional image captioning tasks, medical reports should include professional descriptions of multiple regions in the image, thus requiring high accuracy in word choice and considerable length. To improve the quality of generated image reports, Najdenkoska et al. introduced topic vectors to guide sentence generation, making the generated medical image reports more professional in their word choice. Han et al. combined meta-learning and symbolic reasoning to enhance the causal relationships between medical image report texts. Yang et al., from a multi-view perspective, improved the consistency of the model for the same patient's image photographs across frontal and side views, and designed a hierarchical attention encoder to capture the linguistic structure of the medical report and the importance of different words and sentences to clinical diagnosis. Huang et al. used a keyword-driven approach to guide the descriptive content of medical image reports. However, the paradigms adopted by the above methods are mostly "single CNN+LSTM," which have limited ability to extract global semantic features of medical images. Therefore, this invention proposes the DBFFN model to address this problem, which extracts features from medical images from different perspectives, and more accurately identifies possible anomalies in medical images.

[0003] For images, low-level local features contain more geometric details, while high-level global features contain richer semantic information. To make fuller use of the information in images, feature fusion has become an important means to improve the performance of neural networks. Based on the differences in feature patterns, the fusion schemes proposed by Snoek and Chaib et al. can be divided into early fusion and late fusion. The former fuses features in the feature space, while the latter fuses features in the semantic space. Specifically, early fusion requires combining features from different modalities and representing them. Common strategies include concatenation and parallel strategies. Concatenation involves dimensionally concatenating existing features, such as the method used by Liu et al. Parallel strategies involve cumulatively combining features, such as the approach adopted by Yang et al. Unlike early fusion, late fusion learns different semantic representations from existing features, such as the strategy proposed by Yang et al.

[0004] Recently, some Transformer-based network architectures have achieved remarkable success in the field of computer vision. The multi-head attention module of Transformer networks gives them excellent feature learning capabilities. Furthermore, the positional encoding method in Transformers preserves global positional information well, thus visual Transformer models typically perform well in tasks involving modeling global image features. Inspired by this, this invention proposes a dual-branch feature fusion network for medical report generation (DBFFN) to improve the performance of automatic medical report generation. It leverages the expressive power of visual Transformers in global image positional information to learn global features of medical images, and utilizes the reliable perception of local details by convolutional neural networks to learn local features. Finally, it uses image features that fuse global and local information to generate more reliable medical image reports. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that traditional methods for automatically generating medical image reports only use convolutional neural networks to extract image features, focusing on local details while neglecting the extraction of global image features.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] S1. Extract local features from the image.

[0008] S2. Perform global feature extraction on the image.

[0009] S3. The global and local features extracted from S1 and S2 are sent to the multi-scale feature fusion module for fusion.

[0010] S4. Use the decoder to align the fused features from S3 with the text vectors in the Ground Truth using visual and textual features, and predict to generate a report.

[0011] For a given medical image To augment the input medical images and prevent overfitting, the model first... Perform data augmentation operations such as random rotation, flipping, adjusting brightness, contrast, and random affine transformations, and preprocess the data into a format that can be input into a neural network model.

[0012] Next It will be input into the global-local image feature extraction module, and the process can be summarized as formula (1).

[0013]

[0014] in, Represents a local feature extractor. Represents a global feature extractor. To extract the obtained local features, To extract the obtained global features, specifically, a pre-trained convolutional neural network is first loaded for transfer learning and used as a local feature extractor for the model. This invention uses ResNet101. ResNet101 is a residual network structure that introduces a "skip connection" structure, allowing the network model's parameters to propagate across one or more layers of the network during backpropagation.

[0015]

[0016] in, This represents the parameters passed to the current residual module. This is the result obtained after calculation using the mapping function. This is the output of the residual module, i.e. and To further enhance the expressive power of the network, the ReLU activation function is used in the ResNet structure to further calculate the output of the residual module. The ReLU activation function is shown in formula (3).

[0017]

[0018] Then, ViT is used as the global feature extractor. For the input image ViT first processes it into 2D sequence blocks. .in, Number of channels The number of patches. The length and width of the patch defined in the ViT network.

[0019] Next, the ViT model applies a linear transformation to the sequence blocks. Dimensionality reduction, i.e. , here This refers to the dimension value after dimensionality reduction. To preserve spatial location information between image patches, ViT introduces a location embedding method. ViT will embed the sequence blocks... The learnable category vectors corresponding to the sequence blocks as well as Location information Accumulation is performed to obtain the vector input to the Transformer encoder. .

[0020] The Transformer encoder consists of a multi-head attention module, a multilayer perceptron, and a layer normalizer. For the input... ViT uses different weight matrices within its multi-head attention module. This is used to extract multiple feature matrices from an image. , Represents the number of "heads".

[0021] The calculation process of the multi-head attention module can be summarized by formula (4-6).

[0022]

[0023]

[0024]

[0025] After obtaining the output of multi-head attention, ViT performs layer normalization and multilayer perceptron computation to obtain the final prediction output.

[0026] After obtaining the image features, they are input into the multi-scale feature fusion module for fusion. The process can be summarized as formula (7).

[0027]

[0028] in Represents MSFFM, The image features are fused. To describe the process in detail, firstly, the local features obtained from the two branches are adjusted. and global features This ensures that they remain consistent in dimensions, where This refers to the batch size during training. This refers to the number of feature dimensions in the last layer of the ResNet101 convolutional part. and These are the width and height of the feature matrix of the last layer of the ResNet101 convolutional part, respectively. The number of dimensions of the input MLP Head defined in the ViT network. This process can be described by formula (8).

[0029]

[0030] in and , This represents a specified dimension of the input feature fusion module. This is an intermediate dimension that is dynamically adjusted based on the input tensor.

[0031] After adaptively adjusting the dimensions of features at different scales, the global features are... Use activation function Enhance the nonlinearity of the network, where The corresponding activation function is Tanh. Next, for local features... and global features after operation Dimensional expansion and initial fusion are performed, as shown in formula (9).

[0032]

[0033] in, Represents a fully connected layer, used to... and The last dimension expands to twice its original size. Preliminary fusion features are obtained. Then, it is segmented along the last dimension and an activation function is applied. Perform another calculation. Here... The corresponding activation function is LogSoftmax, which can improve data stability and prevent overflow. The process is shown in formula (10).

[0034]

[0035] For what was obtained They were compared with the original ones. Further calculations are performed to obtain the fusion features.

[0036]

[0037] in This represents the Hadamard product of two matrices. At the end of the MSFFM module, the fused features obtained in the previous step are... Compared with the original global features and local features Each part is multiplied by a specified coefficient, and the sum is then input into the model's decoder to generate the text content of the medical image report. This process is shown in formula (12).

[0038]

[0039] in These are the fusion features. global features and local features The corresponding coefficient.

[0040] Finally, the model uses the decoder part of the Transformer network model as the decoder to receive image features. and text vectors from Ground Truth It learns the correspondence between the two different modalities through the internal attention module, performs cross-modal feature alignment, and generates the predicted medical image report. This process can be summarized as formula (13).

[0041]

[0042] in Represents the decoder structure. The generated medical image report. As shown in formula (14), during the model training process, the negative log-likelihood method is used as the loss function for loss calculation and model optimization.

[0043]

[0044] in, This represents the parameters in the model.

[0045] Finally, the training method for the medical image report generation method based on dual-branch feature fusion is as follows:

[0046] This invention was tested on an Ubuntu 20.04 operating system with a 12GB NVIDIA GeForce RTX 3060 GPU and CUDA 11.0 parallel computing architecture. The development tools used were PyCharm 2021, Python 3.7, and the PyTorch 1.7.1 training framework. During training, the input image was first preprocessed and resized to 256×256. The last two layers of the original ResNet101 network were removed, resulting in a direct output image feature matrix with dimensions 8×8×2048. The last two layers of the ViT network were also removed, resulting in an output image feature matrix with dimensions 256×2048. Additionally, the ViT network had 3 layers, 8 multi-head attention mechanisms, and a Dropout ratio of 0.2. The batch size was 8, the training run was 50 epochs, and the Adam optimizer was used with a learning rate of 5e-5.

[0047] Compared with existing technologies, the beneficial effects of this invention are:

[0048] 1) This invention proposes a medical image report generation method DBFFN based on dual-branch feature fusion, which achieves focused extraction of medical images from both global and local perspectives through two different image feature extraction branches.

[0049] 2) This invention designs a multi-scale feature fusion module (MSFFM) to fuse features obtained from two branches. The effectiveness of MSFFM is verified by comparing different fusion methods. Attached Figure Description

[0050] Figure 1 Here is a diagram of the DBFFN model structure;

[0051] Figure 2 Here is a diagram of the ResNet101 model structure;

[0052] Figure 3 Here is a diagram of the ViT model structure;

[0053] Figure 4 Here is a diagram of the MSFFM model structure;

[0054] Figure 5 Comparison of experimental results for different models on the IU-X-Ray dataset;

[0055] Figure 6 The ablation experiment results for each module on the IU-X-Ray dataset;

[0056] Figure 7 This is a schematic diagram of the CONCAT process;

[0057] Figure 8 Box plots of the three fusion methods on the BLEU-4 index;

[0058] Figure 9 Box plot of the three fusion methods on the METEOR metric;

[0059] Figure 10 Box plots of the three fusion methods on the ROUGE-L index;

[0060] Figure 11 Box plots of the three fusion methods on the CIDEr metric;

[0061] Figure 12 Visualize the experimental results. Detailed Implementation

[0062] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0064] Figure 1 A schematic diagram of the DBFFN network structure designed for this invention is shown below. Figure 1 As shown, the model can be divided into three parts: 1) a global-local image feature extraction module; 2) a multi-scale feature fusion module; and 3) a decoder module.

[0065] Figure 2 This is a schematic diagram of the ResNet101 network structure used in the local feature extractor. ResNet is a residual network structure that introduces a "skip connection" structure, allowing the network model parameters to propagate across one or more layers during backpropagation. This mechanism solves the problem of gradient vanishing or exploding as the network depth increases, thereby further optimizing the performance of deeper networks. This process can be simplified to Equation (1).

[0066]

[0067] in, This represents the parameters passed to the current residual module. This is the result obtained after calculation using the mapping function. This is the output of the residual module, i.e. and To further enhance the expressive power of the network, the ReLU activation function is used in the ResNet structure to further calculate the output of the residual module. The ReLU activation function is shown in formula (6).

[0068]

[0069] Figure 3 This is a schematic diagram of the ViT network structure used in the global feature extractor. (Example:) Figure 3 As shown, for the input image ViT first processes it into 2D sequence blocks. .in, Number of channels The number of patches. The length and width of the patch defined in the ViT network.

[0070] Next, the ViT model applies a linear transformation to the sequence blocks. Dimensionality reduction, i.e. , here This refers to the dimension value after dimensionality reduction. To preserve spatial location information between image patches, ViT introduces a location embedding method. ViT will embed the sequence blocks... The learnable category vectors corresponding to the sequence blocks as well as Location information Accumulation is performed to obtain the vector input to the Transformer encoder. .

[0071] The Transformer encoder consists of a multi-head attention module, a multilayer perceptron, and a layer normalizer. For the input... ViT uses different weight matrices within its multi-head attention module. This is used to extract multiple feature matrices from an image. , Represents the number of "heads".

[0072] The calculation process of the multi-head attention module can be summarized by formula (7-9).

[0073]

[0074]

[0075]

[0076] After obtaining the output of multi-head attention, ViT performs layer normalization and multilayer perceptron computation to obtain the final prediction output.

[0077] Figure 4 This is a schematic diagram of the MSFFM module designed for this invention. This model adjusts the local features obtained from the two branches respectively. and global features This ensures that they remain consistent in dimensions, where This refers to the batch size during training. This refers to the number of feature dimensions in the last layer of the ResNet101 convolutional part. and These are the width and height of the feature matrix of the last layer of the ResNet101 convolutional part, respectively. The number of dimensions of the input MLP Head defined in the ViT network. This process can be described by formula (10).

[0078]

[0079] in and , This represents a specified dimension of the input feature fusion module. This is an intermediate dimension that is dynamically adjusted based on the input tensor.

[0080] After adaptively adjusting the dimensions of features at different scales, the global features are... Use activation function Enhance the nonlinearity of the network, where The corresponding activation function is Tanh. Next, for local features... and global features after operation Dimensional expansion and initial fusion are performed, as shown in formula (11).

[0081]

[0082] in, Represents a fully connected layer, used to... and The last dimension expands to twice its original size. Preliminary fusion features are obtained. Then, it is segmented along the last dimension and an activation function is applied. Perform another calculation. Here... The corresponding activation function is LogSoftmax, which can improve data stability and prevent overflow. The process is shown in formula (12).

[0083]

[0084] For what was obtained They were compared with the original ones. Further calculations are performed to obtain the fusion features.

[0085]

[0086] in This represents the Hadamard product of two matrices. At the end of the MSFFM module, the fused features obtained in the previous step are... Compared with the original global features and local features Each part is multiplied by a specified coefficient, and the sum is then input into the model's decoder to generate the text content of the medical image report. This process is shown in formula (14).

[0087]

[0088] in These are the fusion features. global features and local features The corresponding coefficient.

[0089] Figure 5 This chart compares the DBFFN model with other excellent methods in the field of medical image report generation in recent years. The comparison confirms the reliability of the DBFFN model in the medical image report generation task.

[0090] Figure 6 This image shows a comparison of the results of ablation experiments on the two branches of the image feature extraction network—the global feature extraction branch and the local feature extraction branch—and three fusion methods. Figure 6 The first two lines lead to the conclusion that, when using only a single-branch network for medical image feature extraction, the local feature extractor ResNet101 outperforms the global feature extractor ViT across all metrics. Figure 6 Lines 3 to 5 demonstrate the performance of using a dual-branch network to extract medical image features. It can be observed that the CONCAT method performs poorly across various metrics, failing to achieve effective fusion of image features to meet the experimental objectives. The ADD method significantly outperforms the CONCAT method, surpassing the single-branch structures of ResNet101 and ViT in the BLEU score, but its fusion capability remains limited. In contrast, the MSFFM module designed in this invention achieves significant improvements across all metrics.

[0091] Figure 7 This is a schematic diagram of the CONCAT fusion method used in ablation experiments. The CONCAT method is as follows: Figure 7 As shown, the two feature vectors after scale-adaptive transformation are concatenated along the last dimension, and then the feature vectors are adjusted to the specified dimension through a fully connected layer. Finally, an activation layer is passed to enhance the nonlinearity of the network. In the CONCAT method, this invention uses LeakyReLU as the activation function.

[0092] Figures 8 to 11 The distribution of all data across the entire IU-X-Ray test set during 50 iterations of the three fusion methods used in the ablation experiment (ADD, CONCAT, and MSFFM) shows that the box distribution corresponding to MSFFM is more ideal and stable overall, which further verifies the effectiveness of the MSFFM structure.

[0093] Figure 12 Visualization results generated for medical image reports, such as Figure 12 As shown, the left side displays lung images, while the right side shows standard image reports (Ground Truth) from the dataset and medical image reports generated by the CONCAT, ADD, and MSFFM methods, respectively. The bolded and underlined sections describe abnormal areas, while the rest describe normal areas. It can be observed that the method designed in this invention can provide a more comprehensive and accurate description of both normal and abnormal areas in medical images.

[0094] This invention addresses the problem in traditional medical image report generation models that only use convolutional neural networks to learn semantic features of images, neglecting the extraction of global image features. It proposes the DBFFN model, which employs two different network branches to learn medical image features from both global and local perspectives, paying attention to both global and local details. Furthermore, the MSFFM method designed in this invention effectively fuses the image features obtained from the different branches. Results on the IU-X-Ray dataset confirm the effectiveness of the DBFFN model. Since the IU-X-Ray dataset used in this invention is limited to lung images and lacks other body parts, future work will focus on extending the automatic generation of image reports to other body parts.

[0095] Finally, the details of the above embodiments of the present invention are merely examples for explaining the present invention. For those skilled in the art, any modifications, improvements, and substitutions to the above embodiments should be included within the protection scope of the claims of the present invention.

Claims

1. A method for generating medical image reports based on dual-branch feature fusion, characterized in that, The method includes the following steps: S1. ResNet101 is used to extract local features from the image to obtain local features. ; S2. Use ViT to extract global features from the image to obtain global features. ; S3. Extract the global features from S1 and S2. and local features The data is sent to the multi-scale feature fusion module for fusion. The specific process of S3 is as follows: First, adjust the local features obtained from the two branches respectively. and global features This ensures that they maintain dimensional consistency, resulting in adjusted local features. and global features ,in This refers to the batch size during training. This refers to the number of feature dimensions in the last layer of the ResNet101 convolutional part. and These are the width and height of the feature matrix of the last layer of the ResNet101 convolutional part, respectively. The number of dimensions of the input MLP Head defined in the ViT network is given by the following formula: ， in and , This represents a specified dimension of the input feature fusion module. An intermediate dimension that is dynamically adjusted based on the input tensor; After adaptively adjusting the dimensions of features at different scales, the global features are... Use activation function Enhance the nonlinearity of the network, where The corresponding activation function is Tanh, followed by local features. and global features after operation The formula for dimensional expansion and initial fusion is as follows: ， in, Represents a fully connected layer, used to... and The last dimension expands to twice its original size, resulting in preliminary fusion features. Then, it is segmented along the last dimension and an activation function is applied. Perform another calculation, where The corresponding activation function is LogSoftmax, and the formula is as follows: ， For what was obtained They were respectively paired with Further calculations are performed to obtain the fusion features. The formula is as follows: ， in The Hadamard product of the two matrices represents the resulting fused features. With global features and local features After multiplying each feature by a specified coefficient and summing them, the fused image features are output. The formula is as follows: ， in These are the fusion features. Global features and local features The corresponding coefficient; S4. Use the decoder to process the fused image features from S3. Text vectors of images Align visual and textual features and predict and generate reports.

2. The medical image report generation method based on dual-branch feature fusion according to claim 1, characterized in that, The specific process of S1 is as follows: First, for a given medical image A pre-trained convolutional neural network is loaded for transfer learning and used as a local feature extractor for the model, using the following formula. ， in, This represents the parameters passed to the current residual module. This is the result obtained after calculation using the mapping function. This is the output of the residual module, i.e. and The ResNet101 architecture uses the ReLU activation function to further calculate the output of the residual module. The ReLU activation function is... 。 3. The medical image report generation method based on dual-branch feature fusion according to claim 1, characterized in that, The specific process of S2 is as follows: Using ViT as the global feature extractor, for the input image ViT first processes it into 2D sequence blocks. ,in, Number of channels The number of patches. The length and width of the patch are defined in the ViT network. Then, the ViT model applies a linear transformation to the sequence block. Dimensionality reduction, i.e. ,in To obtain the dimensionality value after dimensionality reduction, ViT introduces positional embedding, which converts sequence blocks... Sequence block Corresponding learnable category vector as well as Location information Accumulation is performed to obtain the vector input to the Transformer encoder. ; The Transformer encoder consists of a multi-head attention module, a multi-layer perceptron mechanism, and layer normalization. For the input... ViT uses different weight matrices within its multi-head attention module. This is used to extract multiple feature matrices from an image, where , The number of heads represents the multi-head attention module, and the calculation formula is as follows: ， ， ， After obtaining the calculation results of multi-head attention, ViT performs layer normalization and multi-layer perception mechanism calculations on them.

4. The medical image report generation method based on dual-branch feature fusion according to claim 1, characterized in that, The specific process of S4 is as follows: The decoder part of the Transformer network model is used as the model's decoder, which receives the fused image features. Text vectors corresponding to images It learns the correspondence between these two different modalities through an internal attention module, performs cross-modal feature alignment, and generates the text content of the final medical image report.

Images