Medical image report generation method
Through a multimodal alignment method based on disease knowledge graph, the problem of poor image-text alignment in medical image report generation is solved, and efficient, accurate and automatic generation of medical images and reports is achieved.
Patent Information
- Application Number
- CN202510165647.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2025-09-16
AI Technical Summary
Existing medical image report generation methods have poor image-text alignment effects in the context of data scarcity in the medical field, making it difficult to achieve efficient image-text alignment, which affects the quality of reports.
A multimodal alignment method based on disease knowledge graph is adopted. Through the image encoding module, text encoding module, image disease decoding module and text disease decoding module, combined with the contrastive learning method, global semantic alignment and disease semantic alignment of images and texts are performed, and medical reports are generated using Transformer encoder and decoder.
It achieves multi-dimensional alignment of medical images and reports, improves the accuracy and consistency of reports, and enhances the quality of automatically generated reports from images.
Smart Images
Figure CN120656631A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and natural language processing, and in particular to a method for generating a medical image report. Background Art
[0002] Automatic medical report generation technology is a technology that uses artificial intelligence and natural language processing technology to automatically extract key information from multiple sources such as electronic health records (EHR), imaging data, laboratory test results, etc., and combines it with medical knowledge bases and treatment guidelines to generate structured medical reports.
[0003] Existing methods typically involve two main components: extracting image features and generating text reports based on these features. Early approaches to image feature extraction relied on hand-crafted features and traditional image processing techniques. In recent years, deep learning, particularly convolutional neural networks (CNNs), has become the mainstream technology for extracting complex medical image features. For text generation, while early approaches relied on basic template filling techniques, advanced language models such as recurrent neural networks (RNNs) and Transformers are now increasingly used to generate more natural and fluent text.
[0004] Image-text alignment is an important research direction in computer vision and natural language processing. It aims to identify the relationship between the specific visual content in an image and the corresponding text description. The quality of image-text alignment has a decisive impact on the quality of image-to-text generation, and is of great significance for achieving deeper multimodal understanding, improving image caption generation, and enhancing interactive applications of images and text. Existing image-text alignment technologies often only perform coarse-grained alignment of images and text based on global features. To achieve good results, such alignment methods require a large amount of training data. Due to the scarcity of data in the medical field, a more efficient image-text alignment method is needed. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a medical image report generation method, which can improve the quality of medical image reports based on a more efficient image-text alignment method.
[0006] The present invention solves the technical problem by providing a method for generating a medical image report, comprising the following steps:
[0007] Obtaining a target medical image and a disease knowledge graph, wherein the disease knowledge graph includes symptoms in the target medical image and corresponding diseases;
[0008] Analyzing the target medical image using a medical report generation model to obtain a target medical report, the medical report generation model comprising:
[0009] Image encoding module, used to extract image feature vectors of input images;
[0010] A graph encoding module, configured to generate a disease embedding vector based on the disease knowledge graph;
[0011] an image disease decoding module, configured to generate image disease query information based on the image feature vector and the disease embedding vector;
[0012] The first text disease decoding module is used to generate a medical report according to the image disease query information.
[0013] Furthermore, the medical report generation model also includes a text encoding module for extracting text feature vectors of input text.
[0014] Furthermore, the medical report generation model is trained by the following method:
[0015] Obtaining several medical image samples and corresponding medical report samples;
[0016] inputting the medical image sample and the medical report sample into the image encoding module and the text encoding module respectively;
[0017] The corresponding medical image samples and the medical report samples are used as positive sample pairs, and the randomly combined medical image samples and the medical report samples are used as negative sample pairs. The contrastive learning method is used to perform global semantic alignment of the image feature vector and the text feature vector.
[0018] Furthermore, extracting the image feature vector of the input image includes:
[0019] Cutting the input image into image blocks of a set size and extracting image block position codes;
[0020] Performing feature mapping on each of the image blocks using a linear layer to obtain a plurality of first intermediate vectors;
[0021] Each first intermediate vector is added to the corresponding image block position code and then input into an image encoder to obtain an image feature vector of the input image as a first vector sequence composed of feature vectors of several image blocks.
[0022] Furthermore, extracting the text feature vector of the input text includes:
[0023] Cutting the input text into text units of a set size and extracting unit position codes;
[0024] Performing feature mapping on the text unit using an embedding layer to obtain a second intermediate vector;
[0025] The second intermediate vector is added to the corresponding unit position code and then input into a text encoder to obtain a text feature vector of the input text as a second vector sequence composed of feature vectors of several text units.
[0026] Furthermore, the global semantic alignment of the image feature vector and the text feature vector using a contrastive learning method includes:
[0027] Setting the global image feature vector of any medical image sample to the average value of the first vector sequence;
[0028] Setting the global feature vector of the text of any of the medical report samples to the average value of the second vector sequence;
[0029] Comparative learning training is performed based on the similarity between the image global feature vector and the text global feature vector to optimize the loss function of the medical report generation model.
[0030] Furthermore, the medical report generation model also includes a second text disease decoding module for generating text disease query information based on the text feature vector and the disease embedding vector.
[0031] Furthermore, after the step of performing global semantic alignment of the image feature vector and the text feature vector by contrastive learning, the method further includes:
[0032] Using the disease embedding vector as an input vector to the image disease decoding module and the second text disease decoding module;
[0033] Connecting the image feature vector and the text feature vector to the cross attention layer of the image disease decoding module and the second text disease decoding module respectively;
[0034] The disease feature information retrieved from the medical image sample and the medical report sample with the same disease information is used as a positive sample pair, and the disease feature information retrieved from the medical image sample and the medical report sample with different disease information is used as a negative sample pair. The contrastive learning method is used to perform disease semantic alignment of the medical report generation model.
[0035] Furthermore, the disease semantic alignment of the medical report generation model using contrastive learning method includes:
[0036] Comparative learning training is performed based on the similarity between the image disease query information and the text disease query information to optimize the loss function of the medical report generation model.
[0037] Furthermore, after the step of using the contrastive learning method to perform disease semantic alignment of the medical report generation model, the method further includes:
[0038] The deep learning stochastic gradient descent method is used to optimize the loss function of the medical report generation model.
[0039] Beneficial effects
[0040] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the existing technology: the present invention performs multi-dimensional image-text multimodal alignment on medical images and reports, applies image-text comparative learning from the two dimensions of global features and disease features, and uses disease knowledge graphs as a guide to promote medical images and reports to maintain a high degree of consistency in disease semantics, thereby improving the accuracy of automatically generated reports from medical images. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 is a flow chart of an embodiment of the present invention;
[0042] Figure 2 2. It is a schematic diagram of an image-text semantic alignment framework according to an embodiment of the present invention;
[0043] Figure 3 is a schematic diagram of an image-to-text generation framework according to an embodiment of the present invention;
[0044] Figure 4 This is a schematic diagram of a knowledge graph including 20 chest X-ray diseases and their relationships according to an embodiment of the present invention;
[0045] Figure 5 Schematic diagram of a disease semantic alignment framework according to an embodiment of the present invention. DETAILED DESCRIPTION
[0046] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.
[0047] The embodiment of the present invention relates to a method for automatically generating medical reports based on knowledge-enhanced multimodal alignment, such as Figure 1 As shown, it includes training process and usage process.
[0048] The training process consists of the following steps:
[0049] Step A1: Enter the model training state.
[0050] In step A2, medical image samples and medical report samples are collected to form a dataset. The collected medical image samples and medical report samples are cleaned to ensure that all medical image samples have corresponding medical report samples for text description, and all medical report samples have corresponding medical image samples. The dataset can be divided into training, validation, and test sets in a ratio of 80%, 10%, and 10%.
[0051] Step A3: Perform semantic alignment training of images and reports. Figure 2 The network framework of the medical report generation model is constructed in this way, which includes an image encoder and decoder based on the Transformer encoder structure, a knowledge graph containing 20 disease nodes (such as Figure 4 As shown in the figure), a graph encoder based on the graph attention mechanism, and an image / text disease decoder based on the Transformer decoder structure. The image is scaled and normalized and then fed into the image encoder, and the text is cleaned and segmented and then fed into the text encoder. Based on the image / text global feature sequences Y_I and Y_T obtained by the two encoders, the contrastive learning loss InfoNCE Loss is calculated to complete global alignment. The graph encoder is used to embed the knowledge graph nodes to obtain the disease query vector. The disease query vector is fed into the image disease decoder and the text disease decoder, which interact with the image global feature sequence Y_I and the text global feature sequence Y_T respectively to obtain the disease query information of the image and text. Based on the disease query information, the same disease is used as a positive sample pair and different diseases are used as a negative sample pair to calculate the contrastive learning loss InfoNCELoss. Model training is completed through deep learning stochastic gradient descent to achieve semantic alignment training of images and reports.
[0052] Step A4: Perform image-to-report generation training. Based on the trained image encoder, graph encoder, and image disease decoder, a text decoder based on the Transformer decoder structure is introduced, such as Figure 3 The disease information obtained by the image disease decoder is used as the input of the text decoder, and an autoregressive language model is trained on the target report, thereby achieving the final image-to-report generation process.
[0053] The usage process includes the following steps:
[0054] Step B1, enter the model use state.
[0055] Step B2: Obtain the medical image for the report to be output and input the medical image into the model.
[0056] Step B3: Use the image encoder to extract the feature Y_I of the medical image for subsequent use.
[0057] In step B4, the graph encoder is used to obtain the disease query vector, which is used as the input of the image disease decoder to interact with the image feature Y_I to obtain the disease feature.
[0058] In step B5, based on the obtained disease features, the features are sent to a text decoder to generate a report text and obtain a final medical report.
[0059] See also Figure 2 As shown, Figure 2 This paper presents an image-text semantic alignment framework for the automated medical report generation system. It includes an image encoder, a text encoder, a knowledge graph of 20 diseases and their relationships, a graph encoder, an image disease decoder, and a text disease decoder. The network framework performs two collaborative contrastive learning tasks: global semantic alignment and disease semantic alignment. Both tasks use the InfoNCE Loss function. The entire network completes 10 rounds of training on a complete medical image / report dataset.
[0060] See also Figure 3 As shown, Figure 3 This is the image-to-text generation framework for the automatic medical report generation system in an embodiment of the present invention. It includes an image encoder that has undergone alignment training, a knowledge graph of 20 diseases and their relationships, a graph encoder, and an image disease decoder. The image features Y_I obtained by the image encoder and the disease query vector D obtained by the knowledge graph encoder are fed into the image disease decoder to obtain the final image disease information. This disease information is then fed into the newly constructed text decoder, completing the image-to-text generation training task.
[0061] The following is a further explanation of the various modules of the medical report generation model and their working principles, including:
[0062] (1) Image encoder extracts image features.
[0063] The image encoder is based on a Transformer encoder and has a 12-layer transformer layer structure. Given a medical image I, image I is first pre-processed by scaling to the model's required size (resolution 224*224) and normalized. Next, image I is cut into multiple 16*16 patches. Each patch p_i is mapped to a high-dimensional feature vector x_i through a linear transformation layer E_I. To enable the model to process sequence information, each feature vector x_i is added with a positional encoding pos1_i, which helps the model understand the relative or absolute position of each patch. The entire sequence X_I is then input into a multi-layer Transformer encoder. Each encoder layer incorporates a self-attention mechanism and a feed-forward neural network, enabling the model to consider information from all patches in the sequence at each encoding layer, thereby capturing dependencies between image patches. After processing by the Transformer encoder, the output sequence Y_I contains rich contextual information that can be further used in subsequent global alignment tasks and disease feature extraction pipelines.
[0064] (2) The text encoder extracts text features.
[0065] Based on the Transformer encoder, it also has a 12-layer transformer layer structure. Given a medical report T, the text T is first cleaned to remove non-standard characters to adapt it to the model's input requirements. Next, the text T is segmented into multiple tokens, and each token t_i is mapped to a high-dimensional feature vector x_t through an embedding layer E_T. To preserve the sequential information in the text, each feature vector x_t is added to a positional encoding pos_t. Each positional encoding is used to help understand the relative or absolute position information of each word. The entire sequence X_T is then input into the Transformer encoder, which consists of multiple layers. Each encoder layer contains a self-attention mechanism and a feedforward neural network, allowing the model to consider the information of all words in the sequence at each encoding layer, thereby capturing the dependencies between words. After processing by the Transformer encoder, the output sequence Y_T is used in the subsequent global alignment task and disease feature extraction process.
[0066] (3) Global semantic alignment.
[0067] For the obtained image feature sequence Y_I and report text feature sequence Y_T, we average the sequences to obtain the global image feature F_I and report text feature F_T. For any image and report, we calculate the cosine similarity between the global feature F_I and F_T vectors to determine their similarity. We use the original image and report pair as positive sample pairs and several randomly combined image and report pairs as negative sample pairs. We calculate the contrastive learning loss InfoNCELoss and optimize this loss function to promote global semantic alignment between images and text.
[0068] (4) Disease semantic alignment.
[0069] Divide 20 common chest X-ray disease symptoms by tissue and construct a knowledge graph, such as Figure 4 As shown. Each of the 20 nodes is randomly initialized to a vector, and a graph encoder is constructed using the Graph Attention Network (GAT). The graph encoder will obtain the final embedding vector sequence D of the 20 disease nodes based on the vector value of the node and the edge association relationship between the nodes, so as to prepare for the subsequent use of the image feature sequence Y_I and the report text feature sequence Y_T to query the image and text disease information. Construct an image and text disease decoder based on the Transformer decoder structure, as shown Figure 5 As shown, the image / text decoder uses a sequence of disease embedding vectors (obtained by the graph encoder) as input to the self-attention layer and a sequence of image / text features as input to the cross-attention layer. After processing by the disease decoder, the corresponding features of the disease in the image / text are extracted (the features contain information such as the presence and location of the disease in the image and report). The disease embedding vector D is used as the input vector, and the output sequences Y_I and Y_T of the image encoder and text encoder are used as input to the cross-attention layer of the decoder to query disease information in the image and text respectively.
[0070] For a specific disease, the disease information retrieved in the image and the corresponding report text should be consistent. Therefore, the information pairs retrieved in the image and text for a certain disease are used as positive sample pairs, and the information pairs retrieved in the image and text for different diseases are used as negative sample pairs. The contrastive learning loss InfoNCE Loss is calculated, and the disease semantic alignment of the image and text is promoted by optimizing this loss function. Specifically, the image and the corresponding report text belong to different modal information, but contain the same disease semantics. In other words, the features of a disease in the image should be consistent with the features in the report (for example, the image has the image feature of "water accumulation in the left lung", and the report should also mention the text description of "water accumulation in the left lung"). However, due to the different modalities of the image and text, it is necessary to use contrastive learning methods to achieve the above-mentioned semantic consistency alignment.
[0071] Contrastive learning methods bring the semantic features of positive pairs closer together and push the semantic features of negative pairs further apart (by optimizing the InfoNCE Loss function). For the same disease, the information retrieved from both the image and text is a positive pair (for example, the semantically consistent feature information for "pulmonary edema" in the image and report should be considered a positive pair). Contrastive learning, on the other hand, brings the semantic features of positive pairs closer together and pushes the semantic features of negative pairs further apart, ultimately aligning the semantics of the disease between the image and text.
[0072] (5) Image-text semantic alignment training.
[0073] Based on the above-mentioned image encoder, text encoder, disease knowledge graph, graph encoder, image disease decoder, and text disease decoder, the network is trained by optimizing the two contrastive learning loss functions in global semantic alignment and disease semantic alignment through deep learning stochastic gradient descent.
[0074] (6) The text decoder generates a report.
[0075] After completing the semantic alignment training described above, a text decoder is implemented based on the Transformer decoder architecture. The disease information extracted by the image disease decoder is used as input to the text decoder, and the report generation task is trained using language model autoregressive training. The resulting image encoder, disease knowledge graph, graph encoder, image disease decoder, and text decoder are then used to automatically generate reports from medical images.
[0076] As can be seen above, this invention implements a system for automatically generating medical reports based on knowledge-enhanced multimodal alignment. By aligning medical images and text globally and utilizing a medical disease knowledge graph to align disease dimensions, this system achieves multimodal semantic unification of medical images and report text, improving the model's ability to extract medical image information. By extracting disease information from medical images, the system automatically generates reports, resulting in a model method capable of generating accurate medical reports based on medical images.
Claims
1. A method for generating a medical image report, characterized in that: The following steps are involved: Obtaining a target medical image and a disease knowledge graph, wherein the disease knowledge graph includes symptoms in the target medical image and corresponding diseases; Analyzing the target medical image using a medical report generation model to obtain a target medical report, the medical report generation model comprising: Image encoding module, used to extract image feature vectors of input images; A graph encoding module, configured to generate a disease embedding vector based on the disease knowledge graph; an image disease decoding module, configured to generate image disease query information based on the image feature vector and the disease embedding vector; The first text disease decoding module is used to generate a medical report according to the image disease query information.
2. The method according to claim 1, characterized in that The medical report generation model also includes a text encoding module for extracting text feature vectors of input text.
3. The method according to claim 2, characterized in that The medical report generation model is trained by the following method: Obtaining several medical image samples and corresponding medical report samples; inputting the medical image sample and the medical report sample into the image encoding module and the text encoding module respectively; The corresponding medical image samples and the medical report samples are used as positive sample pairs, and the randomly combined medical image samples and the medical report samples are used as negative sample pairs. The contrastive learning method is used to perform global semantic alignment of the image feature vector and the text feature vector.
4. The method according to claim 3, characterized in that The step of extracting an image feature vector of an input image includes: Cutting the input image into image blocks of a set size and extracting image block position codes; Performing feature mapping on each of the image blocks using a linear layer to obtain a plurality of first intermediate vectors; Each first intermediate vector is added to the corresponding image block position code and then input into an image encoder to obtain an image feature vector of the input image as a first vector sequence composed of feature vectors of several image blocks.
5. The method according to claim 4, characterized in that The step of extracting a text feature vector from an input text comprises: Cutting the input text into text units of a set size and extracting unit position codes; Performing feature mapping on the text unit using an embedding layer to obtain a second intermediate vector; The second intermediate vector is added to the corresponding unit position code and then input into a text encoder to obtain a text feature vector of the input text as a second vector sequence composed of feature vectors of several text units.
6. The method according to claim 5, characterized in that The global semantic alignment of the image feature vector and the text feature vector using a contrastive learning method includes: Setting the global image feature vector of any medical image sample to the average value of the first vector sequence; Setting the global feature vector of the text of any of the medical report samples to the average value of the second vector sequence; Comparative learning training is performed based on the similarity between the image global feature vector and the text global feature vector to optimize the loss function of the medical report generation model.
7. The method according to claim 3, characterized in that The medical report generation model also includes a second text disease decoding module for generating text disease query information based on the text feature vector and the disease embedding vector.
8. The method according to claim 7, characterized in that After the step of performing global semantic alignment of the image feature vector and the text feature vector by contrastive learning, the method further includes: Using the disease embedding vector as an input vector to the image disease decoding module and the second text disease decoding module; Connecting the image feature vector and the text feature vector to the cross attention layer of the image disease decoding module and the second text disease decoding module respectively; The disease feature information retrieved from the medical image sample and the medical report sample with the same disease information is used as a positive sample pair, and the disease feature information retrieved from the medical image sample and the medical report sample with different disease information is used as a negative sample pair. The contrastive learning method is used to perform disease semantic alignment of the medical report generation model.
9. The method according to claim 8, characterized in that The disease semantic alignment of the medical report generation model using a contrastive learning method includes: Comparative learning training is performed based on the similarity between the image disease query information and the text disease query information to optimize the loss function of the medical report generation model.
10. The method according to claim 8, characterized in that After the step of using contrastive learning method to perform disease semantic alignment of the medical report generation model, the method further includes: The deep learning stochastic gradient descent method is used to optimize the loss function of the medical report generation model.