Lightweight space adapter enhanced medical report generation method
Through the combination of global vision encoder and spatial adapter, the shortcomings of traditional models in understanding image content and capturing abnormal areas in medical report generation are solved, and a lightweight cross-media medical report generation framework is built, which improves the accuracy of report generation and efficiency.
Patent Information
- Application Number
- CN202411007351.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-26
AI Technical Summary
The traditional medical report generation model has shortcomings in understanding the content of medical images and capturing abnormal areas. Due to the large amount of parameters, the model is inefficient and cannot meet the professionalism and accuracy requirements of medical reports.
The global vision encoder is used to extract the global features of medical images, and perform multiple iterative updates through the spatial adapter. Combining cross-media pre-training model and report decoder, it enhances the fusion of vision and text, and builds a lightweight cross-media medical report generation frame.
Significantly reduces the amount of parameters of the pretrained model while maintaining superior report generation performance, allowing for more accurate understanding of medical image content, capturing abnormal areas, improving the accuracy of answer predictions, and reaching or exceeding with fewer parameters Effects of existing methods.
Smart Images

Figure CN119028508A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a deep neural network for medical report generation, and in particular to a medical report generation method enhanced by a lightweight spatial adapter. Background Art
[0002] Medical images are essential for routine diagnosis and treatment of patients. However, despite doctors’ extensive knowledge and experience, it is still time-consuming and error-prone for medical professionals to analyze large amounts of medical images and draft diagnostic reports based on clinical observations. To reduce the burden on medical professionals and improve clinical efficiency, it is essential to automate medical report generation. This technology not only simplifies the diagnostic process and reduces doctors’ workload, but also improves the consistency and accuracy of reports, thereby better facilitating patient care.
[0003] In the field of deep learning, image captioning is a key task that integrates computer vision and natural language processing techniques. Automatically generating descriptive captions for images represents an important research direction in artificial intelligence. The goal of image captioning is to describe images using natural language, with the aim of generating meaningful and grammatically correct text based on a comprehensive understanding of the visual content. The goal of medical reports is to generate detailed reports describing medical impacts, which is a specialized task of image captioning. The generation of medical reports requires not only the generation of longer texts, but also the generation of linguistically professional and accurate descriptions, which is crucial for clinical use. Traditional image captioning methods can no longer meet the requirements of medical report generation.
[0004] For medical cross-media tasks, it is crucial that the model captures overall image features as well as some rare but important details. According to the workflow of radiologists, when obtaining a patient's medical image, they first check the global area to identify abnormalities and check smaller and easily overlooked local areas, and then draft an accurate medical report based on medical knowledge and experience. However, traditional pre-trained models usually use independent Transformer encoders to encode images, which often leads to insufficient representation of spatial details. In addition, although automatic medical report generation and cross-modal pre-trained models have made significant progress, the increase in model parameter size accompanied by small performance improvements has hindered the further development and application of medical report generation.
[0005] In summary, the difficulty of medical report generation lies in how to correctly and effectively understand the content of medical images and accurately capture the abnormal areas in the images, while building a lightweight and efficient cross-media medical report generation framework to improve the accuracy of answer prediction. Summary of the invention
[0006] The present invention provides a lightweight cross-media method for medical report generation.
[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0008] Step (1) constructs a global visual encoder to extract global features of medical images, so that the model can more accurately capture some abnormal areas in the image.
[0009] Step (2): construct a spatial adapter based on the global visual encoder, so that the model can still maintain excellent text report generation performance with a small number of parameters, and use the spatial adapter for multiple rounds of iterative updates to obtain the most relevant visual information for report generation.
[0010] Step (3) uses a variety of medical images and corresponding text report matching tasks to build a cross-media pre-training model, which is used to enhance the fusion between vision and text reports by better aligning complex visual-text relationships under the action of contrastive learning.
[0011] Step (4), construct a report decoder and use the visual features output by the spatial adapter to generate a medical image diagnosis report.
[0012] Furthermore, the construction of the global visual encoder described in step (1) is used to extract the global features of the medical image to more accurately extract the features of the medical image, as follows:
[0013] 1-1. The global visual encoder consists of a patch embedding module and N blocks, each of which contains L Transformer encoding layers. The input medical image is initially input into the patch embedding module, where the image is divided into non-overlapping patches of size 16×16. Each patch is then flattened and enhanced by position embedding to incorporate spatial information. The patch then passes through L Transformer encoding layers and then through N blocks in sequence to obtain visual features. Throughout the process, the resolution of the features is reduced to 1 / 16 of the original medical image, and a marker is introduced to represent the global features of the original medical image.
[0014] The spatial adapter described in step (2) is constructed based on the global visual encoder, and multiple rounds of iterative updates are performed using the spatial adapter to obtain the visual information most relevant to report generation. The spatial adapter includes a spatial injector and a feature extractor, which are as follows:
[0015] 2-1. For the spatial injector: the feature map generated by the convolutional backbone network is integrated into the branch of the global visual encoder. Specifically, the spatial features extracted from the i-th feature extractor As keys and values, the features obtained from the global visual encoder As a query, the features after spatial feature injection are obtained, as shown in formula (1):
[0016]
[0017] where i∈[0,1,2,3,4] represents the index of the block in the global visual encoder, and represents the initial features derived from the convolutional backbone network, represents the output of the medical image after the patch embedding in the global visual encoder. LN(·) represents the layer normalization operation. crossAttn(·) represents the cross attention. λ i is a learnable vector used to adjust the output and input features of the attention layer in the i-th block balance between.
[0018] 2-2. For the feature extractor: the obtained global correlation features Input to the encoding layer of the i-th block, output feature next global related feature A feature extractor consisting of a criss-cross attention layer and a feed-forward network is then used to facilitate the interaction between the hidden layers of the global visual encoder and the spatial injector; in the criss-cross attention layer, the globally relevant features used as key and value inputs, while spatial detail features Used as query input; then the spatial detail features Processed by feed-forward network FFN to obtain the next spatial detail feature As shown in formula (2) and (3):
[0019]
[0020] Among them, the obtained spatial detail features It is then used as input to the subsequent spatial injector.
[0021] 2-3. After N rounds of interaction between the global visual encoder and the spatial adapter, the spatial detail features will be included and contains globally relevant features The fused visual features are input into the report decoder; they are normalized to produce the final visual feature representation F v , as shown in formula (4):
[0022]
[0023] Among them, Norm(·) represents the normalization operation.
[0024] Step (3) is as follows:
[0025] 3-1. In order to enhance the fusion of medical image I and the corresponding text report R, the similarity between the global features of image I and the text report R is first calculated, as shown in formulas (5) and (6):
[0026]
[0027] Among them, v cls 、w cls They refer to the global features of the medical image I and the global features of the text report R respectively.
[0028] 3-2. In order to encourage the model to make matching image report pairs closer in the representation space and alienate mismatched image report pairs, the contrastive learning scores from image to report and report to image are calculated, as shown in formulas (7) and (8):
[0029]
[0030] Among them, f i2r (I) f r2i (R) represents the contrastive learning scores from image to report and from report to image, respectively, and τ is a learnable temperature parameter. M represents the batch size, and m represents the mth sample in the batch. The contrastive loss between the calculated contrastive learning score and its true value is calculated as shown in formula (9):
[0031]
[0032] Where CE(·) represents the cross entropy loss, y i2r (I), y r2i (R) indicates true value.
[0033] 3-3. In order to determine whether a given image report pair is a match or a mismatch, an image report encoder consisting of a transformer block that includes a bidirectional self-attention layer combined with a feedforward network layer and a cross-attention mechanism for enhancing the integration of visual data and optimizing the encoder for multimodal information processing is used to generate multimodal embedding features. A softmax function is then applied to predict the two-class probability p irm, As shown in formula (10):
[0034] L IRM =CE(y irm ,p irm (I,R)) (Formula 10)
[0035] where y irm Represents the one-hot encoding of the true label.
[0036] Step (4) constructs a report decoder to generate a medical image diagnosis report using the visual features from the adapter enhanced spatial visual feature encoder, as follows:
[0037] 4-1. To facilitate the generation of diagnostic reports from medical images, the report decoder uses L layers of Transformer decoding layers to build an autoregressive report model. The decoder uses a cross-attention mechanism to integrate multimodal embeddings and uses a sequence start marker to start the generation process. A sequence end marker indicates that the report generation is complete. The report decoder generates a report corresponding to the provided medical image. The cross-entropy loss is used to guide the model to maximize the possibility of generating a report through autoregressive training. The report token r generated at the previous time step t-1 is used. 1:t-1 ={r1,...,r t-1} Input to causal self-attention, criss-cross attention and feed-forward layers to predict the report tag at time step t As shown in formulas (11), (12), and (13):
[0038] h att =LN(CMSA(r 1:t-1 )+r 1:t-1 ) (Formula 11)
[0039] h ca =LN(CA(h att ,f v )+h att ) (Formula 12)
[0040]
[0041] Where CMSA(·) and CA(·) represent causal multi-head self-attention and cross attention respectively. The entire autoregressive generation process in the inference stage is shown in formula (14):
[0042]
[0043] in It is the word generated before time step t and serves as the input tag when generating the tth word.
[0044] The training goal is to minimize the negative log-likelihood of P(R|I) given the image features and the target report. The predicted label is calculated by and true report t The cross entropy loss between can promote the training process of the language model. The training objective is shown in formula (15):
[0045]
[0046] The overall objective function of the model is a combination of the above three pre-training loss functions, as shown in formula (16):
[0047] L all =L IRC +L IRM +L LM (Formula 16).
[0048] The beneficial effects of the present invention are as follows:
[0049] The present invention extracts global spatial features through a global visual encoder, so that the model can extract complete image information, and further updates the visual features through a spatial adapter based on the global features. The local spatial features from the convolutional backbone network and the global features from the visual transformer are iteratively updated through multiple rounds of spatial injectors and feature extractors in the spatial adapter to obtain the most relevant visual information for report generation, achieve richer and more comprehensive feature representation, and generate a medical image diagnosis report through a report decoder. The number of parameters of the pre-trained model is significantly reduced while maintaining superior report generation performance. Finally, through cross-media pre-training construction, the image and text modalities are better aligned, and the complex visual-text relationship in medical images is processed to enhance the fusion between vision and text. The present invention can obtain results comparable to or even better than existing methods with fewer trainable parameters.
[0050] In summary, the present invention can correctly and effectively understand the content of medical images and accurately capture abnormal areas in the images, while constructing a lightweight and efficient cross-media medical report generation framework, thereby improving the accuracy of answer prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention.
[0052] Figure 2 It is a space adapter constructed in the method of the present invention.
[0053] Figure 3 It is the cross-media pre-training constructed in the method of the present invention. DETAILED DESCRIPTION
[0054] The detailed parameters of the present invention are further described in detail below.
[0055] like Figure 1 As shown, the present invention provides a medical report generation framework enhanced by a lightweight spatial adapter.
[0056] The construction of the global visual encoder described in step (1) is used to extract the global features of the medical image to more accurately extract the features of the medical image, and is specifically as follows:
[0057] 1-1. Extracting global image features Input it into the spatial adapter and use the input as the query and key value of the spatial adapter in turn.
[0058] For the extraction of global medical image data features, the input image features are first cropped into a 224×224 pixel image, first input to the patch embedding module, divided into 16×16 non-overlapping patch blocks, and then flattened and input to the global feature encoder, which contains 4 blocks, each with 3 Transformer encoding layers, which uses a large-scale pre-trained neural network ViT-base to extract the features of the input image and obtain the global image features. in d = 768 is the dimension of the feature. A [CLS] tag is used as the overall representation of the global image feature for subsequent comparison input of medical image and text report pre-training. Next, the global features of the medical image are input into the spatial adapter, and the input is used as the query and key value of the spatial adapter.
[0059] The spatial adapter described in step (2) is constructed based on the global visual encoder, and multiple rounds of iterative updates are performed using the spatial adapter to obtain the visual information most relevant to report generation. The spatial adapter includes a spatial injector and a feature extractor, such as Figure 2 As shown, the details are as follows:
[0060] 2-1. Refer to the existing ResNet framework to build a convolutional backbone network similar to it, which includes convolutional layers, pooling layers, Relu activation functions, etc., and finally obtain the local image features of the spatial dimension d = 768 is the dimension of the feature. Then the feature output by the feature extractor in the i-th block is as keys and values and the features from the global visual encoder branch As a query input to the spatial injector in the spatial adapter, the cross attention is calculated, where i∈[0,1,2,3,4] represents the index of the block in the global visual encoder, and represents the initial features derived from the convolutional backbone network, represents the output of the medical image after the patch embedding in the global visual encoder. A learnable vector λ is set in the calculation process i To adjust the output of the attention layer in the i-th block and the global image features λ i The initial value is set to 0, and finally the characteristics after spatial injection are obtained
[0061] 2-2. For the feature extractor: It contains a cross-attention layer and a feed-forward network layer. We will Input to the encoding layer in the i-th block and output features Then Input to the feature extractor to promote further interaction between the hidden layer of the global visual encoder and the spatial injector. In the cross attention layer, are used as key and value inputs, while local image features It is used as the query input and the cross attention is calculated to obtain Then Processed through a feedforward network to obtain features after a single interaction
[0062] 2-3. We will interact with the spatial details after 4 rounds and contains global related information Add and normalize their sum, and then input the fused feature as the input of the report decoder.
[0063] Step (3) Figure 3 As shown, the details are as follows:
[0064] 3-1. Both the medical image feature and the text report feature contain a [CLS] tag to represent the global feature, and then a dot product operation is performed on the two to obtain the similarity of the two global features.
[0065] 3-2. By calculating the contrastive learning scores of medical images to text reports and reports to images with a batch size of M, matching image report pairs are brought closer in the representation space, while mismatched image report pairs are moved away from each other in the representation space. We set the initial value of the temperature parameter τ of the learning to 0.07 and limit its value range after learning to 0.001 to 0.5.
[0066] 3-3. After the visual features are input into the image report encoder and fused with the report features to obtain the fused features, the fused features are mapped to d = 2 dimensions and the softmax is calculated to obtain the two-category probability to determine whether the input medical image features and the text report features match. When calculating the cross entropy loss, the non-matching pairs are set to 0 and the matching pairs are set to 1 for the true value similarity.
[0067] Step (4) constructs a report decoder to generate a medical image diagnosis report using the visual features from the adapter enhanced spatial visual feature encoder, as follows:
[0068] 4-1. The model uses a 12-layer Transformer decoding layer as a decoder and integrates multi-modal embeddings through a cross-attention mechanism. <bos>and <eos>The two tags serve as the start and end tags of report generation. The decoder generates the tth tag by inputting the t-1 report tag generated before time step t-1 into the causal attention layer, cross attention layer, feedforward layer, etc. in an autoregressive manner. The generated report and the true target report are used to calculate the cross entropy loss, and the generated report and the true target report are minimized to optimize the model. In the downstream task, only the spatial adapter part is trained. For the commonly used medical image report dataset IU X-Ray, it is compared with the existing methods through BLEU, METEOR, ROUGE-L, and CIDEr. From Table 1, it can be observed that this method can still obtain comparable or even better results than the existing methods when the number of trainable parameters is halved or even quartered.
[0069] Table 1
[0070] method BLEU-4 METEOR ROUGE-L CIDEr The number of trainable parameters <![CDATA[M 2 Transformer]]> 0.143 0.170 0.328 0.332 - R2Gen 0.165 0.187 0.371 - 90.8M RATCHET 0.163 0.183 0.356 0.603 51M This method 0.167 0.200 0.373 0.607 21M
[0071] In summary, the present invention can correctly and effectively understand the content of medical images, accurately capture abnormal areas in images, and build a lightweight and efficient cross-media medical report generation framework, thereby improving the accuracy of answer prediction. And with fewer trainable parameters, the results are comparable to or even better than those of existing methods.< / eos> < / bos>
Claims
1. A lightweight spatial adapter enhanced medical report generation method, characterized in that: The steps include: Step (1), constructing a global visual encoder to extract global features of medical images so that the model can more accurately capture some abnormal areas in the image; Step (2), constructing a spatial adapter based on the global visual encoder so that the model can still maintain excellent text report generation performance with a smaller number of parameters, and using the spatial adapter for multiple rounds of iterative updates to obtain the visual features most relevant to report generation; Step (3), using a variety of medical images and corresponding text report matching tasks to build a cross-media pre-trained model, which is used to enhance the fusion between vision and text reports by better aligning complex visual-text relationships under the effect of contrastive learning; Step (4), construct a report decoder and use the visual features output by the spatial adapter to generate a medical image diagnosis report.
2. The method for generating a medical report enhanced by a lightweight spatial adapter according to claim 1, characterized in that: The global visual encoder described in step (1) is constructed to extract global features of medical images, as follows: 1-1. The global visual encoder consists of a patch embedding module and N blocks, each of which contains L Transformer encoding layers; the input medical image is initially input into the patch embedding module, where the medical image is divided into non-overlapping patches of size 16×16; Each patch is then flattened and augmented with position embedding to incorporate spatial information; the patch then passes through L Transformer encoding layers, which in turn pass through N blocks to obtain visual features; During the whole process, the resolution of the features is reduced to 1 / 16 of the original medical image, and a marker is introduced to represent the global features of the original medical image.
3. According to the method for generating medical reports enhanced by a lightweight spatial adapter as described in claim 1, the step (2) of building a spatial adapter based on a global visual encoder and performing multiple rounds of iterative updates using the spatial adapter to obtain visual information most relevant to report generation is characterized in that: The space adapter includes a space injector, as follows: The feature maps generated by the convolutional backbone network are integrated into the branches of the global visual encoder; specifically, the spatial features extracted from the i-th feature extractor are specified As keys and values, the features obtained from the global visual encoder As a query, the features after spatial feature injection are obtained, as shown in formula (1): where i∈[0,1,2,3,4] represents the index of the block in the global visual encoder, and represents the initial features derived from the convolutional backbone network, represents the output of the medical image after the patch embedding in the global visual encoder; LN(·) represents the layer normalization operation; crossAttn(·) represents the cross attention; λ i is a learnable vector used to adjust the output and input global correlation features of the attention layer in the i-th block balance between.
4. The method for generating a medical report enhanced by a lightweight spatial adapter according to claim 3, characterized in that: The spatial adapter also includes feature extractors, as follows: The obtained global correlation features Input to the encoding layer of the i-th block, output feature next global related feature A feature extractor consisting of a criss-cross attention layer and a feed-forward network is then used to facilitate the interaction between the hidden layers of the global visual encoder and the spatial injector; in the criss-cross attention layer, the globally relevant features used as key and value inputs, while spatial detail features Used as query input; Then the spatial detail features Processed by feed-forward network FFN to obtain the next spatial detail feature As shown in formula (2) and (3): Among them, the obtained spatial detail features It is then used as input to the subsequent spatial injector.
5. The method for generating a medical report enhanced by a lightweight spatial adapter according to claim 4, characterized in that: After N rounds of interaction between the global visual encoder and the spatial adapter, the spatial detail features will be included. and contains globally relevant features The fused visual features are input into the report decoder; they are normalized to produce the final visual feature representation F v , as shown in formula (4): Among them, Norm(·) represents the normalization operation.
6. The method for generating a medical report enhanced by a lightweight spatial adapter according to claim 5, characterized in that: Step (3) is as follows: 3-1. In order to enhance the fusion of the medical image I and the corresponding text report R, the similarity between the global features of the medical image I and the global features of the text report R is first calculated, as shown in formulas (5) and (6): s(I,R)=v cls T w cls (Formula 5) s(R,I)=w cls T v cls (Formula 6) Among them, v cls 、w cls They refer to the global features of the medical image I and the global features of the text report R respectively; 3-2. Calculate the contrastive learning scores from image to report and from report to image, as shown in formulas (7) and (8): Among them, f i2r (I) f r2i (R) represents the contrastive learning scores from image to report and from report to image, respectively, τ is a learnable temperature parameter; M represents the batch size, m represents the mth sample in the batch; the contrastive loss between the calculated contrastive learning score and its true value is calculated, as shown in formula (9): Where CE(·) represents the cross entropy loss, y i2r (I), y r2i (R) indicates the true value; 3-3. In order to determine whether a given image report pair is a match or a mismatch; an image report encoder consisting of a transformer block that includes a bidirectional self-attention layer combined with a feedforward network layer and a cross-attention mechanism for enhancing the integration of visual data and optimizing the encoder for multimodal information processing is used to generate multimodal embedding features; a softmax function is then applied to predict the second-class probability p irm , as shown in formula (10): L IRM = CE(y irm , p irm (I, R)) (Formula 10) Among them, y irm Represents the one-hot encoding of the true label.
7. The method for generating a medical report enhanced by a lightweight spatial adapter according to claim 6, characterized in that: Step (4) is as follows: 4-1. The report decoder uses L layers of Transformer decoding layers to build an autoregressive report model; the decoder uses a cross-attention mechanism to integrate multimodal embeddings and uses a sequence start marker to start the generation process; A sequence end marker indicates that the report generation is complete; the report decoder generates a report corresponding to the provided medical image; the model is guided by cross entropy loss through autoregressive training to maximize the possibility of generating a report; the report marker r generated at the previous time step t-1 is passed 1:t-1 ={r1,...,r t-1 } Input to causal self-attention, criss-cross attention and feed-forward layers to predict the report tag at time step t As shown in formulas (11), (12), and (13): h att =LN(CMSA(r 1:t-1 )+r 1:t-1 ) (Formula 11) h ca =LN(CA(h att ,F v )+h att ) (Formula 12) Among them, CMSA(·) and CA(·) represent causal multi-head self-attention and cross attention respectively; the entire autoregressive generation process in the inference stage is shown in formula (14): in is the word generated before time step t and serves as the input tag when generating the tth word; The training goal is to minimize the negative log-likelihood of P(R|I) given the image features and the target report; the predicted label is calculated by and true report t The cross entropy loss between can promote the training process of the language model. The training objective is shown in formula (15): The overall objective function of the model is a combination of the above three pre-training loss functions, as shown in formula (16): L all =L IRC +L IRM +L LM (Formula 16).
Citation Information
Cited By
Cell image classification method based on morphological semantic guidance
CN121937808A
A cell image classification method based on morphological semantic guidance
CN121937808B
