Learable retrieval enhancement-based radiology report generation method for visual text alignment and fusion

Abstract

The invention discloses a learnable radiology report generation method based on visual text alignment and fusion of retrieval enhancement. The method comprises the steps of collecting and respectively constructing a model training data set and an auxiliary data set, then constructing a visual text alignment and fusion model based on retrieval enhancement, and inputting the model training data set and the auxiliary data set into the visual text alignment and fusion model based on retrieval enhancement together for training. Constructing an inference model according to the trained visual text alignment and fusion model based on retrieval enhancement; and inputting the to-be-detected medical image and the auxiliary data set into the reasoning model for processing to obtain a radiology report corresponding to the to-be-detected medical image. According to the method, retrieval correlation is enhanced, meanwhile, a fine-grained vision-text alignment and fusion method is adopted to align and fuse features, and the problem that fine-grained region-sentence alignment is difficult due to weak supervision of an image report level in the medical report generation process is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic radiology report generation, and particularly relates to a radiology report generation method based on retrieval-enhanced visual-text alignment and fusion. BACKGROUND

[0002] Radiology reports play a crucial role in clinical diagnosis and treatment, however, manually generating radiology reports is time-consuming and highly dependent on radiologists' expertise, often leading to delays and inconsistencies in medical decision-making. Traditional radiology report generation methods usually treat it as an extension of image description tasks, however, unlike image description tasks, radiology reports need to prioritize the description of abnormal findings, which are often subtle and imbalanced compared to normal details, leading to models easily ignoring critical diagnostic clues. Moreover, radiologists usually generate reports sentence by sentence, each describing imaging patterns based on regional abnormal findings, while this information is not provided in the training data. The lack of detailed region-sentence correspondence in the training data leads to insufficient visual-text alignment and fusion during report generation.

[0003] In recent years, some methods have attempted to address these issues by introducing external knowledge or integrating reference reports through retrieval algorithms. For example, R2Gen generates radiology reports through a memory-driven Transformer model, utilizing a cross-modal memory network (CMN) to enhance the fusion of image and text data; PromptMRG introduces an auxiliary multi-disease classification task to improve diagnostic accuracy and combines retrieval algorithms to enhance report generation quality. Although these methods have improved the quality of report generation to some extent, they still face challenges such as disease class imbalance and insufficient cross-modal fusion. Especially when dealing with long-tail distribution of disease categories, models often struggle to accurately capture the characteristics of rare diseases, leading to a decline in the quality of generated reports.

[0004] In addition, KiUT proposes a knowledge-injected U-Transformer model that learns multi-level visual representations and adaptively extracts context and clinical knowledge to achieve accurate word prediction; EKAGen converts expert reports into embedding space to prevent loss of key features and enhance attention to key regions. Although these methods have made some progress in handling visual-text alignment, they often ignore the fine-grained region-sentence correspondence, leading to the lack of accurate detailed descriptions in the generated reports.

[0005] In view of the practical value of radiology report generation research and the above limitations, it is difficult to find a way to extract robust and discriminative multi-modal features in the scenario of any missing modal. If this effect is to be achieved, two problems must be solved, one is how to solve the problem of class imbalance, and the other is how to more fully perform cross-modal fusion. SUMMARY

[0006] In order to solve the problems in the background art, the present application provides a learnable radiology report generation method based on retrieval enhancement visual text alignment and fusion, which solves the technical problem that the prior art ignores the fine-grained region-sentence correspondence relationship when generating radiology reports.

[0007] The technical scheme of the radiology report generation method adopted by the present application comprises the following steps:

[0008] S1, a plurality of medical images and the corresponding radiology reports of each medical image are collected, the medical images are taken as input data, and the corresponding radiology reports of the medical images are taken as labels to construct a model training data set; then all the collected medical images and the corresponding radiology reports of each medical image are summarized to obtain an auxiliary data set.

[0009] S2, in a computer, a visual text alignment and fusion model based on retrieval enhancement is constructed, the model training data set and the auxiliary data set are input into the visual text alignment and fusion model based on retrieval enhancement for training, and a trained visual text alignment and fusion model based on retrieval enhancement is obtained.

[0010] S3, an inference model is constructed according to the trained visual text alignment and fusion model based on retrieval enhancement.

[0011] S4, the medical image to be tested is collected, the medical image to be tested and the auxiliary data set are input into the inference model for processing, and the corresponding radiology report of the medical image to be tested is obtained.

[0012] The visual text alignment and fusion model based on retrieval enhancement comprises an input encoding module, a first retrieval enhancement module, a visual text alignment and fusion module, an auxiliary multi-label long-tail classification module and an output decoding module; the medical images in the model training data set are input into the input encoding module for processing, and disease class score features F logit , visual features F v and local text prompt features The disease class score features F kogit are input into the first retrieval enhancement module for processing, the radiology reports in the auxiliary data set are input into the first retrieval enhancement module for processing, and the global reference prompt features visual feature F v , local text prompt feature and global reference prompt feature are input into a visual text alignment and fusion module for processing to obtain a fused visual text feature F c , a visual feature F v and a radiology report in an auxiliary data set are also input into an auxiliary multi-label long-tail classification module for processing to obtain a prompt feature X, and the prompt feature X and the fused visual text feature F c are input into an output decoding module for processing to obtain a radiology report, which is taken as the output of the retrieval-enhanced visual text alignment and fusion model.

[0013] The input encoding module includes a visual encoder and a first MedKLIP model encoder; medical images in a model training data set are input into the visual encoder and the first MedKLIP model encoder respectively, the visual encoder processes to obtain a visual feature F v , and the first MedKLIP model encoder processes to obtain a disease category score feature F logit and a local text prompt feature The visual feature F v is input into an auxiliary multi-label long-tail classification module for processing, the disease category score feature F logit is input into a first retrieval enhancement module and a fusion module for processing, and the visual feature F v and the local text prompt feature are input into a visual text alignment and fusion module for processing.

[0014] The auxiliary multi-label long-tail classification module adopts a multi-label classification model; the output decoding module adopts a Bert network model; and the visual encoder adopts a Resnet network model.

[0015] The first retrieval enhancement module includes a CheXbert module, a first hyperbolic neural network, a ranking metric learning module and a second MedKLIP model encoder; the disease category score feature F logitThe input is processed in the first hyperbolic neural network to obtain hyperbolic curve features, and then the distance between each adjacent hyperbolic curve feature is calculated by using a Poincare ball model, so as to obtain a hyperbolic distance matrix corresponding to all hyperbolic curve features; the radiology report in the auxiliary data set is input into a CheXbert module for processing to obtain a disease classification label vector corresponding to each radiology report, and then the Hamming distance between each two adjacent disease classification label vectors is calculated, so as to obtain a Hamming distance matrix corresponding to all disease classification label vectors; the obtained hyperbolic distance matrix and Hamming distance matrix are input into a ranking metric learning module for processing to obtain an index vector, and the radiology report corresponding to the index vector is screened out from the auxiliary data set according to the index vector; the screened radiology report is input into a second MedKLIP model encoder for processing to obtain a global reference prompt feature The obtained global reference prompt feature The obtained global reference prompt feature is input into a visual-text alignment and fusion module for processing.

[0016] The specific processing process of the ranking metric learning module is as follows: each row in the Hamming distance matrix is sorted in ascending order to obtain a sorted index vector; the hyperbolic distance matrix is taken as the prediction value of the ranking metric learning module, and the sorted index vector is taken as the target value of the ranking metric learning module, the cross-entropy loss between the hyperbolic distance matrix and the index vector is calculated, and the ranking metric learning module is continuously trained in the process of training the visual-text alignment and fusion model based on retrieval enhancement according to the cross-entropy loss.

[0017] The visual-text alignment and fusion module is set according to the following formula:

[0018]

[0019] Wherein, F c represents the fused visual-text feature, and also represents the output of the visual-text alignment and fusion module; represents the feature fusion of the connection of two features along the feature dimension; f proj () represents the first linear layer; f proj2 () represents the second linear layer; LayerNorm() represents the layer normalization layer; F v represents the visual feature; represents the region information of the sentence in the radiology report corresponding to the medical image; represents the region information of the disease category corresponding to the medical image; MPSA1() and MPSA1() represent the first and second multi-prompt Sinkhorn attention mechanisms, respectively; () T represents the transpose; φ1(), φ2(), φ3() and φ4() represent the first, second, third and fourth linear layers, respectively; represents a global reference prompt feature; represents a local text prompt feature.

[0020] The loss function of the retrieval-enhanced visual-text alignment and fusion model is set according to the following formula:

[0021]

[0022] wherein, represents the loss function of the retrieval-enhanced visual-text alignment and fusion model; represents the task loss obtained by the output decoding module; represents the cross-entropy loss obtained by the ranking metric learning module; represents the consistency loss obtained by the visual-text alignment and fusion module; and α and β represent balance coefficients; i represents an index; p(r i |r 1:i-1 ) represents the probability of the i-th word piece r i-1 of the output radiology report under the premise that the model outputs the r1,…,r i word pieces in the radiology report; and T represents the total number of word pieces in the radiology report; represents a hyperbolic distance matrix; Π represents an index vector obtained by ranking a Hamming distance matrix; CE() represents a cross-entropy loss; n and m represent indices; N represents the total number of sentences in the radiology report; and M represents the total number of disease categories in the medical image; represents a global reference prompt feature of the n-th sentence in the radiology report; represents a local text prompt feature of the m-th disease category in the medical image; <·,·> represents an inner product; ||·||2 represents an L2 norm; and IoU() represents an intersection over union; represents the region information of the n-th sentence in the radiology report in the medical image; represents the region information of the m-th disease category in the medical image.

[0023] The inference model is only to replace the first retrieval-enhanced module in the trained retrieval-enhanced visual-text alignment and fusion model with a second retrieval module; the second retrieval module comprises a newly added third MedKLIP model encoder, a second hyperbolic neural network and a trained first hyperbolic neural network, and a second MedKLIP model encoder; the disease category score feature F logitThe hyperbolic features are obtained by inputting into a first hyperbolic neural network for processing, and then the distance between each adjacent hyperbolic feature is calculated by using a Poincare ball model, so as to obtain a hyperbolic distance matrix corresponding to all hyperbolic features; the radiology report in the auxiliary database is input into a third MedKLIP model encoder to obtain disease category score features of the auxiliary database Disease category score features of the auxiliary database The auxiliary hyperbolic features are obtained by inputting into a second hyperbolic neural network for processing, and then the distance between each adjacent auxiliary hyperbolic feature is calculated by using a Poincare ball model, so as to obtain an auxiliary hyperbolic distance matrix corresponding to all auxiliary hyperbolic features; the distance between each hyperbolic distance matrix and each auxiliary hyperbolic distance matrix is calculated, the auxiliary hyperbolic distance matrix with the smallest distance between the hyperbolic distance matrix and the auxiliary hyperbolic distance matrix is screened out, and the radiology report corresponding to the auxiliary data set of the to-be-tested medical image is obtained according to the screened auxiliary hyperbolic distance matrix; the screened radiology report is input into a second MedKLIP model encoder for processing to obtain global reference prompt features Global reference prompt features obtained Input into a visual text alignment and fusion module for processing.

[0024] The present application has the following advantages:

[0025] The method mainly focuses on the attention area information and important spatial details ignored by a learnable retrieval enhancer, enhances retrieval enhancement correlation, and simultaneously adopts a fine-grained visual-text alignment and fusion method to align and fuse features, so as to solve the difficulty of fine-grained region-sentence alignment caused by weak supervision of image report levels in the medical report generation process. The combination of the two makes the model achieve SOTA effect on MIMIC-CXR and IU X-Ray data sets, surpasses other existing models, obtains the best segmentation effect, and realizes the current SOTA. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 It is a visual text alignment and fusion model based on retrieval enhancement of the present application.

[0027] Figure 2 It is a reasoning model of the present application.

[0028] Figure 3 It is a visual text alignment and fusion module of the present application. DETAILED DESCRIPTION

[0029] The application will be described in greater detail in connection with the accompanying drawings and embodiments, but the application is not limited thereto. Those skilled in the art can make several improvements and refinements without departing from the principles of the application, and these improvements and refinements are also considered within the scope of protection of the application. The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

[0030] Embodiment 1

[0031] As shown in Figure 1 and Figure 2 , the radiology report generation method of the embodiment includes the following steps:

[0032] S1, a plurality of medical images and a radiology report corresponding to each medical image are collected, a medical image is taken as input data, and a radiology report corresponding to the medical image is taken as a label of output data to construct a model training data set; and all the collected medical images and the radiology report corresponding to each medical image are summarized to obtain an auxiliary data set.

[0033] The radiology report corresponding to each medical image is a professional document written based on analysis of the medical image. In a specific implementation, the auxiliary data set is stored in a reference report database (Reference Report Database, DB). Each medical image includes region information corresponding to a plurality of disease categories; the radiology report corresponding to each medical image is composed of a plurality of sentences, and each sentence is composed of a plurality of word units.

[0034] S2, in a computer, a visual text alignment and fusion model based on retrieval enhancement is constructed, the model training data set and the auxiliary data set are input into the visual text alignment and fusion model based on retrieval enhancement for training, and a trained visual text alignment and fusion model based on retrieval enhancement is obtained.

[0035] The visual text alignment and fusion model based on retrieval enhancement includes an input encoding module, a first retrieval enhancement module (LRE), a visual text alignment and fusion module (FVTAF), an auxiliary multi-label long-tailed classification module (Auxiliary Multi-label Long-tailed Classification), and an output decoding module (Decoder); the medical image in the model training data set is input into the input encoding module for processing, and disease category score features F logit , visual features F v and local text prompt features Disease category score features F logitThe radiology reports in the auxiliary data set are input into the first retrieval enhancement module for processing, and global reference prompt features obtained by the first retrieval enhancement module finally processing the input radiology reports visual features F v , local text prompt features and global reference prompt features are input into the visual text alignment and fusion module for processing to obtain fused visual text features F c , visual features F v and the radiology reports in the auxiliary data set are also input into the auxiliary multi-label long-tail classification module for processing to obtain prompt features X, and the prompt features X and the fused visual text features F c are input into the output decoding module for processing to obtain the radiology reports, and the obtained radiology reports are taken as the output of the retrieval enhancement-based visual text alignment and fusion model. The visual text alignment and fusion module also processes to obtain the region information of the sentences in the radiology reports corresponding to the medical images and the region information of the disease categories corresponding to the medical images region information of the medical images and the region information of the disease categories corresponding to the medical images for calculating the consistency loss obtained by the visual text alignment and fusion module.

[0036] In specific implementation, the multi-label classification model adopted by the auxiliary multi-label long-tail classification module includes an average pooling layer (AP), a Top K layer and a multi-label classification layer (Multi-label Classfication); the visual features F v are input into the average pooling layer (AP) for processing, the radiology reports in the auxiliary data set are input into the Top K layer for processing, the results of the processing of the average pooling layer (AP) and the Top K layer are subjected to feature concatenation processing, and the result of the feature concatenation processing is input into the multi-label classification layer for processing to obtain the prompt features X, which are taken as the output of the multi-label classification model.

[0037] The input encoding module includes a visual encoder (Visual Encoder) and a first MedKLIP model encoder; the medical images in the model training data set are respectively input into the visual encoder and the first MedKLIP model encoder, the visual encoder processes to obtain visual features F v , and the first MedKLIP model encoder processes to obtain disease category score features F logit and local text prompt features visual features F v are input into the auxiliary multi-label long-tail classification module for processing, and the disease category score features Flogit The input is processed in the first retrieval enhancement module and the fusion module, visual feature F v and local text prompt feature together into the visual text alignment and fusion module for processing.

[0038] The auxiliary multi-label long-tail classification module adopts a multi-label classification model (MLC), the output decoding module adopts a Bert network model, and the visual encoder adopts a Resnet network model.

[0039] The MedKLIP model encoder is a Medical Knowledge-Language-Image Pretraining pre-training model. In this embodiment, the visual encoder specifically adopts a Resnet4 network model.

[0040] The first retrieval enhancement module includes a CheXbert module, a first hyperbolic neural network (HNN), a ranking metric learning module, and a second MedKLIP model encoder; the disease category score feature F logit is input into the first hyperbolic neural network (HNN) for processing to obtain a hyperbolic curve feature, then the Poincare ball model is used to calculate the distance between each adjacent hyperbolic curve feature, thereby obtaining a hyperbolic distance matrix corresponding to all hyperbolic curve features in the batch training; the radiology reports in the auxiliary data set are input into the CheXbert module for processing to obtain a disease classification label vector corresponding to each radiology report, then the Hamming distance between each adjacent two disease classification label vectors is calculated, thereby obtaining a Hamming distance matrix corresponding to all disease classification label vectors in the batch training, the obtained hyperbolic distance matrix and Hamming distance matrix are input into the ranking metric learning module for processing to obtain an index vector, and the radiology reports corresponding to the index vector are screened out from the auxiliary data set according to the index vector; the screened radiology reports are input into the second MedKLIP model encoder for processing to obtain a global reference prompt feature The obtained global reference prompt feature is input into the visual text alignment and fusion module for processing.

[0041] The CheXbert module is a conventional natural language processing module for medical image report understanding in the prior art. The MedKLIP (Medical Knowledge-enhanced Language-Image Pretraining) model encoder is a multi-modal pre-training model in the medical field in the prior art.

[0042] The specific processing process of the ranking metric learning module is: each row in the Hamming distance matrix is sorted in ascending order to obtain an indexed vector after sorting; the hyperbolic distance matrix is taken as the prediction value of the ranking metric learning module, and the indexed vector after sorting is taken as the target value of the ranking metric learning module, the cross entropy loss between the hyperbolic distance matrix and the indexed vector is calculated, and the ranking metric learning module is continuously trained in the training process of the visual text alignment and fusion model based on retrieval enhancement according to the cross entropy loss.

[0043] The hyperbolic distance processing is set according to the following formula:

[0044]

[0045] Wherein, x and y represent two adjacent hyperbolic features respectively; c represents the curvature; represents the distance between the two adjacent hyperbolic features; represents the Mobius addition operation; ||·|| represents the Euclidean norm; x T represents the transpose of x.

[0046] As shown in Figure 3 , the visual text alignment and fusion module is set according to the following formula:

[0047]

[0048] Wherein, F c represents the fused visual text feature, also represents the output of the visual text alignment and fusion module; (,) represents the feature fusion of connecting two features along the feature dimension; f proj1 () represents the first linear layer; f proj () represents the second linear layer; LayerNorm() represents the layer normalization layer; F v represents the visual feature; represents the region information of the sentence in the radiology report corresponding to the medical image; represents the region information of the disease category corresponding to the medical image; MPSA1() and MPSA1() represent the first and second multi-prompt Sinkhorn attention mechanisms respectively; () T represents the transpose; φ1(), φ2(), φ3() and φ4() represent the first, second, third and fourth linear layers respectively; represents the global reference prompt feature; represents the local text prompt feature. Q1, K1, V1, Q2, K2 and V2 all represent intermediate calculation parameters.

[0049] The first and second multi-prompt Sinkhorn attention mechanisms have the same structure, both consisting of a first matrix multiplication layer (MatMul), a scaling layer, an optimal transfer-based Sinkhorn layer (OT-basd Sinkhorn), a second matrix multiplication layer (MatMul), an addition and normalization layer (Add&Norm), and a linear layer, all connected in series. The first matrix multiplication layer (MatMul) receives either feature vectors K1 and Q1 or feature vectors K2 and Q2 as input. The second matrix multiplication layer (MatMul) receives either feature vector V1 or feature vector V2. The optimal transfer-based Sinkhorn layer (OT-basd Sinkhorn) outputs the region information of the medical image corresponding to the sentence in the radiology report. Or the regional information on medical images corresponding to disease categories Linear layer output or

[0050] The loss function of the retrieval-enhanced visual text alignment and fusion model is set according to the following formula:

[0051]

[0052]

[0053] in, This represents the loss function of the retrieval-enhanced visual text alignment and fusion model. This indicates the task loss obtained by the output decoding module; This represents the cross-entropy loss obtained from the ranking metric learning module; The expression represents the consistency loss obtained by the visual text alignment and fusion module; α and β both represent balance coefficients; i represents the index; p(r i |r 1:i-1 ) represents the model in the output radiology report as r1,…,r i-1 Given the given terms, output the i-th term r of the radiology report. i The probability; T represents the total number of terms in the radiology report; Π represents the hyperbolic distance matrix; Π represents the index vector obtained after sorting the Hamming distance matrix; CE() represents the cross-entropy loss; n and m both represent indices; N represents the total number of sentences in the radiology report; M represents the total number of disease categories in the medical images; This represents the global reference cue feature of the nth sentence in a radiology report; This represents the local text cue feature for the m-th disease category in a medical image; <·,·> represents the inner product; ||·||2 represents the L2 norm; IoU() represents the intersection-union ratio; This indicates the region information in the medical image corresponding to the nth sentence in the radiology report; S represents the region information on the medical image corresponding to the m-th disease category; nm and O nm These are all parameters used in the calculation process.

[0054] S3. Construct an inference model based on the trained retrieval-enhanced visual text alignment and fusion model.

[0055] like Figure 2 As shown, the inference model simply replaces the first retrieval enhancement module in the trained retrieval-based visual text alignment and fusion model with a second retrieval module. The second retrieval module includes a newly added third MedKLIP model encoder, a second hyperbolic neural network (HNN), and the trained first hyperbolic neural network (HNN) and second MedKLIP model encoder. The disease category score feature F output by the first MedKLIP model encoder is... logit The data is input into a first hyperbolic neural network (HNN) for processing to obtain hyperbolic features. Then, a Poincaré sphere model is used to calculate the distance between each adjacent hyperbolic feature, thus obtaining the hyperbolic distance matrix corresponding to all hyperbolic features. Radiological reports from the auxiliary dataset are input into a third MedKLIP model encoder to obtain disease category score features from the auxiliary database. Disease category score features of auxiliary databases The input is fed into a second hyperbolic neural network (HNN) for processing of auxiliary hyperbolic features. A Poincaré sphere model is then used to calculate the distance between each adjacent auxiliary hyperbolic feature, resulting in auxiliary hyperbolic distance matrices for all features. The distance between each hyperbolic distance matrix and each auxiliary hyperbolic distance matrix is ​​calculated, and the auxiliary hyperbolic distance matrix with the smallest distance between each hyperbolic distance matrix and its auxiliary counterparts is selected. Based on this selected auxiliary hyperbolic distance matrix, the corresponding radiology report for the medical image in the auxiliary dataset is obtained. The selected radiology report is then input into the second MedKLIP model encoder for processing to obtain global reference cue features. The obtained global reference hint features The input is processed in the visual text alignment and fusion module.

[0056] In practice, when processing in the inference model, the medical image to be tested is input into the visual encoder in the input encoding module for processing.

[0057] S4, collect the to-be-tested medical image, input the to-be-tested medical image and the auxiliary data set into the inference model for processing to obtain a radiology report corresponding to the to-be-tested medical image.

[0058] Embodiment 2

[0059] The same method as in Embodiment 1 was used in this embodiment to perform experiments on the MIMIC-CXR data set, and some SOTA models were compared. The effects of the comparison are as follows:

[0060]

[0061] Note: (1) R2Gen, M2TR, MKSG, M2KT, ME, KiUT, DCL, UAR, HERGen, CVT2Dis, CliBert, RGRG, PromptMRG, and EKAGen are models that have previously achieved SOTA on the MIMIC-CXR data set, and Ours is the method of the present application.

[0062] (2) NLG Metrics is a set of indicators for evaluating the quality of natural language generation output. BLEU-1 measures the degree of coincidence of single words between generated text and reference text; BLEU-2 measures the degree of coincidence of two consecutive words between generated text and reference text; BLEU-3 measures the degree of coincidence of three consecutive words between generated text and reference text; BLEU-4 measures the degree of coincidence of four consecutive words between generated text and reference text; METEOR calculates the alignment between text and reference text to evaluate similarity; ROUGE calculates the n-gram overlap between generated text and reference text to evaluate similarity. The above indicators, the larger the value, the better.

[0063] (3) CE Metrics is commonly used to evaluate the performance of natural language processing models when generating text. Precision, Recall, and F1 score are indicators among them, the larger the value, the better.

[0064] (4) Avg: represents the average value of NLG Metrics and CE Metrics, the larger the value, the better the report generation effect.

[0065] (5) The black and bold values in the table represent the best segmentation performance.

[0066] (6) Compared with the SOTA models R2Gen, M2TR, MKSG, M2KT, ME, KiUT, DCL, UAR, HERGen, CVT2Dis, CliBert, RGRG, PromptMRG, and EKAGen, the average score of the present application on the same data set is improved by 7.4%.

[0067] The method of the present application enhances the retrieval relevance, and solves the difficulty of fine-grained region-sentence alignment caused by weak supervision of image report level in the medical report generation process by using a fine-grained visual-text alignment and fusion method to align and fuse features.

[0068] It should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A learnable method for generating radiological reports based on retrieval-enhanced visual text alignment and fusion, characterized in that, Includes the following steps: S1. Collect several medical images and the corresponding radiology reports for each medical image. Use the medical images as input data and the corresponding radiology reports as labels to construct a model training dataset. Then, summarize all the collected medical images and the corresponding radiology reports for each medical image to obtain an auxiliary dataset. S2. Construct a retrieval-enhanced visual text alignment and fusion model. Input the model training dataset and the auxiliary dataset into the retrieval-enhanced visual text alignment and fusion model for training to obtain the trained retrieval-enhanced visual text alignment and fusion model. S3. Construct an inference model based on the trained retrieval-enhanced visual text alignment and fusion model; S4. Collect the medical images to be tested, input the medical images to be tested and auxiliary datasets together into the inference model for processing, and obtain the radiological report corresponding to the medical images to be tested.

2. The method for generating radiological reports according to claim 1, characterized in that: The retrieval-enhanced visual text alignment and fusion model includes an input encoding module, a first retrieval enhancement module, a visual text alignment and fusion module, an auxiliary multi-label long-tail classification module, and an output decoding module. Medical images from the model training dataset are input into the input encoding module for processing, yielding disease category score features F. logit Visual features F v and local text prompt features Disease category score feature F logit The radiology reports from the auxiliary dataset are input into the first retrieval enhancement module for processing. The global reference hint features are ultimately obtained from the processing by the first retrieval enhancement module. Visual feature F v Local text prompt features and global reference hint features All inputs are processed in the visual text alignment and fusion module to obtain the fused visual text features F. c Visual features F v The radiology reports from the auxiliary dataset are also input into the auxiliary multi-label long-tail classification module for processing to obtain cue features X, cue features X, and fused visual-text features F. c The data are input together into the output decoding module for processing to obtain a radiological report, which is then used as the output of a retrieval-enhanced visual text alignment and fusion model.

3. The method for generating radiological reports according to claim 2, characterized in that: The input encoding module includes a visual encoder and a first MedKLIP model encoder; medical images from the model training dataset are input into the visual encoder and the first MedKLIP model encoder, respectively, and the visual encoder processes them to obtain visual features F. v The first MedKLIP model encoder processes the data to obtain disease category score features F. logit and local text prompt features Visual feature F v The input is processed in the auxiliary multi-label long-tail classification module, and the disease category score feature F is obtained. logit The input is processed in the first retrieval enhancement module and the fusion module, and the visual feature F v and local text prompt features They are then input together into the visual text alignment and fusion module for processing.

4. The method for generating radiological reports according to claim 3, characterized in that: The auxiliary multi-label long-tail classification module adopts a multi-label classification model; the output decoding module adopts a BERT network model; and the visual encoder adopts a ResNet network model.

5. The method for generating radiological reports according to claim 3, characterized in that: The first retrieval enhancement module includes a CheXbert module, a first hyperbolic neural network, a ranking metric learning module, and a second MedKLIP model encoder; the disease category score feature F output by the first MedKLIP model encoder is... logit The data is input into a first hyperbolic neural network to obtain hyperbolic features. A Poincaré sphere model is then used to calculate the distance between each adjacent hyperbolic feature, resulting in a hyperbolic distance matrix for all hyperbolic features. Radiology reports from the auxiliary dataset are input into a CheXbert module to obtain a disease classification label vector for each report. The Hamming distance between each pair of adjacent disease classification label vectors is then calculated, resulting in a Hamming distance matrix for all disease classification label vectors. The hyperbolic distance matrix and the Hamming distance matrix are then input into a ranking metric learning module to obtain index vectors. Radiology reports corresponding to these index vectors are selected from the auxiliary dataset. The selected radiology reports are then input into a second MedKLIP model encoder to obtain global reference cue features. The obtained global reference hint features The input is processed in the visual text alignment and fusion module.

6. The method for generating radiological reports according to claim 5, characterized in that, The specific processing procedure of the ranking metric learning module is as follows: Sort each row of the Hamming distance matrix in ascending order to obtain the sorted index vector. Use the hyperbolic distance matrix as the predicted value of the ranking metric learning module and the sorted index vector as the target value of the ranking metric learning module. Calculate the cross-entropy loss between the hyperbolic distance matrix and the index vector. Based on the cross-entropy loss, continuously train the ranking metric learning module during the training of the retrieval-enhanced visual text alignment and fusion model.

7. The method for generating radiological reports according to claim 3, characterized in that: The visual text alignment and fusion module is set according to the following formula: Q1=φ1(F v ), Q2=φ3(F v ) Among them, F c This represents the fused visual text features, and also the output of the visual text alignment and fusion module; This represents feature fusion where two features are connected along their respective feature dimensions; f proj1 () indicates the first linear layer; f proj2 () indicates the second linear layer; LayerNorm() indicates the layer normalization layer; F v Indicates visual features; This indicates the region information in the medical image corresponding to a sentence in the radiology report; This represents the region information on the medical image corresponding to the disease category; MPSA1() and MPSA1() represent the first multi-cue Sinkhorn attention mechanism and the second multi-cue Sinkhorn attention mechanism, respectively; T φ1(), φ2(), φ3(), and φ4() represent the first, second, third, and fourth linear layers, respectively; Indicates global reference hint features; This indicates a local text prompt feature.

8. The method for generating radiological reports according to claim 2, characterized in that: The loss function of the retrieval-enhanced visual text alignment and fusion model is set according to the following formula: in, This represents the loss function of the retrieval-enhanced visual text alignment and fusion model. This indicates the task loss obtained by the output decoding module; This represents the cross-entropy loss obtained from the ranking metric learning module; The expression represents the consistency loss obtained by the visual text alignment and fusion module; α and β both represent balance coefficients; i represents the index; p(r i |r 1：i-1 ) represents the model's output radiology report as r1, ..., r i-1 Given the given terms, output the i-th term r of the radiology report. i The probability; T represents the total number of terms in the radiology report; Π represents the hyperbolic distance matrix; Π represents the index vector obtained after sorting the Hamming distance matrix; CE() represents the cross-entropy loss; n and m both represent indices; N represents the total number of sentences in the radiology report; M represents the total number of disease categories in the medical images; This represents the global reference cue feature of the nth sentence in a radiology report; represents the local text cue feature for the m-th disease category in a medical image; <·, ·> represent the inner product; ||·||2 represents the L2 norm; IoU() represents the intersection-union ratio; This indicates the region information in the medical image corresponding to the nth sentence in the radiology report; This represents the region information on the medical image corresponding to the m-th disease category.

9. The method for generating radiological reports according to claim 2, characterized in that: The inference model is simply a replacement of the first retrieval enhancement module in the trained retrieval-enhanced visual text alignment and fusion model with a second retrieval module; the second retrieval module includes a newly added third MedKLIP model encoder, a second hyperbolic neural network, and the trained first hyperbolic neural network and second MedKLIP model encoder; the disease category score feature F output by the first MedKLIP model encoder is... logit The data is input into a first hyperbolic neural network for processing to obtain hyperbolic features. Then, a Poincaré sphere model is used to calculate the distance between each adjacent hyperbolic feature, thus obtaining the hyperbolic distance matrix corresponding to all hyperbolic features. Radiological reports from the auxiliary dataset are input into a third MedKLIP model encoder to obtain disease category score features from the auxiliary database. Disease category score features of auxiliary databases The auxiliary hyperbolic features are input into the second hyperbolic neural network for processing. Then, the Poincaré sphere model is used to calculate the distance between each adjacent auxiliary hyperbolic feature, thereby obtaining the auxiliary hyperbolic distance matrix corresponding to all auxiliary hyperbolic features. The distance between each hyperbolic distance matrix and each auxiliary hyperbolic distance matrix is ​​calculated. The auxiliary hyperbolic distance matrix with the smallest distance between each hyperbolic distance matrix and the auxiliary hyperbolic distance matrices is selected. Based on the selected auxiliary hyperbolic distance matrices, the radiology report corresponding to the medical image in the auxiliary dataset is obtained. The selected radiology report is then input into the second MedKLIP model encoder for processing to obtain global reference cue features. The obtained global reference hint features The input is processed in the visual text alignment and fusion module.

Images