Radiology report generation method based on contrast learning and adaptive knowledge integration

Abstract

The present application relates to the technical field of computer vision and image processing, in particular to a radiology report generation method based on contrast learning and adaptive knowledge integration, comprising: obtaining a target medical image and a radiology report text; inputting the target medical image and the radiology report text into a preset information integration model to output a radiology report with unified medical image and knowledge, wherein the information integration model extracts features in the target medical image and the radiology report text, and performs adaptive fusion after aligning the features through contrast learning to output final features. The present application can realize fully automatic radiology report generation with high reliability, significantly reduce the work burden of radiologists, and improve the standardization degree and clinical application value of the report.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, in particular to a radiology report generation method based on contrast learning and adaptive knowledge integration. BACKGROUND

[0002] Radiology report is the core basis for clinical diagnosis, treatment plan formulation and disease tracking, and its accuracy and integrity directly affect the quality of medical decision-making. At present, radiology report mainly relies on radiologists to write manually, which needs to combine the visual features of medical images (such as X-ray, CT, MRI images, etc.) and professional medical knowledge for comprehensive judgment. However, this process has limitations such as low efficiency, subjective differences, and missed diagnosis risks.

[0003] To solve the above problems, radiology report generation (RRG) technology emerges as the times require, and its core is to automatically analyze medical images by computer and generate text reports that meet clinical standards, aiming to reduce the workload of doctors and improve the standardization of reports. The development of existing RRG technology can be divided into the following stages: early template and rule-based method, traditional deep learning-based method, improved method with attention mechanism, multi-modal fusion method based on Transformer, etc.

[0004] (1) Early template and rule method, i.e. report generation method based on pre-defined template and artificial rules, can only generate structured text through "fill-in-the-blank" logic, and cannot cover complex clinical scenarios (such as multiple organ concurrent lesions, atypical symptoms). Its fixed format has poor adaptability to rare diseases and new signs, and relies on manual rule updates, which has insufficient scalability and is difficult to adapt to the dynamic changes of medical practice.

[0005] (2) Traditional deep learning method mainly uses CNN to extract visual features + RNN to generate text method, which is limited by the receptive field range of CNN and is difficult to capture fine-grained lesion features in images (such as small nodules with a diameter of less than 5mm, early tumor edge morphology), resulting in missing key clinical information in the report. At the same time, there is a semantic gap between visual features and medical terminology, and the generated text may be grammatically correct but clinically incorrect (such as misjudging "benign calcification" as "malignant tumor").

[0006] (3) The attention mechanism is not well adapted to the Transformer method. Although the model with the attention mechanism can focus on the local area, it is limited by the network structure of CNN and cannot effectively model the correlation between the areas (such as the distribution rule of multiple metastases). Although the method based on ViT improves the local feature extraction capability, it does not solve the problem of visual-text semantic alignment, and the ability to distinguish normal structure and "abnormal lesions" is weak, which easily leads to redundant information (such as repeated description of normal anatomical structure) in the report.

[0007] (4) The existing technology combined with a medical knowledge graph (such as Kiut, HKRG, etc.) has the following limitations due to the dependence on a static knowledge graph: knowledge lag: unable to integrate emerging clinical insights (such as the imaging features of new diseases, the latest diagnosis and treatment guidelines); granularity misalignment: macroscopic concepts (such as "pneumonia") in the knowledge graph are difficult to accurately match with fine-grained features (such as "ground glass shadow range") in the image; rigid fusion: the fusion of knowledge and multi-modal features is mostly simple splicing, without considering the dynamic differences in clinical scenarios (such as different imaging description specifications for adults and children), resulting in a lack of scenario adaptability in the report.

[0008] The above defects collectively result in the inability of the existing technology to simultaneously ensure the semantic coherence (smooth language) and clinical accuracy (correct medical terminology, rigorous logic) of the report; the lack of ability to capture fine-grained lesion features in the image and map medical semantics; the lack of flexibility in the integration of static medical knowledge, which cannot adapt to medical progress and diverse clinical scenarios; the poor explainability of the generated results, making it difficult to trace the association between the conclusions in the report and the image features and medical knowledge, resulting in low clinical credibility. It is impossible to achieve truly automatic and highly reliable radiology report generation, and doctors still need to perform a large amount of manual review and modification, which does not fundamentally solve the problems of efficiency and accuracy. There is an urgent need for a RRG technology that can dynamically integrate multi-modal data and medical knowledge and accurately capture clinical key information to meet actual clinical needs. SUMMARY

[0009] The purpose of the present application is to provide a radiology report generation method based on contrast learning and adaptive knowledge integration, which breaks the rigid constraints of static knowledge graphs, dynamically integrates medical knowledge and emerging clinical insights, and improves the adaptability of the report to new scenarios and new signs; solves the semantic gap problem of vision-text-knowledge, ensures the unity of the generated report in language coherence and clinical accuracy through fine-grained feature alignment; enhances the model's ability to capture key lesion features (such as subtle abnormalities and cross-area correlations) in the image, reduces missed detection and misjudgment, and achieves highly reliable automatic radiology report generation.

[0010] To achieve the above purpose, the present application provides the following scheme:

[0011] A radiology report generation method based on contrast learning and adaptive knowledge integration, comprising:

[0012] Obtaining a target medical image and a radiology report text;

[0013] Inputting the target medical image and the radiology report text into a preset information integration model, and outputting a radiology report with unified medical image and knowledge, wherein the information integration model extracts features from the target medical image and the radiology report text, aligns the features through contrast learning, and outputs final features through adaptive fusion.

[0014] Optionally, the information integration model comprises:

[0015] A feature extraction module for extracting visual features, text features, and knowledge graph features based on the target medical image and the radiology report text;

[0016] A contrast learning module for performing cross-modal semantic alignment on the visual features, text features, and knowledge graph features using hierarchical contrast learning;

[0017] A feature fusion module for fusing the aligned visual features and knowledge graph features into a unified representation using a bidirectional cross-attention and an adaptive weighting strategy.

[0018] Optionally, the feature extraction module extracts visual features, text features, and knowledge graph features based on the target medical image and the radiology report text, comprising:

[0019] Encoding the target medical image using a pre-trained ViT model to obtain the visual features;

[0020] After performing word segmentation and tokenization on the radiology report text, encoding it using a pre-trained KEBERT model to obtain the text features;

[0021] Starting from a basic medical knowledge graph, extracting triples from the radiology report text using a pre-trained RadGraph model and a KEBERT model to obtain the knowledge graph features.

[0022] Optionally, the contrast learning module comprises:

[0023] An image-text contrast learning unit for calculating the similarity of visual features and text features, and optimizing the features by calculating the similarity loss to obtain aligned visual features and text features;

[0024] A text-knowledge graph contrast learning unit for obtaining aligned text features and knowledge graph features based on text features and knowledge graph features by dynamically updating the knowledge graph and combining supervised contrast learning.

[0025] Optionally, the similarity between the visual feature and the text feature comprises:

[0026] The momentum encoder and the feature queue mechanism are introduced, and after calculating the similarity between the features in the queue, the normalized image-text similarity and the text-image similarity are obtained through the softmax activation function.

[0027] Optionally, the similarity loss comprises:

[0028] ;

[0029] wherein, is the similarity loss between the visual feature and the text feature, L CE is a cross-entropy loss function, is the text feature, is the visual feature, and are the true similarity labels of the text feature and the visual feature, respectively.

[0030] Optionally, the dynamic updating of the knowledge graph comprises:

[0031] The knowledge graph features are matched and added to the basic medical knowledge graph to obtain an updated knowledge graph;

[0032] The updated knowledge graph is encoded by a graph Transformer, structured knowledge features are extracted, the association between nodes is captured by a multi-head self-attention mechanism, the node embedding is optimized by a feedforward neural network, and the updated knowledge graph features are obtained.

[0033] Optionally, the supervised contrastive learning comprises:

[0034] The updated knowledge graph is split into a discrete triple set, the triple mentioned in the text feature is set as a positive sample, and the triple not mentioned is set as a negative sample; and taking the text feature as an anchor, the feature distance between the text feature and the positive sample is narrowed, and the distance from the text feature to the negative sample is pushed away, and the calculation method is:

[0035] ;

[0036] wherein, is the similarity loss between the text feature and the knowledge graph feature, is the text feature of the i-th report, is the k-th positive sample triple of the i-th report, is the number of positive sample triples of the i-th report; N is the total number of triplets, sim is the cosine similarity, exp is the natural exponential function, for the jth total triplet of the ith report, N is the total number of triplets; sim is the cosine similarity, exp is the natural exponential function, is the temperature parameter, and B is the number of reports.

[0037] Optionally, the feature fusion module adopts a bidirectional cross-attention and adaptive weighting strategy to fuse the aligned visual features and knowledge graph features into a unified representation, which includes:

[0038] The visual features and knowledge graph features output by the contrast learning module are linearly projected to generate query vectors and value vectors;

[0039] Based on the query vector and the value vector, a scaled dot product attention is used to calculate a bidirectional similarity matrix, and based on the similarity matrix, an attention weight is generated, and through weight normalization and weighted fusion, cross-modal information aggregation is performed, and a residual connection is used to combine layer normalization optimization representation to obtain enhanced visual features and knowledge graph features;

[0040] The enhanced visual features and knowledge graph features are spliced and input into a multi-layer perceptron to learn adaptive weights, and then a unified feature representation is generated through weighted summation;

[0041] The unified feature is input to combine token embedding and learnable position encoding to generate a report word by word at each time step.

[0042] The beneficial effects of the present application are:

[0043] The present application can break through the rigid constraints of static knowledge graph, realize the dynamic fusion of medical knowledge and emerging clinical insights, and improve the adaptability of reports to new scenes and new signs; solve the semantic gap problem of vision-text-knowledge, ensure the unity of the generated report in language coherence and clinical accuracy through fine-grained feature alignment; enhance the model's ability to capture key lesion features in images (such as subtle abnormalities and cross-region associations), reduce missed detection and misjudgment; realize high-reliability fully automatic radiology report generation, significantly reduce the workload of radiologists, and improve the standardization degree and clinical application value of the report.

[0044] In practical application in the field of automatic generation of radiology reports, it has significant clinical value and technical advantages, effectively solves the cross-modal semantic misplacement problem between medical images and report texts through hierarchical contrast learning, and breaks through the rigid constraints of static knowledge through adaptive knowledge graph integration, so that the generated report can accurately associate anatomical structures and disease logic. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and other drawings can be obtained by those of ordinary skill in the art without any creative effort based on these drawings.

[0046] Figure 1 A hierarchical contrastive learning flowchart of the embodiment of the present application;

[0047] Figure 2 An adaptive feature fusion flowchart of the embodiment of the present application;

[0048] Figure 3 A radiology report generation method flowchart based on contrastive learning and adaptive knowledge integration of the embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort belong to the scope of protection of the present application.

[0050] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] The embodiment provides a radiology report generation method based on contrastive learning and adaptive knowledge integration, as shown in Figure 3 , which comprises the following steps.

[0052] Obtaining a target medical image and a radiology report text;

[0053] Inputting the target medical image and the radiology report text into a preset information integration model to output a radiology report with unified medical image and knowledge, wherein the information integration model extracts features in the target medical image and the radiology report text, and performs adaptive fusion after aligning the features through contrastive learning to output final features.

[0054] Further, the information integration model comprises:

[0055] A feature extraction module for extracting visual features, text features and knowledge graph features based on the target medical image and the radiology report text;

[0056] a contrast learning module configured to perform cross-modal semantic alignment on the visual features, the text features, and the knowledge graph features by using hierarchical contrast learning;

[0057] a feature fusion module configured to fuse the aligned visual features and the knowledge graph features into a unified representation by using a bidirectional cross-attention and an adaptive weighting strategy.

[0058] Further, the feature extraction module is configured to extract the visual features, the text features, and the knowledge graph features based on the target medical image and the radiology report text, including:

[0059] encoding the target medical image by using a pre-trained ViT model to obtain the visual features;

[0060] after performing word segmentation and tokenization on the radiology report text, encoding the radiology report text by using a pre-trained KEBERT model to obtain the text features;

[0061] starting from a basic medical knowledge graph, extracting triples from the radiology report text by using a pre-trained RadGraph model and a KEBERT model to obtain the knowledge graph features.

[0062] Further, the contrast learning module includes:

[0063] an image-text contrast learning unit configured to obtain the aligned visual features and the text features by calculating the similarity of the visual features and the text features and performing feature optimization by calculating a similarity loss;

[0064] a text-knowledge graph contrast learning unit configured to obtain the aligned text features and the knowledge graph features by dynamically updating the knowledge graph and combining supervised contrast learning based on the text features and the knowledge graph features.

[0065] Further, the feature fusion module is configured to fuse the aligned visual features and the knowledge graph features into a unified representation by using a bidirectional cross-attention and an adaptive weighting strategy, including:

[0066] performing linear projection on the visual features and the knowledge graph features output by the contrast learning module to generate a query vector and a value vector;

[0067] based on the query vector and the value vector, calculating a bidirectional similarity matrix by using scaled dot-product attention, generating attention weights based on the similarity matrix, performing cross-modal information aggregation by weight normalization and weighted fusion, and obtaining enhanced visual features and knowledge graph features by using residual connection to combine layer normalization and optimization representation;

[0068] The enhanced visual features and knowledge graph features are spliced and input into a multi-layer perceptron to learn adaptive weights, and a unified feature representation is generated by weighted summation.

[0069] The unified features are input to combine token embedding and learnable position encoding, and a report is generated word by word at each time step.

[0070] Specifically, the embodiment can break through the rigid constraints of static knowledge graph, realize dynamic fusion of medical knowledge and emerging clinical insights, and improve the adaptability of the report to new scenarios and new signs; solve the semantic gap problem of vision-text-knowledge, ensure the unity of the generated report in language coherence and clinical accuracy through fine-grained feature alignment; enhance the model's ability to capture key lesion features in images (such as subtle abnormalities and cross-region associations), reduce missed detection and misjudgment; realize high-reliability automatic radiology report generation, significantly reduce the workload of radiologists, and improve the standardization and clinical application value of the report.

[0071] In practical application to the radiology report automatic generation scene, it has significant clinical value and technical advantages. Through hierarchical contrastive learning, the cross-modal semantic misplacement problem between medical images and report texts is effectively solved, and the adaptive knowledge graph integration breaks through the rigid constraints of static knowledge, enabling the generated report to accurately associate anatomical structures and disease logic.

[0072] The information integration model structure and workflow proposed in the embodiment will be described in detail below. Figure 1 , Figure 2 The information integration model structure and workflow proposed in the embodiment will be described in detail below.

[0073] 1. Hierarchical contrastive learning part:

[0074] The hierarchical contrastive learning process is shown in Figure 1 . First, align the image and report features through vision-text contrastive learning: input the medical image and its corresponding radiology report text, and use the pre-trained ViT model to encode the image and the pre-trained KEBERT model to encode the token sequence after tokenization to generate visual feature vectors and text feature vectors.

[0075] Then, construct positive and negative sample pairs, calculate the vision-text contrastive loss, and update the parameters of the image encoder (ViT) and the text encoder (KEBERT) based on the loss function. The matching vision-text features are aggregated in space, and the non-matching features are separated, achieving cross-modal alignment. The output of this process is the aligned visual features and text features.

[0076] Then, the alignment between the text and the knowledge graph modal is realized through supervised contrast learning between the text features and the triples in the knowledge graph: taking the text features generated in the last stage and the triples in the knowledge graph as inputs, first updating the basic graph based on the triples extracted in the report, then encoding the updated graph through the graph Transformer to generate the knowledge graph feature vector; and further constructing positive and negative sample pairs. The similarity score of the text features and the knowledge graph features is calculated as a loss function, and by minimizing the loss function, the representations of the report and the positive sample triples are pulled closer, and the representations of the negative samples are pushed away, realizing the fine-grained alignment between the radiology text and the structured knowledge.

[0077] Specifically includes:

[0078] (1) Data preprocessing and initial feature extraction:

[0079] First, the medical images are preprocessed by resampling the images to 224*244 and normalizing them; secondly, the uppercase letters in the radiology report text are changed to lowercase and the punctuation marks are removed. Ensure that the training samples are free of noise interference. Then the initial features are extracted, the specific process is as follows:

[0080] Image feature extraction: encode the input medical images using a pre-trained ViT model to generate initial visual features .

[0081] Text feature extraction: perform word segmentation and tokenization on the radiology report text, and use a pre-trained KEBERT model to encode to generate initial text features .

[0082] Knowledge graph feature initialization: take the basic medical knowledge graph as the starting point, wherein the vertex set V contains a number of medical entities, and the edge set E is used to describe the relationship between the medical entities. The triples are extracted from the report by the pre-trained RadGraph model and KEBERT model, which are prepared for graph updating, wherein h represents the head entity, i.e. the subject of the triple, which corresponds to a medical entity (from the vertex set V) in the basic medical knowledge graph; r represents the relationship, which describes the association between the head entity and the tail entity, which corresponds to the relationship between the medical entities described in the edge set E; t represents the tail entity, which is the object of the triple, which is also a medical entity (from the vertex set V) in the basic medical knowledge graph.

[0083] (2) Hierarchical contrast learning:

[0084] Through two-level contrast learning, the semantic bias between the modalities is eliminated, and a unified semantic space is established:

[0085] The first level is image-text contrastive learning. First, projection and scaling are performed, and a learnable matrix (vision feature projection matrix) and (text feature projection matrix) are introduced to map and to the same dimensional space; the similarity is scaled by a learnable temperature parameter (initial value is 0.07, dynamically adjusted) to avoid numerical saturation. The similarity between the u-th vision feature and the v-th text feature is calculated as follows:

[0086] (1).

[0087] where denotes the transpose of the matrix, is the u-th vision feature, is the v-th text feature, and the larger the value of the similarity indicates the closer the semantics of the two.

[0088] To enhance the stability of contrastive learning, a momentum encoder and a feature queue mechanism are introduced. After calculating the similarity of the features in the queue, the normalized image-text similarity and the text-image similarity are obtained by the softmax activation function:

[0089] (2).

[0090] where M represents the queue length stored in the momentum encoder, s represents the similarity function, I represents the current input vision feature, I m represents the m-th vision feature in the queue, T represents the current input text feature, and T m represents the m-th text feature in the queue.

[0091] The feature alignment is optimized by the cross-entropy loss:

[0092] (3).

[0093] where is the similarity loss of the vision feature and the text feature, L CE is the cross-entropy loss function, is the text feature, is the vision feature, and are the true similarity labels (the matching sample label is 1, and the non-matching sample is 0). By minimizing ​to "gather" (maximize similarity) matching image-text features and "separate" (minimize similarity) non-matching features.

[0094] In addition, to further strengthen the association between visual features and text key information, a cross-attention mechanism is introduced. In the features output by the text encoder, the key tokens related to the image content are dynamically focused to generate attention weights. The visual features are updated by weighting with the weights, enhancing the representation of the visual region corresponding to the key semantic of the text.

[0095] The second level is text-knowledge graph contrastive learning. Since there is a semantic gap between the natural language description of the radiology report and the structured medical knowledge graph, direct fusion may lead to medical logic errors in report generation. This level realizes the precise alignment of text and structured knowledge by dynamically updating the knowledge graph and combining supervised contrastive learning, ensuring that the report generation conforms to the rules of the medical field. First, the entity disambiguation and synonym mapping of the knowledge graph features are performed. The aligned entities are matched to the existing nodes in V, and the unmatched entities are added as new nodes to V. Then check if the relationship r exists in E; if not, insert a new directed edge; if it exists, weight the new evidence and the existing edge by the following formula:

[0096] (4) ;

[0097] where λ is a control weight ratio hyperparameter, is the fused edge, is the existing edge, is the new directed edge. In this way, general medical knowledge and report-specific information are integrated together.

[0098] Then perform graph encoding and embedding generation. The updated graph is encoded by the graph Transformer to extract structured knowledge features. The multi-head self-attention mechanism (MHA) is used to capture the association between nodes, and the feedforward neural network (FFN) is used to optimize the node embedding, with the formula:

[0099] (5) ;

[0100] where is the node a embedding of the l+1 layer, is the node a embedding of the l layer, is the node b embedding of the l layer, is the edge feature between nodes a and b.

[0101] The global pooling layer pool is performed on all node embeddings to obtain the feature vector :

[0102] (6).

[0103] To avoid the loss of details caused by different reports associated with the same graph, the fine-grained alignment is strengthened by triple-level contrastive learning, and the updated graph is divided into a discrete triple set K = {k1, k2,..., k N} The triple mentioned in the report is the positive sample, and the unmentioned is the negative sample. With the report as the anchor point, the feature distance between the report and the positive triple is narrowed, and the distance with the negative sample is pushed away, and the formula is:

[0104] (7);

[0105] wherein, is the similarity loss of text features and knowledge graph features, is the text feature of the i-th report, is the k-th positive triple of the i-th report, is the number of positive triples of the i-th report; is the j-th total triple (including positive and negative samples) of the i-th report, N is the total triple number; sim is the cosine similarity, exp is the natural exponential function, is the temperature parameter (adjusts the steepness of the similarity distribution), B is the number of reports. Through the above process, the knowledge graph can dynamically adapt to the report-specific information, and the alignment accuracy of text and knowledge is significantly improved. The graph features output in this stage provide structured knowledge support for subsequent multi-modal fusion, significantly reducing the medical logic error rate of report generation.

[0106] 2. Adaptive feature fusion part:

[0107] The adaptive feature fusion process is shown in Figure 2 . By using bidirectional cross-attention and adaptive weighting strategy, the visual features and knowledge graph features are fused into a unified representation, as follows:

[0108] After hierarchical contrastive learning, the visual features and knowledge graph features have achieved preliminary semantic alignment, but they are still different modal feature representations. Direct splicing or weighted fusion may easily dilute key information. Therefore, an adaptive feature fusion (AFF) module is designed to capture fine-grained correlations between modalities through bidirectional cross-attention and dynamically allocate feature weights to generate a unified representation that takes into account image details and medical knowledge, providing core feature support for accurate report generation. The specific implementation steps are as follows:

[0109] First, the visual features and knowledge graph features output by hierarchical contrastive learning are linearly projected to generate query vectors​ and value vectors . wherein, and come from visual features, and come from knowledge graph features.

[0110] The scaled dot-product attention is used to calculate the bidirectional similarity matrix, and the attention weight is generated based on the similarity matrix. The precise aggregation of cross-modal information is realized through weight normalization and weighted fusion, and the enhanced feature and is obtained. The residual connection combined with layer normalization is used to optimize the feature representation to retain the original feature information and stabilize the training. The enhanced visual feature and knowledge graph feature are obtained, and then they are spliced to input the multi-layer perception (MLP) to learn adaptive weights:

[0111] (8);

[0112] wherein, is the weight of the visual feature, is the weight of the knowledge graph feature, and the sum of the two is 1. For example, when the image feature is clear, the weight is higher; when the image is blurred but the knowledge association is clear, the weight is higher. Then the final unified feature representation z is generated by weighted summation:

[0113] (9);

[0114] Further, in the report generation stage, the unified feature z is input to combine the token embedding (convert text symbols to vectors) and the learnable position encoding (capture sentence order) to generate the report word by word according to the time step c:

[0115] (10);

[0116] And the cross-entropy loss is used to minimize the prediction error to ensure the semantic coherence and accuracy of the generated text:

[0117] (11);

[0118] wherein, P is the conditional probability distribution, C is the total time, is the true label of the cth time step, is the true label of the (c-1)th time step, n represents the length of the sequence, represents the prediction result at the cth time step, represents the prediction result at the (c-1)th time step.

[0119] Validation:

[0120] On the IU X-Ray dataset, the proposed method improves the BLEU-1-BLEU-4 indicators of the SOTA model DWL by 1%-8%, among which the BLEU-4 is improved from 0.191 to 0.208; the METEOR indicator is improved from 0.211 to 0.244, and the ROUGE-L is improved from 0.409 to 0.413, all of which are the current optimal. On the CE (Clinical Effect) indicator for measuring the clinical value of the report, the performance of the proposed method on the MIMIC-CXR dataset is far superior to existing methods, with a precision of 0.518 and a recall of 0.468, which is more in line with the actual diagnosis requirements.

[0121] The above-described embodiments are only descriptions of the preferred modes of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A radiology report generation method based on contrastive learning and adaptive knowledge integration, characterized in that, The method comprises the following steps: obtaining a target medical image and a radiology report text; inputting the target medical image and the radiology report text into a preset information integration model to output a radiology report with unified medical image and knowledge, wherein the information integration model extracts features from the target medical image and the radiology report text, aligns the features through contrast learning, and outputs final features after adaptive fusion, and the information integration model comprises: a feature extraction module for extracting visual features, text features and knowledge graph features based on the target medical image and the radiology report text; a contrast learning module for performing cross-modal semantic alignment on the visual features, text features and knowledge graph features through hierarchical contrast learning, wherein the contrast learning module comprises: an image-text contrast learning unit for calculating the similarity of visual features and text features, and optimizing the features through similarity loss to obtain aligned visual features and text features; a text-knowledge graph contrast learning unit for obtaining aligned text features and knowledge graph features based on text features and knowledge graph features by dynamically updating the knowledge graph and combining supervised contrast learning; a feature fusion module for fusing the aligned visual features and knowledge graph features into a unified representation using a bidirectional cross-attention and adaptive weighting strategy, wherein the feature fusion module comprises: linearly projecting the visual features and knowledge graph features output by the contrast learning module to generate query vectors and value vectors; based on the query vectors and value vectors, calculating a bidirectional similarity matrix using scaled dot-product attention, generating attention weights based on the similarity matrix, aggregating cross-modal information through weight normalization and weighted fusion, and optimizing the representation using residual connection and layer normalization to obtain enhanced visual features and knowledge graph features; concatenating the enhanced visual features and knowledge graph features and inputting them into a multi-layer perceptron to learn adaptive weights, and then generating a unified feature representation through weighted summation; using the unified feature as input, combining token embedding and learnable position encoding, and generating a report word by word at each time step.

2. The radiology report generation method based on contrastive learning and adaptive knowledge integration according to claim 1, characterized in that, The feature extraction module extracts visual features, text features and knowledge graph features based on the target medical image and the radiology report text, which comprises: using a pre-trained ViT model to encode the target medical image to obtain the visual features; after tokenizing the radiology report text, using a pre-trained KEBERT model to encode the text features; starting from a basic medical knowledge graph, extracting triples from the radiology report text through a pre-trained RadGraph model and a KEBERT model to obtain the knowledge graph features.

3. The radiology report generation method based on contrastive learning and adaptive knowledge integration according to claim 1, characterized in that, The similarity of the visual features and the text features is calculated by: introducing a momentum encoder and a feature queue mechanism, calculating the similarity of the features in the queue, and then obtaining normalized image-text similarity and text-image similarity through a softmax activation function.

4. The radiology report generation method based on contrastive learning and adaptive knowledge integration according to claim 1, characterized in that, The calculating the similarity loss comprises: ； wherein, Lvis is a similarity loss for visual features and text features, CE Lce is a cross-entropy loss function, vis is a visual feature, text is a text feature, and yvis and ytext are true similarity labels for the text feature and the visual feature, respectively.

5. The radiology report generation method based on contrastive learning and adaptive knowledge integration according to claim 1, characterized in that, The dynamic updating knowledge graph comprises: The knowledge graph features are matched and added to the basic medical knowledge graph, and an updated knowledge graph is obtained; The updated knowledge graph is encoded by a graph Transformer, structured knowledge features are extracted, a multi-head self-attention mechanism is used to capture the association between nodes, and a feedforward neural network is used to optimize node embedding, so as to obtain updated knowledge graph features.

6. The method of claim 1, wherein, The supervised contrast learning comprises: The updated knowledge graph is split into a discrete triple set, the triple mentioned in the text features is set as a positive sample, and the triple not mentioned is set as a negative sample; and the text features are used as an anchor point to shorten the feature distance between the text features and the positive sample and to lengthen the distance from the negative sample, and the calculation method is: ； wherein, is the similarity loss of text features and knowledge graph features, is the text feature of the i-th report, is the k-th positive sample triple of the i-th report, is the number of positive sample triples of the i-th report; is the j-th total triple of the i-th report, N is the total number of triples; sim is the cosine similarity, exp is the natural exponential function, is the temperature parameter, B is the number of reports.

Images