Text prompt learning method and device for enhancing X-ray image diagnosis
By extracting and injecting information from radiology reports, fine-grained prompt embeddings are generated, which solves the data and labeling limitations of medical VLP models for X-ray image diagnosis, and achieves significant improvement and good in benchmark tests such as ChestX-ray14 and CheXpert Interpretability.
Patent Information
- Application Number
- CN202410984961.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
AI Technical Summary
The existing medical VLP models are limited by the scarcity and high cost labeling of medical imaging data in chest X-ray imaging diagnosis, and fail to effectively utilize the information in unpaired radiological reports, ignoring the semantic content and reports in the images. Fine-grained relationships between concepts.
The naming entity recognition method is used to extract concepts, locations and positives in radiological reports, form triple data, and inject external medical knowledge, generate report-level and concept-level prompt embeddings, and learn the similarity of reports and concepts through text encoders. , construct inference model for X-ray image diagnosis.
This method reduces dependence on image data and labeling, extracts rich information directly from radiological reports, enhances the diagnostic ability of medical VLP models, significantly outperforms advanced methods, and has good interpretable performance.
Smart Images

Figure CN118919013A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of X-ray image diagnosis, visual-language model, prompt learning and multimodal technology, and more specifically, to a text prompt learning method and device for enhancing X-ray image diagnosis. Background Art
[0002] Chest X-ray (CXR) is a commonly used medical imaging examination used to evaluate the structure and function of chest organs such as the lungs, heart, and thorax. Deep learning models have shown great potential in chest X-ray (CXR) image analysis, such as disease classification. However, the training of these models relies on a large number of annotated images, which limits their application in practical settings. Pre-trained language models, such as BERT, provide a new perspective for medical vision and language pre-training. Recently, large-scale vision-language pre-training (VLP) models such as CLIP and BLIP, which use natural paired image-text contrastive learning, have achieved remarkable progress in vision tasks. However, it remains challenging to translate basic vision-language pre-training models into practical clinical applications.
[0003] First, the limited medical imaging data poses a challenge to language models in understanding free-form reports; unlike natural images that can be annotated relatively easily, the annotation of medical images needs to be verified by experienced domain experts (such as clinicians), which leads to high barriers and high costs, resulting in a scarcity of annotated medical images and a large number of unpaired radiology reports that are not fully utilized. Second, although MedCLIP learns from these unpaired texts, the involvement of images is still necessary. Most current medical VLP models only pair CXR images directly with original radiology reports, ignoring the modeling of fine-grained relationships between semantic content in images and concepts in reports. Although efforts have been made to denoise radiology reports and integrate multi-source medical knowledge into pre-training, overcoming the limitations of coarse-grained knowledge and reliance on image data remains difficult to achieve. Third, interpretability is crucial, and linking diagnostic results with visual evidence is essential to help radiologists understand the system and build trust between patients and technology. To this end, it is indeed necessary to develop a text prompt learning method and device for enhancing X-ray image diagnosis. Summary of the invention
[0004] The purpose of the present invention is to provide a text prompt learning method and device for enhancing X-ray image diagnosis, so as to overcome the defects of the prior art.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0006] A text prompt learning method for enhancing X-ray image diagnosis comprises the following steps:
[0007] S1. Input patient chest X-ray image dataset, which contains radiology reports and corresponding images and human annotated labels;
[0008] S2, using a named entity recognition method to extract concepts, locations, and positivity in the radiology report to form triple data;
[0009] S3, injecting external medical knowledge into the triple data to form new fine-grained data;
[0010] S4, concatenating the learnable word embedding with the word embedding of the i-th concept to obtain the report-level prompt, and inputting the report-level prompt into a text encoder to generate a report-level prompt embedding for each concept;
[0011] S5, input the concept-level prompts into a text encoder to generate concept-level prompt embeddings for each concept;
[0012] S6. Use a text encoder to encode the radiology report to obtain global text features and concept description features, learn the prompt by maximizing the probability of classifying each concept description into its true category, and calculate the report-level similarity p and concept-level similarity p';
[0013] S7. Set the training rounds, training batch size, learning rate and optimizer, and use the loss function Train to obtain the inference model;
[0014]
[0015] S8. Input the patient's chest X-ray image into the reasoning model, infer the existence of certain concepts and / or diseases, and determine the visual evidence of the concepts and / or diseases.
[0016] Furthermore, the expression of the triple data in step S2 is:
[0017]
[0018] In the formula, represents the total number of concepts contained in a radiology report, c i is a specific medical concept name (such as pulmonary nodule), l i is the anatomical position of the medical concept (such as left lower lobe, right upper lobe, etc.), pos i A positive result for a medical concept (True, Fasle, or Uncertain).
[0019] Furthermore, the report level prompt in step S4 is embedded in E R The formula is:
[0020]
[0021] In the formula, For report-level prompts, Enc T is the text encoder, i is the i-th medical concept, and L is the total number of concepts.
[0022] Furthermore, the concept-level hint embedding in step S5 The formula is:
[0023]
[0024] In the formula, is a concept-level prompt, i is the i-th medical concept, and L is the total number of concepts.
[0025] Furthermore, in step S6, the report-level similarity p and the concept-level similarity p' are calculated using the following formula:
[0026]
[0027]
[0028] In the formula, u represents the global text feature in training or the visual feature in testing, U represents the concept description feature or the flattened image feature, the information in the local branch P is aggregated in a spatially weighted manner, and τ c Adjust the focus on a specific location.
[0029] The present invention also provides a device for implementing the text prompt learning method for enhancing X-ray image diagnosis, comprising:
[0030] An input module is used to input a patient chest X-ray image dataset, wherein the X-ray image dataset includes a radiology report and corresponding images and human annotated labels;
[0031] An extraction module, for extracting concepts, locations, and positivity from the radiology report using a named entity recognition method to form triple data;
[0032] An injection module, used for injecting external medical knowledge into the triple data to form new fine-grained data;
[0033] a first encoding module, configured to concatenate the learnable word embedding with the word embedding of the i-th concept to obtain a report-level prompt, and input the report-level prompt into a text encoder to generate a report-level prompt embedding for each concept;
[0034] A second encoding module, for inputting concept-level hints into a text encoder to generate concept-level hint embeddings for each concept;
[0035] A computational module, which is used to encode radiology reports using a text encoder to obtain global text features and concept description features, learn hints by maximizing the probability of classifying each concept description into its true category, and calculate report-level similarity p and concept-level similarity p';
[0036] The training module is used to set the training epochs, training batch size, learning rate, and optimizer, and use the loss function Train to obtain the inference model;
[0037] The reasoning module is used to input the patient's chest X-ray image into the reasoning model, infer the existence of certain concepts and / or diseases, and determine the visual evidence of the concepts and / or diseases.
[0038] Compared with the prior art, the advantages of the present invention are: the present invention uses easily accessible radiology reports as medical images in prompt learning to adapt the medical VLP model to the diagnosis of X-ray images, and its classification labels can be directly exported, so the method is less affected by image data limitations and annotation limitations; the present invention directly extracts rich pathological and anatomical information from radiology reports, and injects medical knowledge from knowledge bases or LLMs to enhance the medical VLP model. At the same time, it can be flexibly applied to medical classification tasks with or without image annotations. The report-concept text prompt learning (RC-TPL) proposed by the present invention is used to extract coarse-grained and fine-grained prompt embeddings to enhance X-ray image diagnosis. The present invention significantly outperforms the most advanced methods in several benchmark tests (such as ChestX-ray14, CheXpert) without using any labeled images. At the same time, it has good interpretability performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 It is a text prompt learning framework used by the present invention to enhance X-ray image diagnosis.
[0041] Figure 2 Fine classification performance of different methods on ChestX-ray 14 (left) and CheXpert (right).
[0042] Figure 3 The selected disease categories and the learned concept cues are visualized in the same image for the present invention.
[0043] Figure 4 The same disease categories and learned concept cues are displayed on different images for the present invention. DETAILED DESCRIPTION
[0044] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0045] See also Figure 1-Figure 4 As shown, this embodiment discloses a text prompt learning method for enhancing X-ray image diagnosis, comprising the following steps:
[0046] Step S1: Input a patient's chest X-ray (CXR) image dataset, which contains a radiology report. and the corresponding images and human annotation labels. In training, only radiology reports were used. Instead of using images with human annotated labels, human annotated image labels are required in the evaluation stage of model performance.
[0047] Step S2, concept extraction: using the named entity recognition method Rad-Graph to extract concepts, locations and positivity from the radiology report to form triple data.
[0048]
[0049] In the formula, represents the total number of concepts contained in a radiology report, c i is a specific medical concept name (such as pulmonary nodule), l i is the anatomical position of the medical concept (such as left lower lobe, right upper lobe, etc.), pos i A positive result for a medical concept (True, Fasle, or Uncertain).
[0050] Step S3, knowledge injection: injecting external medical knowledge into the triple data to form new fine-grained data, which can enhance the comprehensibility of the triple data and achieve fine-grained semantic alignment for guiding the learning of concept-level prompts.
[0051] Among them, concept-level prompts mainly learn to align medical concept entities such as lung nodules and pneumonia mentioned in the report content, focusing on more fine-grained information.
[0052] Step S4, report-level prompt: concatenate the learnable word embedding with the word embedding of the i-th concept to obtain the report-level prompt Report-level prompts Input to a text encoder Enc T Generate report-level hint embeddings for each concept The specific formula is as follows.
[0053]
[0054] In the formula, For report-level prompts, Enc T is the text encoder, i is the i-th medical concept, and L is the total number of concepts.
[0055] Among them, report-level prompts are mainly used to learn to align the content of an entire report and focus on global information.
[0056] Specifically, concept-level prompts and report-level prompts are essentially vectors, which are learnable parameters. Generally, a vector of a fixed dimension size (128 dimensions in this embodiment) is directly randomly initialized and then learned during training.
[0057] Step S5: Concept level prompt Input to a text encoder Enc T In the above example, concept-level cue embeddings are generated for each concept. The details are as follows:
[0058]
[0059] In the formula, is a concept-level prompt, i is the i-th medical concept, and L is the total number of concepts.
[0060] Step S6, report and concept description embedding: Use a text encoder to encode the radiology report to obtain global text features and concept description features, learn the prompt by maximizing the probability of classifying each concept description into its true category (meaning that in training, the text description of each concept needs to predict the corresponding extracted positive label to learn the concept-level prompt), calculate the report-level similarity p and the concept-level similarity p', as follows:
[0061]
[0062] In the formula, u represents the global text feature in training or the visual feature in testing, U represents the concept description feature or the flattened image feature, the information in the local branch P is aggregated in a spatially weighted manner, and τ c Adjust the focus on a specific location.
[0063] Specifically, a radiology report can be output as a whole text to the network to obtain a vector feature as a global text feature. The text of the radiology report contains many medical concept entities, such as lung nodules, pneumonia, etc. In this embodiment, the corresponding explanation of the medical entity concept query mentioned in the report (for example, lung nodules refer to various sizes, clear or blurred edges on lung images...) is output as text to the network to extract concept description features.
[0064] Step S7, set the training round epoch to 20, the training batch size batch_size to 64, the learning rate learning_rate to 2e-5 and the optimizer AdamW, and use the loss function The inference model is trained (by optimizing global and local cues to minimize the overall objective).
[0065]
[0066] Step S8: Input the patient's chest X-ray image into the reasoning model, infer the existence of certain concepts and / or diseases, and determine the visual evidence of the concepts and / or diseases.
[0067] In order to verify the effectiveness of the present invention, Table 1 shows the comparison results of the inference model constructed by the present invention and other state-of-the-art methods on the zero-shot classification task. The area under the receiver operating characteristic curve (AUC) and accuracy (ACC) commonly used in binary classification problems are used as evaluation indicators. By comparing the results of previous models, it can be seen that the performance of the model trained by the present invention is significantly better than the existing model methods in multiple data sets.
[0068] Table 1 Comparison results of different models on zero-shot classification tasks
[0069]
[0070] The present invention also provides a device for implementing the text prompt learning method for enhancing X-ray image diagnosis, comprising:
[0071] An input module is used to input a patient chest X-ray image dataset, wherein the X-ray image dataset includes a radiology report and corresponding images and human annotated labels;
[0072] An extraction module, for extracting concepts, locations, and positivity from the radiology report using a named entity recognition method to form triple data;
[0073] An injection module, used for injecting external medical knowledge into the triple data to form new fine-grained data;
[0074] a first encoding module, configured to concatenate the learnable word embedding with the word embedding of the i-th concept to obtain a report-level prompt, and input the report-level prompt into a text encoder to generate a report-level prompt embedding for each concept;
[0075] A second encoding module, for inputting concept-level hints into a text encoder to generate concept-level hint embeddings for each concept;
[0076] A computational module, which is used to encode radiology reports using a text encoder to obtain global text features and concept description features, learn hints by maximizing the probability of classifying each concept description into its true category, and calculate report-level similarity p and concept-level similarity p';
[0077] The training module is used to set the training epochs, training batch size, learning rate, and optimizer, and use the loss function Train to obtain the inference model;
[0078] The reasoning module is used to input the patient's chest X-ray image into the reasoning model, infer the existence of certain concepts and / or diseases, and determine the visual evidence of the concepts and / or diseases.
[0079] In the prompt learning, the present invention uses easily accessible radiology reports as medical images to adapt the medical VLP model to the diagnosis of X-ray images, and its classification labels can be directly exported, so the method is less affected by image data limitations and annotation limitations; the present invention directly extracts rich pathological and anatomical information from radiology reports, and injects medical knowledge from knowledge bases or LLMs to enhance the medical VLP model. At the same time, it can be flexibly applied to medical classification tasks with or without image annotations. The report-concept text prompt learning (RC-TPL) proposed by the present invention is used to extract coarse-grained and fine-grained prompt embeddings to enhance X-ray image diagnosis. The present invention significantly outperforms the most advanced methods in several benchmark tests (such as ChestX-ray14, CheXpert) without using any labeled images. At the same time, it has good interpretability performance.
[0080] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.
Claims
1. A text prompt learning method for enhancing X-ray image diagnosis, characterized in that: The following steps are involved: S1. Input patient chest X-ray image dataset, which contains radiology reports and corresponding images and human annotated labels; S2, using a named entity recognition method to extract concepts, locations, and positivity in the radiology report to form triple data; S3, injecting external medical knowledge into the triple data to form new fine-grained data; S4, concatenating the learnable word embedding with the word embedding of the i-th concept to obtain the report-level prompt, and inputting the report-level prompt into a text encoder to generate a report-level prompt embedding for each concept; S5, input the concept-level prompts into a text encoder to generate concept-level prompt embeddings for each concept; S6. Use a text encoder to encode the radiology report to obtain global text features and concept description features, learn the prompt by maximizing the probability of classifying each concept description into its true category, and calculate the report-level similarity p and concept-level similarity p'; S7. Set the training rounds, training batch size, learning rate and optimizer, and use the loss function Train to obtain the inference model; S8. Input the patient's chest X-ray image into the reasoning model, infer the existence of certain concepts and / or diseases, and determine the visual evidence of the concepts and / or diseases.
2. The text prompt learning method for enhancing X-ray image diagnosis according to claim 1, characterized in that: The expression of the triple data in step S2 is: In the formula, represents the total number of concepts contained in a radiology report, c i is the name of a specific medical concept. i Anatomical position for medical concept,pos i Positive result for medical concept.
3. The text prompt learning method for enhancing X-ray image diagnosis according to claim 1, characterized in that: In step S4, the report level prompt is embedded The formula is: In the formula, For report-level prompts, Enc T is the text encoder, i is the i-th medical concept, and L is the total number of concepts.
4. The text prompt learning method for enhancing X-ray image diagnosis according to claim 1, characterized in that: The concept-level hint embedding in step S5 The formula is: In the formula, is a concept-level prompt, i is the i-th medical concept, and L is the total number of concepts.
5. The text prompt learning method for enhancing X-ray image diagnosis according to claim 1, characterized in that: In step S6, the report-level similarity p and the concept-level similarity p' are calculated using the following formula: In the formula, u represents the global text feature in training or the visual feature in testing, U represents the concept description feature or the flattened image feature, the information in the local branch P is aggregated in a spatially weighted manner, and τ c Adjust the focus on a specific location.
6. A device for implementing the text prompt learning method for enhancing X-ray image diagnosis according to any one of claims 1 to 5, characterized in that: include: An input module is used to input a patient chest X-ray image dataset, wherein the X-ray image dataset includes a radiology report and corresponding images and human annotated labels; An extraction module, for extracting concepts, locations, and positivity from the radiology report using a named entity recognition method to form triple data; An injection module, used for injecting external medical knowledge into the triple data to form new fine-grained data; a first encoding module, configured to concatenate the learnable word embedding with the word embedding of the i-th concept to obtain a report-level prompt, and input the report-level prompt into a text encoder to generate a report-level prompt embedding for each concept; A second encoding module, for inputting concept-level hints into a text encoder to generate concept-level hint embeddings for each concept; A computational module, which is used to encode radiology reports using a text encoder to obtain global text features and concept description features, learn hints by maximizing the probability of classifying each concept description into its true category, and calculate report-level similarity p and concept-level similarity p'; The training module is used to set the training epochs, training batch size, learning rate, and optimizer, and use the loss function Train to obtain the inference model; The reasoning module is used to input the patient's chest X-ray image into the reasoning model, infer the existence of certain concepts and / or diseases, and determine the visual evidence of the concepts and / or diseases.
