A medical knowledge enhanced image-text pre-training system and method
By using medical entity filtering and Transformer modal interaction technology, the data dependency and interpretability issues of existing medical image and text pre-training systems have been resolved, thereby improving the accuracy of lesion localization and diagnostic capabilities.
Patent Information
- Application Number
- CN202211690422.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-27
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2042-12-27
AI Technical Summary
Existing medical image and text pre-training systems suffer from problems such as high data dependence, inability to understand professional medical knowledge, and low interpretability in medical scenarios, which limits their ability to diagnose diseases.
By employing medical entity filtering, medical description coding, and Transformer modal interaction technology, and through triplet processing and multimodal fusion training, the professional knowledge fusion capability of the image-text pre-training system is enhanced, thereby improving the accuracy and interpretability of lesion localization.
It significantly improves the diagnostic capabilities of the medical image pre-training system in various medical scenarios, enabling accurate lesion localization and enhancing the system's interpretability.
Smart Images
Figure CN115858817B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of medical artificial intelligence, and in particular to a medical knowledge enhanced image-text pre-training system and method. BACKGROUND
[0002] Pre-training system is an important way to solve the generalization of deep learning algorithm and reduce data requirements, especially using image-text multimodal data for pre-training. In the past few years, the image-text pre-training-fine-tuning paradigm has achieved great success in the field of natural images. However, in the medical field, the general pre-training system performs poorly. The main reason for this is that the medical field has characteristics such as data scarcity, dependence on professional knowledge, and high requirements for explainability.
[0003] After searching, the paper "Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing" published in 2022 European Conference on Computer Vision (ECCV) and published at https: / / arxiv.org / abs / 2204.09817 proposes a medical image-text pre-training algorithm based on report text characteristics. In the pre-training stage, it first uses the sequence independence of the report text for text enhancement, then uses the correspondence between the conclusions and findings in the report to adjust the text encoder in the pre-training system using classical contrastive learning, and finally uses the paired data of text and image to complete the training of the entire pre-training system using contrastive learning.
[0004] The prior art has the following disadvantages: high data dependence. Most current methods directly process raw text and use whether the image and text match as a rough supervision signal. This results in the need for a large amount of data pairs to provide sufficient information to understand the text; inability to understand professional medical knowledge. Medical tasks require highly professional knowledge assistance, but the current pre-training system lacks the ability to integrate knowledge, and the diagnosis ability for some diseases depends on the simple memory of the system for training data; low explainability. The current medical image-text pre-training model cannot accurately locate the lesion, lacks explainability, and greatly limits its ability to assist radiologists in diagnosis. In summary, we propose a medical knowledge enhanced image-text pre-training system and method. SUMMARY
[0005] The medical knowledge enhanced image-text pre-training system and method are designed for the medical scene, and solve the problems of high data dependency, inability to understand professional medical knowledge, and low interpretability of the existing medical image-text pre-training system.
[0006] To achieve the above object, the present application provides the following technical scheme: a medical knowledge enhanced image-text pre-training system, the system comprises:
[0007] A data acquisition module is configured to acquire text-image data pairs in data;
[0008] A triple processing module is configured to perform triple processing on medical text data acquired by the data acquisition module, wherein the triple contains three parts: entity name, entity position, and entity existence;
[0009] A multi-modal fusion training module is configured to encode images and combine entity description content, and fuse an encoder.
[0010] As a preferred embodiment of the present application, the triple processing module comprises:
[0011] An entity text extraction module is configured to extract entity text data and summarize entity triples;
[0012] An entity text conversion module is configured to convert entity names and entity positions into corresponding medical highly professional medical description sentences and position sentences, respectively;
[0013] A medical text encoder is configured to encode entity descriptions and schematic position sentences to obtain encoded triples.
[0014] As a preferred embodiment of the present application, the multi-modal fusion training module comprises:
[0015] An image encoder is configured to encode image information;
[0016] A fusion encoder is configured to fuse and compare encoded triples and image encodings to output training results.
[0017] A medical knowledge enhanced image-text pre-training system, the method comprises:
[0018] Step S1 acquires text-image data pairs in medical data;
[0019] Step S2 carries out preprocessing of the medical text into triples, to obtain encoded triples;
[0020] Step S3 carries out encoding processing on the image in the medical data, and carries out prediction and judgment in combination with the encoding information, and carries out supervision on the output result in combination with the encoded triples, to complete the training.
[0021] As a preferred embodiment of the present application, the step S2 is as follows:
[0022] Step S2.1, based on a medical text entity extraction method, processes the data pair to obtain an entity triple set capable of summarizing the text content;
[0023] Step S2.2, for the entity name in the triple, an additional medical knowledge base is used to further convert the entity name into a highly professional medical description sentence;
[0024] Step S2.3, for the entity position in the triple, a text template is used to convert the position word entity into a position sentence;
[0025] Step S2.4, after the triple is converted as above, a medical text encoder Φ Text is used to encode the entity description and the position sentence, and the existence of the entity in the triple is converted into 0 / 1 digital information, to obtain the encoded triple;
[0026] Step S2.5, all the encoded triples appearing in the text data are integrated to arrange the most common entity description sentences, which contain |Q| encoded entity descriptions, and these encoded descriptions are integrated to constitute an entity description query set Q;
[0027] Step S2.6, all the encoded triples appearing in the text data are integrated to arrange the most common entity position sentences, which contain |P| encoded entity positions, and these encoded positions are integrated to constitute a position candidate set P.
[0028] As a preferred embodiment of the present application, in the step S3, a specific text-image pair is considered alone, a letter is used to represent a specific image, and T is used to represent the text pair corresponding to X, and T is equivalent to a series of encoded triples according to step S2, each triple contains: entity description code e, entity position code p, and entity existence code s, and the detailed steps of the step S3 are as follows.
[0029] Step S3.1, an image encoder is used to encode the image: V=Φ Image (X), V is the image encoding feature, and Φ ImageRepresents an image encoder;
[0030] Step S3.2 Input the image encoding features V and the entity description query set Q into a fast modal fusion encoder Φ based on the Transformer structure. fusion The fusion encoder contains four mutual attention interaction layers. The output of the fusion encoder is passed through a position prediction layer and an existence determination layer to obtain two output position predictions. And the existence of prediction This represents the pre-trained system's response to each entity in the entity description query set Q;
[0031] Step S3.3: A segmentation prediction can be obtained by averaging the mutual attention weights in each mutual attention layer. The original image X has equal height and width, representing the system's confidence level regarding the location of the lesion in the original image;
[0032] Step S3.4 requires using the encoded triplet pair corresponding to T to output. as well as To supervise;
[0033] Step S3.5 combines deep learning gradient descent algorithm to complete the training of the pre-trained model.
[0034] In a preferred embodiment of the present invention, in step S3.3, the encoded triplet pair corresponding to T is output. as well as The details of the supervision are as follows;
[0035] For the existence of prediction Supervision is performed using binary cross-entropy, i.e. For location prediction Supervision is performed using contrastive loss. Positive samples are provided by the corresponding triples, while negative samples are randomly drawn from the candidate location set P. The final loss is obtained by weighted averaging of the two losses mentioned above, i.e., L = α1L cl +α2L cls α1 and α2 are two hyperparameters.
[0036] Compared with the prior art, the beneficial effects of the present invention are: the novel medical pre-training paradigm at the entity level reconstructed by the present invention using medical entity filtering technology; the knowledge fusion technology of medical image and text pre-training using medical description coding; and the better lesion localization function in the pre-training system implemented by the transformer structure. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.
[0038] Figure 1 This is a system structure diagram of the present invention.
[0039] Figure 2 This is a flowchart of the method of the present invention.
[0040] Figure 3 This is a flowchart of the medical text triplet preprocessing process of the present invention.
[0041] Figure 4 This is a flowchart of the multimodal fusion process of the present invention. Detailed Implementation
[0042] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
[0043] This invention provides a medical knowledge-enhanced image-text pre-training system, the system comprising:
[0044] The data acquisition module is used to acquire text-image data pairs from a vehicle's data.
[0045] The triplet quantification module is used for preprocessing medical text from the data acquisition module by quantifying it into triplets. Each triplet consists of three parts: entity name, entity location, and entity presence or absence.
[0046] The multimodal fusion training module is used to encode images and combine them with entity descriptions by incorporating them into the encoder.
[0047] Furthermore, the tripletization processing module includes:
[0048] The entity text extraction module is used to extract entity text data and summarize it into entity triplets;
[0049] The entity text conversion module is used to convert entity names and entity locations into corresponding highly specialized medical descriptions and location statements, respectively.
[0050] A medical text encoder, wherein the selected text encoder is used to encode entity descriptions and indication of location statements to obtain encoded triples.
[0051] Furthermore, the multimodal fusion training module includes:
[0052] An image encoder is used to encode image information;
[0053] A fusion encoder is used to combine encoded triples and image encoding to perform fusion comparison and output training results.
[0054] A medical knowledge-enhanced image-text pre-training system, the method comprising:
[0055] Step S1: Obtain text-image data pairs from the medical data;
[0056] Step S2 performs preprocessing on the medical text by tripling it to obtain coded triplets;
[0057] Step S3 encodes the images in the medical data, makes predictions based on the encoded information, and supervises the output results using the encoded triples to complete the training.
[0058] Furthermore, the details of step S2 are as follows:
[0059] Step S2.1: Based on the medical text entity extraction method, the data pairs are processed to obtain a set of entity triples that can summarize the text content;
[0060] Step S2.2 For the entity names in the triples, use an additional medical knowledge base to further transform the entity names into highly specialized medical descriptions.
[0061] Step S2.3 For the entity positions in the triples, use a text template to convert the positional word entities into positional statements;
[0062] After performing the above transformation on the triples in step S2.4, the medical text encoder Φ is used. Text The entity description and position statement are encoded, and the existence of the entity in the triplet is converted into 0 / 1 numeric information to obtain the encoded triplet.
[0063] Step S2.5: Collect all the coded triples that appear in the text data and sort out the most frequently occurring entity description statements, which contain |Q| coded entity descriptions. These coded descriptions are integrated to form an entity description query set Q.
[0064] Step S2.6: Take all the encoded triples that appear in the text data and sort out the most frequently occurring entity location statements, which contain |P| encoded entity locations. These encoded locations are integrated to form a location candidate set P.
[0065] Furthermore, in step S3, a specific text-image pair is considered separately, with a letter representing a specific image and T representing the text pair corresponding to X. According to step S2, T is equivalent to a series of encoded triples, each triple containing: entity description code e, entity location code p, and entity existence code s. The detailed steps of step S3 are as follows.
[0066] Step S3.1 Encode the image using an image encoder: V = Φ Image (X), V are image coding features, Φ Image Represents an image encoder;
[0067] Step S3.2 Input the image encoding features V and the entity description query set Q into a fast modal fusion encoder Φ based on the Transformer structure. fusion The fusion encoder contains four mutual attention interaction layers. The output of the fusion encoder is passed through a position prediction layer and an existence determination layer to obtain two output position predictions. And the existence of prediction This represents the pre-trained system's response to each entity in the entity description query set Q;
[0068] Step S3.3: A segmentation prediction can be obtained by averaging the mutual attention weights in each mutual attention layer. The original image X has equal height and width, representing the system's confidence level regarding the location of the lesion in the original image;
[0069] Step S3.4 requires using the encoded triplet pair corresponding to T to output. as well as To supervise;
[0070] Step S3.5 combines deep learning gradient descent algorithm to complete the training of the pre-trained model.
[0071] Furthermore, in step S3.3, the encoded triplet pair corresponding to T is output. as well as The details of the supervision are as follows;
[0072] For the existence of prediction Supervision is performed using binary cross-entropy, i.e. For location prediction Supervision is performed using contrastive loss. Positive samples are provided by the corresponding triples, while negative samples are randomly drawn from the candidate location set P. The final loss is obtained by weighted averaging of the two losses mentioned above, i.e., L = α1L cl +α2L cls α1 and α2 are two hyperparameters.
[0073] Example
[0074] Please combine further Figures 1-4 This invention provides a training method and system for a pre-training system based on medical knowledge, and its practical application is illustrated below:
[0075] Retrieves text-image data pairs from medical data;
[0076] First, the medical text is preprocessed using ternary grouping:
[0077] For all the provided medical texts, the data pairs are processed based on medical text entity extraction methods, such as "RadGraph", to obtain a set of entity triples that can summarize the text content. A triple contains three important components: entity name, entity location, and entity presence or absence.
[0078] For entity names in triples, additional medical knowledge bases, such as Wikipedia, are used to further transform the entity names into highly specialized medical descriptive statements.
[0079] For the entity location in the triple, a text template, such as "Its location is at [input location]", is used to convert the location word entity into a location statement;
[0080] After performing the above transformation on the triples, the medical text encoder Φ was used. Text For example, the "ClinicalBERT" model encodes entity descriptions and location statements, and converts the existence of entities in triples into 0 / 1 numerical information to obtain encoded triples.
[0081] By summarizing all the coded triples appearing in the text data, the most frequently occurring entity description statements are sorted out, which contain |Q| coded entity descriptions. These coded descriptions are integrated to form an entity description query set Q.
[0082] By summarizing all the encoded triples appearing in the text data, the most frequently occurring entity location statements are sorted out, which contain |P| encoded entity locations. These encoded locations are integrated to form a location candidate set P.
[0083] Next, we consider a specific text-image pair separately, using letters to represent a specific image and T to represent the text pair corresponding to X. Note that since the entity extraction and entity description encoding operations have been performed above, T can be equivalent to a series of encoded triples, each triple containing: entity description code e, entity location code p, and entity presence code s.
[0084] First, an image encoder, such as the common "Residual Network 50", is used to encode the image X: V = Φ Image (X), V are image coding features, Φ Image Represents an image encoder;
[0085] The image encoding features V and the entity description query set Q are input into a fast modality fusion encoder Φ based on the Transformer structure. fusion The fusion encoder comprises four mutual attention interaction layers. Of course, other variations and improvements can be adopted without affecting the main theme of this invention.
[0086] The fusion encoder consists of four mutual attention interaction layers. The output of the fusion encoder is passed through a position prediction layer and an existence determination layer to obtain two output position predictions. And the existence of prediction This represents the pre-trained system's response to each entity in the entity description query set Q;
[0087] Additionally, a segmentation prediction can be obtained by averaging the mutual attention weights in each mutual attention layer. The original image X has equal height and width, representing the system's confidence level regarding the location of the lesion in the original image;
[0088] During training, the system uses the encoded triplet pairs corresponding to T to output... as well as To supervise;
[0089] Specifically, for Supervision is performed using the classic Binary Cross Entropy (BCE) loss, i.e. for Supervision is performed using the classic "contrast loss". Positive samples are provided by corresponding triples, and negative samples are randomly drawn from the candidate location set P.
[0090]
[0091] The final loss is obtained by weighted averaging of the two losses mentioned above, i.e., L = α1L cl +α2L cls α1 and α2 are two hyperparameters, and this invention recommends that they both be selected as 1.0, but can be specifically selected by those skilled in the art.
[0092] By utilizing the supervised loss described above, and combining it with any mainstream deep learning gradient descent algorithm, the training of the pre-trained model can be completed.
[0093] Furthermore, this invention has two usage modes for inference: zero-sample mode and fine-tuning mode;
[0094] In zero-shot mode, given an image X and a description of a disease to be diagnosed or located, the image encoder Φ can be input. Image Text encoder Φ Text Fusion encoder Φ fusion Obtain the corresponding output as well as It allows for direct disease diagnosis. It can locate the lesion;
[0095] In fine-tuning mode, users can initialize the downstream network with any of the encoders mentioned above as needed, then perform classic neural network training on the downstream dataset, and finally conduct targeted usage tests.
[0096] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0097] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A medical knowledge-enhanced image-text pre-training system, characterized in that, The system includes: The data acquisition module is used to acquire text-image data pairs from medical data; The triplet processing module is used to preprocess the medical text from the data acquisition module by tripling it. Each triplet consists of three parts: entity name, entity location, and entity presence or absence. The multimodal fusion training module is used to encode images and combine them with entity descriptions by incorporating them into the encoder; A text-image pre-training method for a text-image pre-training system based on medical knowledge enhancement, the method comprising: Step S1: Obtain text-image data pairs from the medical data; Step S2 performs preprocessing on the medical text by tripling it to obtain coded triplets; Step S3 encodes the images in the medical data and uses the encoded triples to make predictions and judgments. At the same time, the output results are supervised using the encoded triples to complete the training. The details of step S2 are as follows: Step S2.1: Based on the medical text entity extraction method, the data pairs are processed to obtain a set of entity triples that can summarize the text content; Step S2.2 For the entity names in the triples, use an additional medical knowledge base to further transform the entity names into highly specialized medical descriptions. Step S2.3 For the entity positions in the triples, use a text template to convert the positional word entities into positional statements; After performing the above transformation on the triples in step S2.4, the medical text encoder is used. Encode medical description statements and positional statements, and convert the existence or non-existence of entities in triples into... The digital information is used to obtain the encoded triplet; Step S2.5: Combine all the encoded triples appearing in the text data and compile the most frequently occurring entity description statements, which include... These medical description statements, when integrated, constitute an entity description query set. ; Step S2.6: Combine all the encoded triples appearing in the text data and extract the most frequently occurring entity location statements, which include... These positional statements, whose encoded positions are integrated, constitute a candidate position set. ; In step S3, a specific text-image pair is considered individually, with letters representing a specific image. Substitute and The corresponding text pairs, and according to step S2, Equivalent to a series of encoded triples, each triple containing: entity description encoding Entity location coding Entity Existence Encoding The detailed steps of step S3 are as follows; Step S3.1 Encode the image using an image encoder: , Image encoding features, Represents an image encoder; Step S3.2 Encode image features and entity description query set Input a fast modal fusion encoder based on the Transformer architecture The fusion encoder contains four mutual attention interaction layers. The output of the fusion encoder passes through a position prediction layer and an existence determination layer to obtain two outputs: position prediction and existence determination. And existence prediction This represents the pre-trained system's ability to process entity description query sets. The responses of each entity in the process; Step S3.3: A segmentation prediction can be obtained by averaging the mutual attention weights in each mutual attention interaction layer. This segmentation prediction Compared with the original image Equal height and width represent the system's confidence level regarding the location of the lesion in the original image; Step S3.4 requires utilizing The corresponding encoded triplet pair output as well as To supervise; Step S3.5: Combine deep learning gradient descent algorithm to complete the training of the pre-trained model; In step S3.3, for The corresponding encoded triplet pair output as well as The details of the supervision are as follows; For the existence of prediction Supervision is performed using binary cross-entropy, i.e. For location prediction Supervision is performed using contrastive loss, with positive samples provided by corresponding triples and negative samples drawn from a candidate set of locations. Randomly selected from, i.e. The final loss is obtained by weighted averaging of the two losses mentioned above, i.e. , and These are two hyperparameters.
2. The medical knowledge enhancement image-text pre-training system according to claim 1, characterized in that, The tripletization processing module includes: The entity text extraction module is used to extract entity text data and summarize it into entity triplets; The entity text conversion module is used to convert entity names and entity locations into corresponding highly specialized medical descriptions and location statements, respectively. A medical text encoder is used to encode medical description statements and location statements to obtain encoded triples.
3. The medical knowledge enhancement image-text pre-training system according to claim 2, characterized in that, The multimodal fusion training module includes: An image encoder is used to encode image information; A fusion encoder is used to combine encoded triples and image encoding to perform fusion comparison and output training results.
Citation Information
Patent Citations
Pre-training method and device for medical multi-modal model
CN114972929A