Disease classification method and system based on knowledge enhanced image text alignment

Abstract

The invention discloses a disease classification method and system based on knowledge enhanced image text alignment, and belongs to the technical field of artificial intelligence and digital medical treatment. In order to solve the problems that in an existing method, only medical entity information is considered to promote alignment between a medical image and a text report, and graph structure information of a knowledge graph is ignored, firstly, data of the medical image and the text report are preprocessed; connecting the processed medical image and report data with a medical knowledge graph UMLS to construct an external knowledge graph, constructing a disease classification network model, and training the disease classification network model by using the processed medical image and report data and the external knowledge graph; and finally, performing disease classification on the medical image of the new patient by using the trained disease classification network model. The diagnosis efficiency is improved by outputting the category information of the medical image of the diagnosed patient.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and digital medical technology, and specifically relates to a disease classification method and system based on knowledge-enhanced image-text alignment. Background Art

[0002] Modality is the way things are experienced and happen. We live in a world composed of multiple modal information, including visual information, auditory information, text information, olfactory information, etc. Among these modalities, vision and language are the two main modalities. In the medical field, visual and language modalities also exist, including medical images and paired text reports. By utilizing these data, the efficiency of doctors' diagnosis is fully improved. However, the existing medical vision and language pre-training methods follow the method of simply comparing images and texts to perform pre-training, ignoring the importance of medical field knowledge. Recently, learning the relationship between medical images and text reports by referring to knowledge graphs has been used by many researchers, which has improved the utilization rate of medical images and reports and enabled researchers to build more efficient medical vision and language pre-training models.

[0003] Medical image and text alignment is now mainly divided into three methods: global alignment, local alignment and multi-granularity alignment. Specifically, global alignment is to associate medical images and text reports from the same patient, local alignment is to correspond each word in the text to the area in the image, and multi-granularity alignment is to combine global alignment and local alignment. However, these methods are only aimed at images and texts themselves. Few studies consider the importance of expert knowledge in the medical field and explicitly use this knowledge to promote image-text alignment. The medical knowledge graph is a structured semantic knowledge base that contains a large number of medical entity concepts and the relationships between them. Its basic component unit is the "entity-relationship-entity" triple. It can be connected with medical images and texts to extract the corresponding entities and the relationships between them. By using the knowledge graph, the key medical information in images and texts can be learned more accurately.

[0004] Methods that promote image-text alignment by using knowledge graphs include: The ARL method is proposed in the article "Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge". By introducing the medical knowledge graph UMLS as an intermediate medium between images and texts, the entities of the medical knowledge graph are first aligned with the image and text respectively, and then the knowledge is fused with the image and text respectively through a multimodal encoder, and finally a multimodal vector representation after knowledge fusion is obtained, and then the obtained representation is input into the classification network to obtain the final classification result. The article "Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training" proposes a knowledge-boosted contrastive vision-language pre-training method Kobo, which integrates clinical knowledge into the learning of visual language semantic consistency. Use unbiased, open set sample knowledge representation to measure negative sample noise and supplement the correspondence between visual language mutual information and clinical knowledge. The paper "MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training" proposes a knowledge-enhanced medical visual language pre-training method MedKLIP, which uses a new triple extraction module to extract medical-related information, avoids unnecessary complexity of language grammar, and enhances supervision signals; it proposes a new triple encoding module to perform entity translation by querying the knowledge base to utilize the rich domain knowledge in the medical field.

[0005] The patent application number CN116797848A, "A disease localization method and system based on medical image text alignment", proposes a method for a medical multimodal pre-training model, which uses the alignment relationship of different granularities between medical images and texts to help match texts and medical images. However, it only focuses on the information of the image and text itself, and does not take into account the importance of external domain knowledge. It is composed of many medical entities and the relationships between them. Through external domain knowledge, the key entity information in medical images and texts and the relationship information between these entities can be better captured. This information can help medical images and texts to be better aligned.

[0006] In summary, it is feasible to perform disease classification by enhancing image-text alignment through reference knowledge. However, existing methods only consider using medical entity information to promote the alignment between medical images and text reports, ignoring the graph structure information of the knowledge graph itself, which can enrich the information of medical entities and help medical images and text reports to correspond. Summary of the invention

[0007] In view of the fact that existing methods only consider using medical entity information to promote the alignment between medical images and text reports, and ignore the problem of graph structure information of the knowledge graph itself, the present invention provides a disease classification method and system based on knowledge-enhanced image-text alignment.

[0008] In order to achieve the above objectives, the present invention adopts the following technical solutions: medical images and text reports are connected with the knowledge graph to construct image knowledge subgraphs and text report knowledge subgraphs, and graph contrast learning is used to align images and texts, and finally they are fused with image features and obtained through a classification network to obtain diagnosis results.

[0009] A disease classification method based on knowledge-enhanced image-text alignment includes the following steps:

[0010] Step 1: Preprocess medical images, text reports, and medical knowledge graph data to construct a medical knowledge graph closely related to the medical image text, specifically including the following steps:

[0011] Step 1.1, preprocessing medical images and text report data; Since there is some useless information in the text report, in order to improve the quality of the text report, first extract the sentences containing key fields in the text report to obtain the processed text report sample T = {T1, T2, …, T n}, where T i represents the i-th text report in the medical dataset, and n represents the number of text reports; then the medical image-text report-label tuple required for training is constructed, denoted as {(I1,T1,L1),(I2,T2,L2),…,(I n ,T n ,L n )}, where n represents the total number of samples, (I i ,T i ,L i ) represents the i-th pair of medical images, text reports and their corresponding disease labels;

[0012] Step 1.2: Based on the medical image-text report-label tuple obtained in step 1.1, use the entity extractor to extract the corresponding medical entity set ε from the UMLS medical knowledge graph. i ={ε i1 ,ε i2 ,ε i3 ,…,ε iV} and the relation set r i = {r i1 ,r i2 ,r i3 ,…,r iR}, where V represents the number of medical entities and R represents the number of relationships between entities. The external knowledge graph is constructed based on the obtained entity set and relationship set. Where N represents the number of medical knowledge graph triples, g i represents the i-th knowledge triple, h i ,r i and t i Represent the head entity, relationship and tail entity respectively;

[0013] Step 1.3, adopt the method of small batch data training, set the batch size to M, that is, each round of training is divided into S = n / M steps of iteration, and M medical image-text report-label tuple samples are input simultaneously during each training step to obtain small batch medical image samples I, small batch text report samples T and small batch label samples L;

[0014] Step 1.4: Perform random cropping, scaling, horizontal flipping, random color jittering, data format conversion, and normalization operations on the small batch of medical image samples I in step 1.3 to perform medical image augmentation.

[0015] Step 2: Construct each network module of the disease classification network model, which specifically includes the following steps:

[0016] Step 2.1: Use the Vision-Transformer model to build an image coding network. The image coding network consists of 12 transformer blocks, each of which consists of a self-attention layer and a feedforward network layer.

[0017] Step 2.2: Use the Bert model to build an image-based text decoding network. The text decoding network consists of 12 transformer blocks, each of which consists of a causal self-attention layer, a cross-attention layer, and a feedforward network layer.

[0018] Step 2.3: Use the Vision-Transformer model in step 2.1 and the Bert model in step 2.2 to build a text generation network;

[0019] Step 2.4: Use the two-layer Graph Convolutional Networks model to build a graph encoding network g(·) to encode the relationship between nodes to enrich node embeddings;

[0020] Step 2.5: Build a graph-level projection head network; graph-level projection head g graph (·) is a single linear layer, denoted by g graph (·)=W graph (·), where Wgraph are the trainable parameters in the graph-level projection head network.

[0021] Step 3: construct an image knowledge subgraph and a text report knowledge subgraph based on the medical images, text reports, and medical knowledge graph data preprocessed in step 1 and the medical knowledge graph;

[0022] The text generation network obtained in step 3.1 and step 2.3 does not participate in the training. It only needs to load the pre-trained weights. The text generation network loaded with the pre-trained weights is used for each medical image I i Perform text generation to obtain the generated text for each medical image

[0023] Step 3.2: Generate text from the image obtained in step 3.1 The text report sample T obtained in step 1 is used to construct an imaging knowledge subgraph and a text report knowledge subgraph with the medical knowledge graph respectively;

[0024] For Medical Imaging I i For example, converting images into text Obtain the corresponding image text medical entity set through SCIBERT Among them, t represents the text from the image The number of medical entities extracted;

[0025] For the text report sample T, the preprocessed text report sample T is passed through SCIBERT to obtain the corresponding text report medical entity set Where m represents the text report T i The number of medical entities extracted;

[0026] The extracted image text medical entity set and text report medical entity set Corresponding to the external knowledge graph G, duplicate entities are filtered out, the corresponding entities and their neighbor information and the relationship between them are extracted to enrich their own information, and the image text knowledge subgraph is constructed. and text report knowledge subgraph in represents the node of the i-th image text knowledge subgraph, represents the entity-relationship-entity edge set of the i-th image text knowledge subgraph, represents the node of the i-th text report knowledge subgraph, Represents the entity-relationship-entity edge set of the i-th text report knowledge subgraph, the image text medical entity set and text report medical entity set The corresponding adjacency matrices are and

[0027] Step 4: Use the image knowledge subgraph and text report knowledge subgraph obtained in step 3 to train the disease classification network model in step 2, determine the training parameters of the disease classification network model, design an optimizer and loss function to update the disease classification network model, and obtain the optimized disease classification network model; the specific steps are as follows:

[0028] Step 4.1, process the adjacency matrix corresponding to the knowledge subgraph first;

[0029] For the image knowledge subgraph, first calculate the degree of each image entity node in the graph and obtain the degree matrix Then in the adjacency matrix Add a unit matrix to get a new adjacency matrix Then the normalization operation is performed, and the calculation method is

[0030] For the text report knowledge subgraph, we first calculate the degree of each text entity node in the graph and obtain the degree matrix Then in the adjacency matrix Add a unit matrix to get a new adjacency matrix Then the normalization operation is performed, and the calculation method is

[0031] Step 4.2: Input the graph data obtained in step 4.1 into step 2.4 in batches (32 in a batch) to obtain a graph coding network g(·), and use the graph coding network to aggregate the node information and structure information of the graph to recursively update the node representation;

[0032] For the image knowledge subgraph, the calculation method is:

[0033]

[0034] Among them, l represents the index of the graph convolutional network layer, represents the feature matrix of the image-text node at the lth layer, and the superscript IG indicates that the node comes from the image-text knowledge subgraph;

[0035] For the text report knowledge subgraph, it is calculated as follows:

[0036]

[0037] The superscript TG indicates that the node comes from the text report knowledge subgraph;

[0038] Step 4.3: Use the last layer output of the graph encoding network as feature representation and Among them, M represents the batch size. The feature representation of the image knowledge subgraph and the feature representation of the text report knowledge subgraph are reduced in dimension and input into the graph-level projection head network to obtain the corresponding latent representation and in, represents the latent representation of the i-th medical imaging knowledge subgraph, represents the latent representation of the i-th text report knowledge subgraph;

[0039] Step 4.4: Construct a graph-level contrastive learning loss. The graph-level contrastive learning loss brings the image text knowledge subgraph and text report knowledge subgraph samples belonging to the same pair closer together and pushes them further away from other samples. The formula is as follows:

[0040]

[0041] Where M represents the sample batch size, represents the latent representation of the generated text of the i-th medical image, represents the latent representation of the i-th text report, represents the latent representation of the generated text of the kth medical image, represents the latent representation of the k-th text report, and τ is the temperature hyperparameter; represents the contrast loss at the graph level, represents the image-to-text contrast loss, Represents the graph contrast loss for text to image;

[0042] Step 4.5: Input the augmented medical images into the image encoding network in step 2.1 in the same batch manner as step 4.2 to obtain the feature representation of the samples. Specifically, given a medical image I i , first split it into N blocks, and then add an additional [CLS] token to learn the overall information of the medical image to form an image token sequence Input the image token sequence into the image encoding network to obtain the medical image representation The obtained medical image feature representation is input into the image-level projection head to obtain the corresponding latent representation

[0043] Step 4.6: Aggregate the latent representations of the medical images obtained in step 4.5 to obtain the image [CLS] token vector representation Then perform weighted summation with the knowledge subgraph latent representation obtained in step 4.3 to obtain the knowledge fused image vector The obtained image [CLS] token vector is input into the classification network to obtain the predicted disease label

[0044] Step 4.7, construct disease classification loss, the formula is as follows:

[0045]

[0046] Where M represents the sample batch size, represents the predicted disease category label, f(·) represents the classification network, and the classification network f(·) is a linear classification layer. i Represents the disease category label of the medical image in step 1.1;

[0047] The final training loss is:

[0048]

[0049] Step 4.8. Repeat steps 4.1-4.7 in an iterative manner, train the disease classification network model a certain number of times, and use the Adam optimizer to update the parameters of the disease classification network model in a stochastic gradient descent manner.

[0050] Step 5: Use the disease classification network model trained in step 4 to preprocess the medical image text data according to the preprocessing method in step 1, input the preprocessed data into the classification network and output the disease classification results. The specific steps are as follows:

[0051] Step 5.1, performing a random augmentation operation on the patient's medical image to obtain a medical image augmentation sample;

[0052] Step 5.2: Input the augmented medical image and text samples into the trained disease classification network model to obtain the knowledge-fused medical image vector, and input it into the classification network to obtain the disease classification result.

[0053] A disease classification system based on knowledge-enhanced image-text alignment, the disease classification system is used to implement a disease classification method based on knowledge-enhanced image-text alignment, including a computer processor, a medical image-text data storage unit, a medical image-text data preprocessing unit, a medical knowledge graph data storage unit, a medical knowledge graph data preprocessing unit, a disease classification network model training unit, and a disease classification unit; wherein:

[0054] The medical image text data storage unit stores the medical image text data and loads it into the computer memory;

[0055] The medical knowledge graph data storage unit stores the medical knowledge graph data and loads it into the computer memory;

[0056] The medical image text data preprocessing unit executes step 1, extracts medical image text samples of batch size M from the memory in steps, and loads them into the graphics processor;

[0057] The medical knowledge graph data preprocessing unit executes step 1 to load the constructed medical knowledge graph data into the image processor;

[0058] The disease classification network model training unit uses the medical image text data and the medical knowledge graph data to perform steps 3-4 to obtain a disease classification network model;

[0059] The disease classification unit executes step 5 to obtain a disease classification result;

[0060] The specific data processing and calculation work in all units is completed by the computer processor.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] 1. The disease classification method based on knowledge-enhanced image-text alignment proposed in this invention constructs the medical image text dataset and the knowledge graph into a knowledge structure subgraph composed of "entity-relationship-entity" triples. This method can capture the key medical entity information in medical images and text reports, and effectively improve the utilization rate of medical image text data and the quality of data representation;

[0063] 2. The knowledge structure graph designed by the disease classification method based on knowledge-enhanced image-text alignment proposed in the present invention not only takes into account the information of medical entities, but also the relationship and structural information between them. By introducing the relationship between medical entities, the representation of medical entities themselves can be enriched, and graph comparative learning is used to help medical images and text reports to be more effectively aligned;

[0064] 3. The disease classification method and system based on knowledge-enhanced image-text alignment proposed in the present invention can effectively handle disease tasks. It uses the feature vector representation of the knowledge structure graph containing rich structural information, and fuses it with the medical image vector representation to obtain the knowledge-fused image representation, and then performs disease classification, which can improve the accuracy of disease classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A disease classification method based on knowledge-enhanced image-text alignment and a structure diagram of a disease classification model in the system according to the present invention;

[0066] Figure 2 This is a structural diagram of a disease classification module in a disease classification method based on knowledge-enhanced image-text alignment described in the present invention;

[0067] Figure 3 A computer-implemented system structure diagram of a disease classification method based on knowledge-enhanced image-text alignment according to the present invention;

[0068] Figure 4 This is a flowchart of a disease classification method based on knowledge-enhanced image-text alignment described in the present invention. DETAILED DESCRIPTION

[0069] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0070] The disease classification method based on knowledge-enhanced image-text alignment proposed in the present invention is implemented through a computer program. Figure 3 The following is a system structure diagram of a computer implementation, where "computer processor" represents the type of computing resources, which can be a separate graphics card that supports GPU computing, or a local server or cloud server that supports GPU computing. Figure 1 , 2 The model structure diagram shown and Figure 4 The method flow chart in the embodiment of the present invention is specifically and detailedly described. The implementation method mainly includes the following key contents:

[0071] Step 1: Preprocess medical images, text reports, and medical knowledge graph data to construct a medical knowledge graph closely related to medical image texts;

[0072] Step 1.1, preprocessing medical images and text report data; the dataset used in the present invention is MIMIC-CXR, which is a large publicly available chest X-ray and radiology report dataset, which contains 377,110 chest X-rays and 227,835 radiology reports, covering 14 diseases. Since there is some useless information in the radiology text report, in order to improve the quality of the text report, it is first divided into single sentences according to the delimiter, and the sentences containing key fields in the text report are first extracted to obtain the processed text report sample T = {T1, T2, ..., T n}, where T i represents the i-th text report in the medical dataset, and n represents the number of reports; then the medical image-text report-label tuple required for training is constructed, denoted as {(I1,T1,L1),(I2,T2,L2),…,(I n ,T n ,L n )}, where n represents the total number of samples, (I i ,Ti ,L i ) represents the i-th pair of medical images, text reports and their corresponding disease labels;

[0073] Step 1.2: Based on the medical image-text report-label tuple obtained in step 1.1, use the entity extractor to extract the corresponding medical entity set ε from the UMLS medical knowledge graph. i ={ε i1 ,ε i2 ,ε i3 ,…,ε iV} and the relation set r i = {r i1 ,r i2 ,r i3 ,…,r iR}, where V represents the number of medical entities and R represents the number of relationships between entities. The external knowledge graph is constructed based on the obtained entity set and relationship set. Where N represents the number of medical knowledge graph triples, g i represents the i-th knowledge triple, h i ,r i and t i Represent the head entity, relationship and tail entity respectively;

[0074] Step 1.3: adopt the small batch data training method and set the batch size to 32, that is, each round of training is divided into 12796 steps of iteration. During each training step, 32 medical image-text report-label tuple samples are input simultaneously to obtain small batch medical image samples I, small batch text report samples T and small batch label samples L;

[0075] Step 1.4: The small batch of medical image samples I in step 1.3 are randomly cropped, scaled, horizontally flipped, randomly color jittered, converted to data format, and normalized to perform medical image augmentation. Specifically, for each medical image sample, random cropping is performed in sequence, where the cropped image size is [224, 224]; the image is horizontally mirrored with a parameter of 0.5; a random color jitter operation with a parameter of 0.8 is performed; finally, the processed image is converted into a tensor data with a dimension of [32, 3, 224, 224] and normalized with a mean of 0.5862 and a variance of 0.2795; 32 is the batch size, 3 is the number of RGB channels, and [224, 224] is the medical image size.

[0076] Step 2: Construct each network module of the disease classification network model, which specifically includes the following steps:

[0077] Step 2.1, use the Vision-Transformer model to build an image coding network. The image coding network consists of 12 transformer blocks, each of which consists of a self-attention layer and a feedforward network layer. The input dimension of the image coding network is a small batch of image samples of [32, 3, 224, 224], where 32 is the batch size, 3 is the number of color channels of the medical image, and [224, 224] is the size of the medical image. Its output dimension is a medical image tensor representation of [32, 50, 768], where 50 is the sum of the number of [CLS] tokens and image block tokens, and 768 is the feature dimension of each image block token.

[0078] Step 2.2: Use the Bert model to build an image-based text decoding network. The text decoding network consists of 12 transformer blocks, each of which consists of a causal self-attention layer, a cross-attention layer, and a feedforward network layer. The input of the text decoding network is the tensor representation of the medical image obtained in step 2.1, and its output is a text list of length 32, each element of which is a text description of a medical image.

[0079] Step 2.3: Use the Vision-Transformer model in step 2.1 and the Bert model in step 2.2 to build a text generation network;

[0080] Step 2.4: Use the two-layer Graph Convolutional Networks model to build a graph encoding network g(·) to encode the relationship between nodes to enrich node embedding; its input dimension is [32, num_nodes, 512] node feature matrix and dimension is [32, num_nodes, num_nodes] adjacency matrix, and its output dimension is [32, 512] knowledge structure graph tensor representation;

[0081] Step 2.5: Build a graph-level projection head network; graph-level projection head g graph (·) is a single linear layer, denoted by g graph (·)=W graph (·), where W graph are the trainable parameters in the graph-level projection head network; its input dimension is [32,768] and its output dimension is [32,512].

[0082] Step 3: construct an image knowledge subgraph and a text report knowledge subgraph based on the medical images, text reports, and medical knowledge graph data preprocessed in step 1 and the medical knowledge graph;

[0083] The text generation network obtained in step 3.1 and step 2.3 does not participate in the training. It only needs to load the pre-trained weights. The text generation network loaded with the pre-trained weights is used for each medical image I i Perform text generation to obtain the generated text for each medical image Specifically, a small batch of image samples with a dimension of [32, 3, 224, 224] is input into the text generation network, and the network outputs a text list of length 32, each element of which is a text description of a medical image;

[0084] Step 3.2: Generate text from the image obtained in step 3.1 The text report T obtained in step 1 i Construct imaging knowledge subgraph and text report knowledge subgraph respectively with medical knowledge graph;

[0085] For Medical Imaging I i For example, converting images into text Obtain the corresponding image text medical entity set through SCIBERT Among them, t represents the text from the image The number of medical entities extracted;

[0086] For the text report sample T, the preprocessed text report sample T is passed through SCIBERT to obtain the corresponding text report medical entity set Where m represents the text report T i The number of medical entities extracted;

[0087] The extracted image text medical entity set and text report medical entity set Corresponding to the external knowledge graph G, filtering duplicate entities and constructing image text knowledge subgraphs and text report knowledge subgraph in represents the node of the i-th image text knowledge subgraph, represents the entity-relationship-entity edge set of the i-th image text knowledge subgraph, represents the node of the i-th text report knowledge subgraph, Represents the entity-relationship-entity edge set of the i-th text report knowledge subgraph, the image text medical entity set and text report medical entity set The corresponding adjacency matrices are and

[0088] Step 4: Use the image knowledge subgraph and text report knowledge subgraph obtained in step 3 to train the disease classification network model in step 2, determine the training parameters of the disease classification network model, design an optimizer and loss function to update the disease classification network model, and obtain an optimized disease classification network model;

[0089] Step 4.1, process the adjacency matrix corresponding to the knowledge subgraph first;

[0090] For the image knowledge subgraph, first calculate the degree of each image entity node in the graph and obtain the degree matrix Then in the adjacency matrix Add a unit matrix to get a new adjacency matrix Then the normalization operation is performed, and the calculation method is

[0091] For the text report knowledge subgraph, we first calculate the degree of each text entity node in the graph and obtain the degree matrix Then in the adjacency matrix Add a unit matrix to get a new adjacency matrix Then the normalization operation is performed, and the calculation method is

[0092] Normalized adjacency matrix The sizes are [32,img_num_nodes,img_num_nodes] and [32,txt_num_nodes,txt_num_nodes] respectively, where 32 represents the batch size, img_num_nodes represents the number of nodes in the image text knowledge subgraph, and txt_num_nodes represents the number of nodes in the text report knowledge subgraph;

[0093] Step 4.2: Input the graph data obtained in step 4.1 into step 2.4 in batches to obtain a graph coding network g(·), and use the graph coding network to aggregate the node information and structure information of the graph to recursively update the node representation;

[0094] For the image knowledge subgraph, the calculation method is:

[0095]

[0096] Among them, l represents the index of the graph convolutional network layer, Represents the image text node feature matrix of the lth layer. The superscript IG indicates that the node comes from the image text knowledge subgraph. Its dimension size is [32, max_num_nodes, 512]. max_num_nodes is the maximum number of entity nodes in the same batch of samples. The insufficient number of entity nodes in the same batch will be filled with 0.

[0097] For the text report knowledge subgraph, it is calculated as follows:

[0098]

[0099] The superscript TG indicates that the node comes from the text report knowledge subgraph, and its output dimension size is [32, max_num_nodes, 512];

[0100] Step 4.3: Use the last layer output of the graph encoding network as feature representation and Among them, M represents the batch size. The feature representation of the image knowledge subgraph and the feature representation of the text report knowledge subgraph are reduced in dimension and input into the graph-level projection head network to obtain the corresponding latent representation and in, represents the latent representation of the i-th medical imaging knowledge subgraph, represents the latent representation of the i-th text report knowledge subgraph, whose dimension is [32,512];

[0101] Step 4.4: Use the latent representation z of the medical image text knowledge subgraph in a small batch (32 pairs) IG and text report knowledge subgraph latent representation z TG Calculate graph-level contrast loss: The formula is as follows:

[0102]

[0103] Among them, 256 represents the sample batch size, represents the latent representation of the generated text of the i-th medical image, represents the latent representation of the i-th text report, represents the latent representation of the generated text of the kth medical image, represents the latent representation of the k-th text report, and τ is the temperature hyperparameter; represents the contrast loss at the graph level, represents the image-to-text contrast loss, Represents the graph contrast loss for text to image;

[0104] Step 4.5: Input the augmented medical images into the image encoding network in step 2.1 in the same batch manner as step 4.2 to obtain the feature representation of the samples. Specifically, given a medical image I i , first split it into N blocks, and then add an additional [CLS] token to learn the overall information of the medical image to form an image token sequence Input the image token sequence into the image encoding network to obtain the medical image representation Its dimension is [50,768]. For a batch of images (32), a small batch image tensor representation with a dimension of [32,50,768] can be obtained. The obtained medical image feature representation is input into the graph-level projection head to obtain the corresponding latent representation. Its dimensions are [32,50,512];

[0105] The latent representations of medical images obtained in step 4.6 and step 4.5 are aggregated to obtain the image [CLS] token vector representation Then perform weighted summation with the knowledge subgraph latent representation obtained in step 4.3 to obtain the knowledge fused image vector Specifically, we first introduce a hyperparameter β to integrate and The two latent representations are combined into a knowledge subgraph feature Then, a hyperparameter α is introduced to fuse the fused knowledge subgraph features and the latent representation of the medical image to obtain the knowledge fused image [CLS] token vector representation Its dimension is [32,512]. The newly obtained image [CLS] token vector is input into the disease classification network in step S26 to obtain the predicted disease label.

[0106] Step 4.7, construct disease classification loss, the formula is as follows:

[0107]

[0108] Where M represents the sample batch size, represents the predicted disease category label, f(·) represents the classification network, and the classification network f(·) is a linear classification layer with an input dimension of [32,512] and an output dimension of [32,14]. i Represents the disease category label of the medical image in step 1.1;

[0109] The final training loss is:

[0110]

[0111] Step 4.8: Repeat steps 4.1-4.7 in an iterative manner, train the disease classification network model a certain number of times, and use the Adam optimizer to update the parameters of the disease classification network model in a stochastic gradient descent manner. Specifically, the weight decay rate of the Adam optimizer is set to 0.02. The initial learning rate is 3e -4 , the learning rate follows cosine decay and decays to 1e -6, train the disease classification network model for 20 rounds. After the training, determine the parameters of the disease classification network model.

[0112] Step 5: Use the disease classification network model trained in step 4 to preprocess the medical image text data according to the preprocessing method in step 1, input the preprocessed data into the classification network and output the disease classification results. The specific steps are as follows:

[0113] Step 5.1, performing a random augmentation operation on the patient's medical image to obtain a medical image augmentation sample;

[0114] Step 5.2: Input the augmented medical image and text samples into the trained disease classification network model to obtain the knowledge-fused medical image vector, which is then input into the classification network to obtain the disease classification result. The classification results are shown in Table 1:

[0115] Table 1 Disease classification results

[0116]

[0117] As attached Figure 3 As shown, a disease classification system based on knowledge-enhanced image-text alignment is used to implement a disease classification method based on knowledge-enhanced image-text alignment, including a computer processor, a medical image-text data storage unit, a medical image-text data preprocessing unit, a medical knowledge graph data storage unit, a medical knowledge graph data preprocessing unit, a disease classification network model training unit, and a disease classification unit. Among them:

[0118] The medical image text data storage unit stores the medical image text data and loads it into the computer memory;

[0119] The medical knowledge graph data storage unit stores the medical knowledge graph data and loads it into the computer memory;

[0120] The medical image text data preprocessing unit executes step 1, extracts medical image text samples of batch size M from the memory in steps, and loads them into the graphics processor;

[0121] The medical knowledge graph data preprocessing unit executes step 1 to load the constructed medical knowledge graph data into the image processor;

[0122] The disease classification network model training unit uses the medical image text data and the medical knowledge graph data to perform steps 3-4 to obtain a disease classification network model;

[0123] The disease classification unit executes step 5 to obtain a disease classification result;

[0124] The specific data processing and calculation work in all units is completed by the computer processor.

[0125] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A disease classification method based on knowledge-enhanced image-text alignment, characterized in that: The following steps are involved: Step 1: Preprocess medical images, text reports, and medical knowledge graph data to construct a medical knowledge graph closely related to medical image texts; Step 2: Construct each network module of the disease classification network model; Step 3: construct an image knowledge subgraph and a text report knowledge subgraph based on the medical images, text reports, and medical knowledge graph data preprocessed in step 1 and the medical knowledge graph; Step 4: Use the image knowledge subgraph and text report knowledge subgraph obtained in step 3 to train the disease classification network model in step 2, determine the training parameters of the disease classification network model, design an optimizer and loss function to update the disease classification network model, and obtain an optimized disease classification network model; Step 5: Use the disease classification network model trained in step 4 to preprocess the medical image text data according to the preprocessing method in step 1, input the preprocessed data into the classification network and output the disease classification result.

2. The disease classification method based on knowledge-enhanced image-text alignment according to claim 1 is characterized in that: The step 1, preprocessing medical images, text reports and medical knowledge graph data to construct a medical knowledge graph closely related to the medical image text, specifically includes the following steps: Step 1.1: Preprocess medical image and text report data; first extract sentences containing key fields in the text report to obtain the processed text report sample T = {T1, T2, …, T n }, where T i represents the i-th text report in the medical dataset, and n represents the total number of text report samples; then the medical image-text report-label tuple required for training is constructed, denoted as {(I1,T1,L1),(I2,T2,L2),…,(I n ,T n ,L n )}, where (I i ,T i ,L i ) represents the i-th pair of medical images, text reports and their corresponding disease labels; Step 1.2: Based on the medical image-text report-label tuple obtained in step 1.1, use the entity extractor to extract the corresponding medical entity set ε from the UMLS medical knowledge graph. i ={ε i1 ,ε i2 ,ε i3 ,…,ε iV } and the relation set r i = {r i1 ,r i2 ,r i3 ,…,r iR }, where V represents the number of medical entities and R represents the number of relationships between entities. The external knowledge graph is constructed based on the obtained entity set and relationship set. Where N represents the number of medical knowledge graph triples, g i represents the i-th knowledge triple, h i ,r i and t i Represent the head entity, relationship and tail entity respectively; Step 1.3, adopt the method of small batch data training, set the batch size to M, that is, each round of training is divided into S = n / M steps of iteration, and M medical image-text report-label tuple samples are input simultaneously during each training step to obtain small batch medical image samples I, small batch text report samples T and small batch label samples L; Step 1.4: Perform random cropping, scaling, horizontal flipping, random color jittering, data format conversion, and normalization operations on the small batch of medical image samples I in step 1.3 to perform medical image augmentation.

3. The disease classification method based on knowledge-enhanced image-text alignment according to claim 2 is characterized in that: The step 2, constructing each network module of the disease classification network model, specifically includes the following steps: Step 2.1: Use the Vision-Transformer model to build an image coding network. The image coding network consists of 12 transformer blocks, each of which consists of a self-attention layer and a feedforward network layer. Step 2.2: Use the Bert model to build an image-based text decoding network. The text decoding network consists of 12 transformer blocks, each of which consists of a causal self-attention layer, a cross-attention layer, and a feedforward network layer. Step 2.3: Use the Vision-Transformer model in step 2.1 and the Bert model in step 2.2 to build a text generation network; Step 2.4: Use the two-layer Graph Convolutional Networks model to build a graph encoding network g(·) to encode the relationship between nodes to enrich node embedding; Step 2.5: Build a graph-level projection head network; graph-level projection head g graph (·) is a single linear layer, denoted by g graph (·)=W graph (·), where W graph are the trainable parameters in the graph-level projection head network.

4. The disease classification method based on knowledge-enhanced image-text alignment according to claim 3 is characterized in that: The step 3, constructing an image knowledge subgraph and a text report knowledge subgraph according to the medical images, text reports and medical knowledge graph data preprocessed in step 1 and the medical knowledge graph, specifically comprises the following steps: The text generation network obtained in step 3.1 and step 2.3 does not participate in training, only the pre-trained weights are loaded, and the text generation network loaded with the pre-trained weights is used to generate a text for each medical image I i Perform text generation to obtain the generated text for each medical image Step 3.2: Generate text from the image obtained in step 3.1 The text report sample T obtained in step 1 is used to construct an imaging knowledge subgraph and a text report knowledge subgraph with the medical knowledge graph respectively; For Medical Imaging I i For example, converting images into text Obtain the corresponding image text medical entity set through SCIBERT Among them, t represents the text from the image The number of medical entities extracted; For the text report sample T, the preprocessed text report sample T is passed through SCIBERT to obtain the corresponding text report medical entity set Where m represents the text report T i The number of medical entities extracted; The extracted image text medical entity set and text report medical entity set Corresponding to the external knowledge graph C, filtering duplicate entities and constructing the image text knowledge subgraph and text report knowledge subgraph in represents the node of the i-th image text knowledge subgraph, represents the entity-relationship-entity edge set of the i-th image text knowledge subgraph, represents the node of the i-th text report knowledge subgraph, Represents the entity-relationship-entity edge set of the i-th text report knowledge subgraph, image text medical entity set and text report medical entity set The corresponding adjacency matrices are and 5. The disease classification method based on knowledge-enhanced image-text alignment according to claim 4 is characterized in that: The step 4 uses the image knowledge subgraph and text report knowledge subgraph obtained in step 3 to train the disease classification network model in step 2, determines the training parameters of the disease classification network model, designs an optimizer and a loss function to update the disease classification network model, and obtains an optimized disease classification network model; the specific steps are as follows: Step 4.1, process the adjacency matrix corresponding to the knowledge subgraph first; For the image knowledge subgraph, first calculate the degree of each image entity node in the graph and obtain the degree matrix Then in the adjacency matrix Add a unit matrix to get a new adjacency matrix Then the normalization operation is performed, and the calculation method is For the text report knowledge subgraph, we first calculate the degree of each text entity node in the graph and obtain the degree matrix Then in the adjacency matrix Add a unit matrix to get a new adjacency matrix Then the normalization operation is performed, and the calculation method is Step 4.2: Input the graph data obtained in step 4.1 into step 2.4 in batches to obtain a graph coding network g(·), and use the graph coding network to aggregate the node information and structure information of the graph to recursively update the node representation; For the image knowledge subgraph, the calculation method is: Among them, l represents the index of the graph convolutional network layer, represents the feature matrix of the image-text node at the lth layer, and the superscript IG indicates that the node comes from the image-text knowledge subgraph; For the text report knowledge subgraph, it is calculated as follows: The superscript TG indicates that the node comes from the text report knowledge subgraph; Step 4.3: Use the last layer output of the graph encoding network as feature representation and Among them, M represents the batch size. The feature representation of the image knowledge subgraph and the feature representation of the text report knowledge subgraph are reduced in dimension and input into the graph-level projection head network to obtain the corresponding latent representation in, represents the latent representation of the i-th medical imaging knowledge subgraph, represents the latent representation of the i-th text report knowledge subgraph; Step 4.4: Construct a graph-level contrastive learning loss. The graph-level contrastive learning loss brings the image text knowledge subgraph and text report knowledge subgraph samples belonging to the same pair closer together and pushes them further away from other samples. The formula is as follows: Where M represents the sample batch size, represents the latent representation of the generated text of the i-th medical image, represents the latent representation of the i-th text report, represents the latent representation of the generated text of the kth medical image, represents the latent representation of the k-th text report, and τ is the temperature hyperparameter; represents the contrast loss at the graph level, represents the image-to-text contrast loss, Represents the graph contrast loss for text to image; Step 4.5: Input the augmented medical images into the image encoding network in step 2.1 in the same batch manner as step 4.2 to obtain the feature representation of the samples. Specifically, given a medical image I i , first split it into N blocks, and then add an additional [CLS] token to learn the overall information of the medical image to form an image token sequence Input the image token sequence into the image encoding network to obtain the medical image representation The obtained medical image feature representation is input into the graph-level projection head to obtain the latent representation of the medical image. Step 4.6: Aggregate the latent representations of the medical images obtained in step 4.5 to obtain the image [CLS] token vector representation Then perform weighted summation with the knowledge subgraph latent representation obtained in step 4.3 to obtain the knowledge fused image vector The obtained image [CLS] token vector is input into the classification network to obtain the predicted disease label Step 4.7, construct disease classification loss, the formula is as follows: Where M represents the sample batch size, represents the predicted disease category label, f(·) represents the classification network, and the classification network f(·) is a linear classification layer. i Represents the disease category label of the medical image in step 1.1; The final training loss is: Step 4.

8. Repeat steps 4.1-4.7 in an iterative manner, train the disease classification network model a certain number of times, and use the Adam optimizer to update the parameters of the disease classification network model in a stochastic gradient descent manner.

6. The disease classification method based on knowledge-enhanced image-text alignment according to claim 5 is characterized in that: The step 5 uses the disease classification network model trained in step 4 to preprocess the medical image text data according to the preprocessing method in step 1, inputs the preprocessed data into the classification network and outputs the disease classification result. The specific steps are as follows: Step 5.1, performing a random augmentation operation on the patient's medical image to obtain a medical image augmentation sample; Step 5.2: Input the augmented medical image and text samples into the trained disease classification network model to obtain the knowledge-fused medical image vector, and input it into the classification network to obtain the disease classification result.

7. A disease classification system based on knowledge-enhanced image-text alignment, characterized by: The disease classification system is used to implement a disease classification method based on knowledge-enhanced image-text alignment as described in any one of claims 1 to 6, comprising a computer processor, a medical image-text data storage unit, a medical image-text data preprocessing unit, a medical knowledge graph data storage unit, a medical knowledge graph data preprocessing unit, a disease classification network model training unit, and a disease classification unit.

8. The disease classification system based on knowledge-enhanced image-text alignment according to claim 7, characterized in that: The medical image text data storage unit stores the medical image text data and loads it into the computer memory; The medical knowledge graph data storage unit stores the medical knowledge graph data and loads it into the computer memory; The medical image text data preprocessing unit executes step 1, extracts medical image text samples of batch size M from the memory in steps, and loads them into the graphics processor; The medical knowledge graph data preprocessing unit executes step 1 to load the constructed medical knowledge graph data into the image processor; The disease classification network model training unit uses the medical image text data and the medical knowledge graph data to perform steps 3-4 to obtain a disease classification network model; The disease classification unit executes step 5 to obtain a disease classification result; The specific data processing and calculation work in all units is completed by the computer processor.

Images