Zero-sample medical image classification method in combination with knowledge graph
By combining medical knowledge graphs to train the image classification model, the problem of multimodal learning models in the prior art is easily overfitted in the zero-sample medical image classification task, achieving higher classification accuracy.
Patent Information
- Application Number
- CN202510196319.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-21
- Publication Date
- 2025-04-25
AI Technical Summary
Existing large multimodal learning models are prone to overfitting in zero-sample medical image classification tasks and require knowledge participation for correction.
In combination with medical knowledge graphs, the image classification model is trained, and by obtaining medical images and diagnostic descriptive text, the image encoder, text encoder and conditional attention network module are used to determine the classification category of images to avoid overfitting.
It effectively avoids overfitting problems, enhances the model's zero-sample transmission capability, and improves the accuracy of zero-sample medical image classification.
Smart Images
Figure CN119888369A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a zero-sample medical image classification method combined with a knowledge graph, and belongs to the technical field of artificial intelligence. Background Art
[0002] As a possible way to facilitate zero-shot learning, some visual language models have been proposed for small-scale data processing. The CLIP (Contrastive Language-Image Pre-training) multimodal machine learning model is considered to be one of the most promising techniques for achieving multimodal modeling, which demonstrates its impressive zero-shot classification performance by properly considering the natural distribution shift.
[0003] However, when it is simply applied to zero-shot medicine, existing large multimodal learning models are prone to overfitting when applied to zero-shot classification tasks and require knowledge involvement for correction. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a zero-shot medical image classification method combined with a knowledge graph, which combines the medical knowledge graph to train the image classification model, avoiding the problem that the existing large-scale multimodal learning model is prone to overfitting when applied to zero-shot classification tasks and requires knowledge participation for correction.
[0005] In order to achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions: A zero-shot medical image classification method combined with knowledge graph, comprising: Acquire a new category of medical images and add corresponding diagnostic descriptive text to the new category of medical images; Input the new category of medical images and their corresponding diagnostic descriptive texts into the pre-trained image classification model to determine the classification category that is closest to the diagnostic descriptive text; Classify new category medical images into the classification category that is closest to the diagnostic descriptive text; Among them, the training of the pre-trained image classification model includes: Select multiple images from the medical image data set for preprocessing, and add corresponding classification labels for representing image classification categories according to the image information of each image, and construct a training sample image set based on the multiple images with added classification labels; The image classification model is trained based on the medical knowledge graph and the training sample image set until any diagnostic descriptive text can correctly represent the classification category of the target sample image.
[0006] Furthermore, the image classification model includes: An image encoder for extracting image features of an input image; A text encoder, used to extract text features of the input text; The conditional attention network module is used to determine the classification category of the input image based on the input image features and text features.
[0007] Furthermore, the classification labels include: pulmonary edema, atelectasis, pulmonary consolidation, cardiac hypertrophy and pleural effusion.
[0008] Furthermore, the image classification model is trained based on the medical knowledge graph and the training set until any diagnostic descriptive text can correctly represent the classification category of the target sample image, including: Obtain the first sample image feature of the sample image in the training set and the sample text feature of the classification label; Obtain graph embedding features of medical knowledge graph; Based on the first sample image features and the graph embedding features, determining a fusion feature that aggregates similar entity relationships in the medical knowledge graph; Based on the cross-modal alignment strategy between fusion features and sample text features, a trained image classification model is obtained.
[0009] Furthermore, the determining of the fusion features that aggregate the similar entity relationships in the medical knowledge graph based on the first sample image features and the graph embedding features includes: The graph embedding feature of the medical knowledge graph is represented as: ; in: Medical knowledge graph includes entity sets E and relationship set R , which can be expressed by triples as: ; S represents a triple; t Represents the tail entity, Represents the tail entity projection vector; h Represents the tail entity t have r The head entity of the relationship, represents the head entity projection vector, Represents the tail entity projection vector; Represented in the knowledge graph embedding model Generate a triple in ; Based on the graph embedding features, determine the information representation of the relationship between entities in the medical knowledge graph; Determine the second sample image feature according to the information representation of the relationship between entities in the medical knowledge graph; The first sample image feature and the second sample feature are concatenated to obtain a fusion feature.
[0010] Furthermore, the information representation of the relationship between entities in the medical knowledge graph is determined based on the graph embedding feature, including: The information distribution weight of the relationship between entities in the medical knowledge graph is calculated by the following formula: ; in: Represent the information assignment weights between the relationships between entities in the medical knowledge graph; Indicates calculating the cosine similarity between two vectors; express function; The information representation of the relationship between entities in the medical knowledge graph is calculated by the following formula: ; in: Information representation of relations between entities in medical knowledge graphs.
[0011] Furthermore, determining the second sample image feature according to the information representation of the relationship between entities in the medical knowledge graph includes: The second sample image feature is calculated by the following formula: ; in: represents the second sample image feature; Information representation of the relationships between entities in the medical knowledge graph; Represents the fusion function that uses nonlinear transformation as information aggregation.
[0012] Furthermore, the cross-modal alignment strategy includes: The first loss function is used to minimize the contrast loss between the fusion feature and the sample text feature as the first strategy optimization goal to align the sample image and the diagnostic descriptive text; The second loss function is used to minimize the similarity loss between fusion features and sample text features as the second strategy optimization goal to align sequences within the same feature modality; The optimization objectives of the first strategy and the second strategy are integrated to obtain a cross-modal alignment strategy.
[0013] Furthermore, the contrast loss value is calculated using the following formula: ; in: Indicates the contrast loss value; Represents the fusion features of the sample image; Represents sample text features; represents the temperature parameter for adjusting the similarity distribution; represents an arbitrary diagnostic descriptive text, Included with Content that is not related to the corresponding sample text; Represents text The eigenvector of Represents the set of all possible diagnostic descriptive texts; It means calculating the similarity between two vectors.
[0014] Furthermore, the similarity loss value is calculated using the following formula: ; in: Represents the similarity loss value; represents the fusion features of the sample image, Indicates i The fusion features of sample images, Indicates j The fusion features of sample images; represents the sample text features, Indicates i Sample text features, Indicates j Sample text features; N Represents the total number of sample images; It means calculating the similarity between two vectors.
[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses medical knowledge graphs to train image classification models, avoiding the problem that existing large-scale multimodal learning models are prone to overfitting when applied to zero-sample classification tasks and require knowledge participation for correction.
[0016] 2. The present invention combines the relationships between modal features and within modal features at the same time, avoiding the distortion of multi-modality in the latent space, enhancing the model's zero-sample transmission capability, and improving the accuracy of zero-sample medical image classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flowchart of a zero-sample medical image classification method combined with a knowledge graph provided by an embodiment of the present invention; Figure 2 is a training flow chart of a pre-trained image classification model provided by an embodiment of the present invention; Figure 3 It is a principle framework diagram of a zero-sample medical image classification method combined with a knowledge graph provided by an embodiment of the present invention; Figure 4 is a schematic diagram of determining fusion features provided by an embodiment of the present invention; Figure 5 This is the cross-modal alignment strategy provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0019] In the description of the present invention, it is to be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the feature. In the description of the present invention, unless otherwise specified, "plurality" means two or more. Example
[0020] like Figure 1 As shown, a zero-shot medical image classification method combined with a knowledge graph includes: Acquire a new category of medical images and add corresponding diagnostic descriptive text to the new category of medical images; Input the new category of medical images and their corresponding diagnostic descriptive texts into the pre-trained image classification model to determine the classification category that is closest to the diagnostic descriptive text; Classify new category medical images into the classification category that is closest to the diagnostic descriptive text.
[0021] like Figure 2 As shown, the training of the pre-trained image classification model includes: Select multiple images from the medical image data set for preprocessing, and add corresponding classification labels for indicating the image classification categories according to the image information of each image, wherein the image preprocessing includes removing noise, enhancing contrast, and normalizing each medical image in the medical image data set; then performing data enhancement processing such as random rotation, flipping, and scaling on each medical image in the medical image data set; adjusting all medical images in the medical image data set to a uniform size; the classification labels include: pulmonary edema, atelectasis, pulmonary consolidation, cardiac hypertrophy, and pleural effusion; Collect multiple images with classification labels and build a training sample image set.
[0022] The image classification model is trained based on the medical knowledge graph and the training sample image set until any diagnostic descriptive text can correctly represent the classification category of the target sample image. Specifically: Obtain the first sample image feature of the sample image in the training set and the sample text feature of the classification label; Obtain graph embedding features of medical knowledge graph; Based on the first sample image features and the graph embedding features, a fusion feature that aggregates similar entity relationships in the medical knowledge graph is determined, including: The graph embedding feature of the medical knowledge graph is represented as: ; in: Medical knowledge graph includes entity sets E and relationship set R , which can be expressed by triples as: ; S represents a triple; t Represents the tail entity, Represents the tail entity projection vector; h Represents the tail entity t have r The head entity of the relationship, represents the head entity projection vector, Represents the tail entity projection vector; Represented in the knowledge graph embedding model Generate a triple in ; Based on the graph embedding features, the information representation of the relationship between entities in the medical knowledge graph is determined, including: The information distribution weight of the relationship between entities in the medical knowledge graph is calculated by the following formula: ; in: Represent the information assignment weights between the relationships between entities in the medical knowledge graph; Indicates calculating the cosine similarity between two vectors; express function; The information representation of the relationship between entities in the medical knowledge graph is calculated by the following formula: ; in: Information representation of the relationships between entities in the medical knowledge graph; According to the information representation of the relationship between entities in the medical knowledge graph, the second sample image features are determined, including: The second sample image feature is calculated by the following formula: ; in: represents the second sample image feature; Indicates that nonlinear transformation is used as a fusion function for information aggregation; The first sample image feature and the second sample feature are spliced together to obtain a fusion feature; Based on the cross-modal alignment strategy between fusion features and sample text features, a trained image classification model is obtained. Example
[0023] like Figure 1 As shown, a zero-shot medical image classification method combined with a knowledge graph includes: Acquire a new category of medical images and add corresponding diagnostic descriptive text to the new category of medical images; Input the new category of medical images and their corresponding diagnostic descriptive texts into the pre-trained image classification model to determine the classification category that is closest to the diagnostic descriptive text; Image classification models include: An image encoder for extracting image features of an input image; A text encoder, used to extract text features of the input text; The conditional attention network module is used to determine the classification category of the input image based on the input image features and text features; Classify new category medical images into the classification category that is closest to the diagnostic descriptive text.
[0024] like Figure 2 As shown, the training of the pre-trained image classification model includes: Select multiple images from the medical image data set for preprocessing, and add corresponding classification labels for indicating the image classification categories according to the image information of each image, wherein the image preprocessing includes removing noise, enhancing contrast, and normalizing each medical image in the medical image data set; then performing data enhancement processing such as random rotation, flipping, and scaling on each medical image in the medical image data set; adjusting all medical images in the medical image data set to a uniform size; the classification labels include: pulmonary edema, atelectasis, pulmonary consolidation, cardiac hypertrophy, and pleural effusion; Collect multiple images with classification labels and build a training sample image set.
[0025] The image classification model is trained based on the medical knowledge graph and the training sample image set until any diagnostic descriptive text can correctly represent the classification category of the target sample image. Specifically: like Figure 3 As shown, the first sample image feature of the sample image in the training set and the sample text feature of the classification label are obtained; The sample image is input into the image encoder to obtain the first sample image feature. In this embodiment, the image encoder of the Vision Transformer architecture is used as the backbone network to process the sample image. Specifically: Perform image patch embedding on the sample image to form an image patch sequence; Take the sequence of image patches as input to the visual transformer; Add position encoding (Position Embedding) to the image block sequence. Since the Transformer itself is an unordered model and lacks the perception of the position of the input sequence, in order to preserve the spatial position information between image blocks, it is necessary to add position encoding to each image block embedding vector; The sequence of image blocks after embedding and position encoding is input into the standard Transformer Encoder, which includes the following parts: Layer Normalization (Norm) and Multi-Head Self-Attention (MHSA). The role of Norm is to normalize the embedding vector of each image block, making the model more robust to input scale changes, thereby accelerating convergence and improving performance; Multi-Head Self-Attention captures global dependencies and long-distance context information by calculating the similarity between image blocks; It should be noted that in the figure Indicates the number of repetitions of the Transformer Encoder.
[0026] Input the classification label into the text encoder to obtain sample text features. In this embodiment, the biomedical knowledge BERT (Bidirectional Encoder Representations from Transformers) model is used to process the classification label text; Obtain graph embedding features of medical knowledge graph; Based on the first sample image features and the graph embedding features, a fusion feature that aggregates similar entity relationships in the medical knowledge graph is determined, including: The TransR model is used in the conditional attention network module to map entities and relations into separate spaces for representation learning. The graph embedding feature of the medical knowledge graph is represented as: ; in: Medical knowledge graph includes entity sets E and relationship set R , which can be expressed by triples as: ; S represents a triple; t Represents the tail entity, Represents the tail entity projection vector; h Represents the tail entity t have r The head entity of the relationship, represents the head entity projection vector, Represents the tail entity projection vector; Represented in the knowledge graph embedding model Generate a triple in ; like Figure 4 As shown in the figure, the conditional attention mechanism of the conditional attention network module is used to determine the information representation of the relationship between entities in the medical knowledge graph based on the graph embedding features, including: The information distribution weight of the relationship between entities in the medical knowledge graph is calculated by the following formula: ; in: Represent the information assignment weights between the relationships between entities in the medical knowledge graph; Indicates calculating the cosine similarity between two vectors; express function; The information representation of the relationship between entities in the medical knowledge graph is calculated by the following formula: ; in: Information representation of the relationships between entities in the medical knowledge graph; According to the information representation of the relationship between entities in the medical knowledge graph, the second sample image features are determined, including: Feature fusion is performed in the conditional attention network module, and the second sample image feature is calculated by the following formula: ; in: represents the second sample image feature; Indicates that nonlinear transformation is used as a fusion function for information aggregation; like Figure 3 As shown, the first sample image feature and the second sample feature are spliced together, specifically: First, the first sample image feature and the second sample feature are normalized (Norm) so that they have the same scale. Then, a multi-layer perceptron (MLP) is used to perform nonlinear changes on the first sample image feature and the second sample feature with the same scale. Finally, the output result is spliced with the first sample image feature and the second sample feature to obtain the fusion feature. like Figure 5 As shown in FIG. 1 , based on the cross-modal alignment strategy between the fusion features and the sample text features, a trained image classification model is obtained, wherein the cross-modal alignment strategy includes: The first loss function is used to minimize the contrast loss between the fusion features and the sample text features as the first strategy optimization goal to align the sample image and the diagnostic descriptive text. The contrast loss value is calculated using the following formula: ; in: Indicates the contrast loss value; Represents the fusion features of the sample image; Represents sample text features; represents the temperature parameter for adjusting the similarity distribution; represents an arbitrary diagnostic descriptive text, Included with Content that is not related to the corresponding sample text; Represents text The eigenvector of Represents the set of all possible diagnostic descriptive texts; Indicates calculating the similarity between two vectors; The second loss function is used to minimize the similarity loss between fusion features and sample text features as the second strategy optimization goal to align the sequences within the same feature modality. The similarity loss value is calculated using the following formula: ; in: Represents the similarity loss value; represents the fusion features of the sample image, Indicates i The fusion features of sample images, Indicates j The fusion features of sample images; represents the sample text features, Indicatesi Sample text features, Indicates j Sample text features; N Represents the total number of sample images; Indicates calculating the similarity between two vectors; The first strategy optimization objective and the second strategy optimization objective are integrated by the following formula: ; in: represents the total loss value of the cross-modal alignment strategy, represents the intra-modal alignment loss coefficient; A cross-modal alignment strategy is obtained with the optimization objective of minimizing the total loss value.
[0027] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A zero-shot medical image classification method combined with knowledge graph, characterized in that: include: Acquire a new category of medical images and add corresponding diagnostic descriptive text to the new category of medical images; Input the new category of medical images and their corresponding diagnostic descriptive texts into the pre-trained image classification model to determine the classification category that is closest to the diagnostic descriptive text; Classify new categories of medical images into the classification category that is closest to the diagnostic descriptive text; Among them, the training of the pre-trained image classification model includes: Select multiple images from the medical image dataset for preprocessing, and add a classification label for indicating the corresponding image classification category to each image according to the image information of each image; Based on multiple images with classification labels added, a training sample image set is constructed; The image classification model is trained based on the medical knowledge graph and the training sample image set until any diagnostic descriptive text can correctly represent the classification category of the target sample image.
2. The zero-shot medical image classification method combined with knowledge graph according to claim 1, characterized in that: The image classification model includes: An image encoder for extracting image features of an input image; A text encoder, used to extract text features of the input text; The conditional attention network module is used to determine the classification category of the input image based on the input image features and text features.
3. The zero-shot medical image classification method combined with knowledge graph according to claim 1, characterized in that: The classification labels include: pulmonary edema, atelectasis, pulmonary consolidation, cardiomegaly, and pleural effusion.
4. The zero-shot medical image classification method combined with knowledge graph according to claim 1, characterized in that: The image classification model is trained based on the medical knowledge graph and the training set until any diagnostic descriptive text can correctly represent the classification category of the target sample image, including: Obtain the first sample image feature of the sample image in the training set and the sample text feature of the classification label; Obtain graph embedding features of medical knowledge graph; Based on the first sample image features and the graph embedding features, determining a fusion feature that aggregates similar entity relationships in the medical knowledge graph; Based on the cross-modal alignment strategy between fusion features and sample text features, a trained image classification model is obtained.
5. The zero-shot medical image classification method combined with knowledge graph according to claim 4, characterized in that: The step of determining, based on the first sample image feature and the graph embedding feature, a fusion feature that aggregates similar entity relationships in the medical knowledge graph includes: The graph embedding feature of the medical knowledge graph is represented as: ； in: Medical knowledge graph includes entity sets E and relationship set R , which can be expressed by triples as: ； S represents a triple; t Represents the tail entity, Represents the tail entity projection vector; h Represents the tail entity t have r The head entity of the relationship, represents the head entity projection vector, Represents the tail entity projection vector; Represented in the knowledge graph embedding model Generate a triple in ; Based on the graph embedding features, determine the information representation of the relationship between entities in the medical knowledge graph; Determine the second sample image feature according to the information representation of the relationship between entities in the medical knowledge graph; The first sample image feature and the second sample feature are concatenated to obtain a fusion feature.
6. The zero-sample medical image classification method combined with knowledge graph according to claim 5, characterized in that: The information representation of the relationship between entities in the medical knowledge graph is determined based on the graph embedding feature, including: The information distribution weight of the relationship between entities in the medical knowledge graph is calculated by the following formula: ； in: Represent the information assignment weights between the relationships between entities in the medical knowledge graph; Indicates calculating the cosine similarity between two vectors; express function; The information representation of the relationship between entities in the medical knowledge graph is calculated by the following formula: ； in: Information representation of the relationships between entities in the medical knowledge graph; Represents tail entity t and tail entity t have r The head entity of the relationship h Perform the summation.
7. The zero-sample medical image classification method combined with knowledge graph according to claim 5, characterized in that: The determining of the second sample image feature according to the information representation of the relationship between entities in the medical knowledge graph includes: The second sample image feature is calculated by the following formula: ； in: represents the second sample image feature; Information representation of the relationships between entities in the medical knowledge graph; Represents the fusion function that uses nonlinear transformation as information aggregation.
8. The zero-sample medical image classification method combined with knowledge graph according to claim 4, characterized in that: The cross-modal alignment strategy includes: The first loss function is used to minimize the contrast loss between the fusion feature and the sample text feature as the first strategy optimization goal to align the sample image and the diagnostic descriptive text; The second loss function is used to minimize the similarity loss between fusion features and sample text features as the second strategy optimization goal to align sequences within the same feature modality; The optimization objectives of the first strategy and the second strategy are integrated to obtain a cross-modal alignment strategy.
9. The zero-shot medical image classification method combined with knowledge graph according to claim 8, characterized in that: The contrast loss value is calculated using the following formula: ； in: Indicates the contrast loss value; Represents the fusion features of the sample image; Indicates sample text features; represents the temperature parameter for adjusting the similarity distribution; represents an arbitrary diagnostic descriptive text, Included with Content that is not related to the corresponding sample text; Represents text The eigenvector of Represents the set of all possible diagnostic descriptive texts; It means calculating the similarity between two vectors.
10. The zero-shot medical image classification method combined with knowledge graph according to claim 8, characterized in that: The similarity loss value is calculated using the following formula: ； in: Represents the similarity loss value; represents the fusion features of the sample image, Indicates i The fusion features of sample images, Indicates j The fusion features of sample images; represents the sample text features, Indicates i Sample text features, Indicates j Sample text features; N Represents the total number of sample images; It means calculating the similarity between two vectors.