Medical image classification method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to a medical image classification method and device, a storage medium and an electronic device, which can acquire metadata information corresponding to a medical image to be classified; acquire medical text information corresponding to the medical image according to the metadata information, the medical text information including clinical diagnosis and treatment information and medical prior knowledge; and classify the medical image by a target classification model pre-trained according to the medical image and the medical text information.

Description

Technical Field

[0001] This disclosure relates to the field of medical data analysis, and more specifically, to a medical image classification method, apparatus, storage medium, and electronic device. Background Technology

[0002] Multimodality refers to the multiple ways in which information is carried and presented. Multimodality carries rich, heterogeneous, and diverse information, possessing immense value. Therefore, in the field of computer science, multimodal research has gradually become a hot topic in both academia and industry.

[0003] In the medical field, researchers use machine learning to integrate data from multiple modalities, fusing features from each modality to improve the accuracy of target prediction. However, unlike multimodal fusion in general domains, current multimodal information in the medical imaging field often focuses on fusing and reasoning information from different examination techniques. For example, in brain image classification models, auxiliary medical references are typically provided by fusing results from imaging techniques such as computed tomography (CT scan), positron emission tomography (PET), magnetic resonance imaging (MRI), and functional magnetic resonance imaging (fMRI). Summary of the Invention

[0004] The purpose of this disclosure is to provide a medical image classification method, device, storage medium, and electronic device.

[0005] In a first aspect, this disclosure provides a medical image classification method, the method comprising:

[0006] Obtain the metadata information corresponding to the medical images to be classified;

[0007] Based on the metadata information, obtain the medical text information corresponding to the medical image, wherein the medical text information includes clinical diagnosis and treatment information and medical prior knowledge;

[0008] The medical images are classified using a pre-trained target classification model based on the medical images and the medical text information.

[0009] Optionally, the metadata information includes user identification information of the target user corresponding to the medical image, and obtaining the medical text information corresponding to the medical image based on the metadata information includes:

[0010] The clinical diagnosis and treatment information of the target user is determined based on the user identification information;

[0011] The medical prior knowledge associated with the target user is determined based on the clinical diagnosis and treatment information.

[0012] Optionally, classifying the medical images based on the medical images and the medical text information using a pre-trained target classification model includes:

[0013] A multimodal knowledge graph is constructed based on the medical images, the clinical diagnosis and treatment information, and the medical prior knowledge. The multimodal knowledge graph represents the image features of the medical images and the text features of the medical text information.

[0014] The medical images are classified using the target classification model based on the multimodal knowledge graph.

[0015] Optionally, the step of constructing a multimodal knowledge graph based on the medical images, the clinical diagnosis and treatment information, and the prior medical knowledge includes:

[0016] Based on the clinical diagnosis and treatment information, a first knowledge graph corresponding to the target user is constructed. The first knowledge graph includes a central node and multiple first nodes connected to the central node. The central node is a node corresponding to the target user, and the first nodes represent the diagnosis and treatment information of the target user.

[0017] Based on the medical prior knowledge, a second knowledge graph is constructed on the basis of the first knowledge graph. The second knowledge graph includes the central node, a plurality of first nodes connected to the central node, and a plurality of second nodes connected to the first nodes. The second nodes represent the medical prior knowledge associated with the target user.

[0018] The multimodal knowledge graph is generated by adding preset virtual nodes corresponding to the medical images to the second knowledge graph, and the preset virtual nodes are connected to the central node.

[0019] Optionally, the target classification model includes a medical image vector representation model, a knowledge graph vector representation model connected to the medical image vector representation model, and a classifier connected to the knowledge graph vector representation model. The step of classifying the medical image using the target classification model based on the multimodal knowledge graph includes:

[0020] Based on the metadata information, a first vector representation of the medical image is determined through the medical image vector representation model. The first vector representation represents the semantic and sequence features of the medical image.

[0021] The second vector representation of each node in the multimodal knowledge graph is determined using the knowledge graph vector representation model.

[0022] The target vector representation of the multimodal knowledge graph is determined based on the first vector representation and the second vector representation;

[0023] The medical images are classified using the classifier based on the target vector representation.

[0024] Optionally, the medical image vector representation model includes a semantic feature extraction model and a sequence feature extraction model connected to the semantic feature extraction model, wherein determining the first vector representation of the medical image based on the metadata information through the medical image vector representation model includes:

[0025] Read the image acquisition sequence information corresponding to the medical image from the metadata information;

[0026] The medical images are divided into multiple sequence images based on the image acquisition sequence information;

[0027] For each of the image sequences, the semantic features of the image sequences are extracted using the semantic feature extraction model;

[0028] Based on the semantic features corresponding to each of the sequence images, the first vector representation of the medical image is determined by the sequence feature extraction model.

[0029] Optionally, the knowledge graph vector representation model includes a GAT (Graph Attention Network), which comprises multiple sequentially connected network layers. Determining the second vector representation of each node in the multimodal knowledge graph using the knowledge graph vector representation model includes:

[0030] For each network layer of the GAT, the correlation coefficient between two nodes in the associated node pair in the multimodal knowledge graph is calculated through the network layer; the associated node pair is two nodes connected by an edge.

[0031] For the third node in each of the associated node pairs, the vector representation of the third node in the network layer is determined based on the correlation coefficient of the associated node pair, the vector representation of the third node obtained in the network layer above the network layer, and the vector representation of the fourth node obtained in the network layer above the network layer. The third node is one node in the associated node pair, and the fourth node is the other node in the associated node pair.

[0032] When the network layer is the last layer of the GAT network, the vector representation of the third node corresponding to the network layer is used as the second vector representation of the third node.

[0033] Optionally, the method further includes:

[0034] If the network layer is not the last layer of the GAT, obtain the correlation coefficients corresponding to each target-related node pair in the multimodal knowledge graph calculated by the network layer, wherein the target-related node pair includes a central node and nodes connected to the central node;

[0035] The correlation coefficient of a specified node pair in the target associated node pair is updated according to the correlation coefficient corresponding to each target associated node pair, wherein one node in the specified node pair is the central node, and the other node in the specified node pair is a preset virtual node connected to the central node;

[0036] The vector representation of each node is calculated through the next network layer based on the updated correlation coefficient, and if the next network layer is the last layer of the GAT network, the vector representation of each node is used as the second vector representation.

[0037] Optionally, updating the correlation coefficient of a specified node pair in the target associated node pair based on the correlation coefficient corresponding to each target associated node pair includes:

[0038] The maximum correlation coefficient is determined from the correlation coefficients corresponding to each of the target associated node pairs;

[0039] The target correlation coefficient is a preset multiple of the maximum correlation coefficient, where the preset multiple is greater than or equal to a preset value.

[0040] Update the correlation coefficient of the specified node pair to the target correlation coefficient.

[0041] Optionally, determining the target vector representation of the multimodal knowledge graph based on the first vector representation and the second vector representation includes:

[0042] Replace the second vector representation corresponding to the preset virtual node with the first vector representation to obtain the updated second vector representation of the preset virtual node;

[0043] The target vector representation is determined based on the preset weights corresponding to the preset virtual nodes and the second vector representation of each node.

[0044] Secondly, this disclosure provides a medical image classification device, the device comprising:

[0045] The first acquisition module is used to acquire metadata information corresponding to the medical images to be classified;

[0046] The second acquisition module is used to acquire medical text information corresponding to the medical image based on the metadata information. The medical text information includes clinical diagnosis and treatment information and medical prior knowledge.

[0047] The classification module is used to classify the medical images based on the medical images and the medical text information using a pre-trained target classification model.

[0048] Thirdly, this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect of this disclosure.

[0049] Fourthly, this disclosure provides an electronic device, comprising:

[0050] A memory on which computer programs are stored;

[0051] A processor for executing the computer program in the memory to implement the steps of the method described in the first aspect of this disclosure.

[0052] The above technical solution can classify medical images based on several different modalities of data, including the user's medical image results, clinical diagnosis and treatment information associated with the medical image, and medical prior knowledge. Since the semantic information contained in different modalities of data is different, this disclosure classifies medical images based on multimodal medical data. Compared with the method of classification based solely on one-dimensional image features of medical images, this method can significantly improve the accuracy of medical image classification results.

[0053] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 This is a flowchart illustrating a medical image classification method according to an exemplary embodiment;

[0056] Figure 2 It is based on Figure 1 The illustrated embodiment presents a flowchart of a medical image classification method;

[0057] Figure 3 It is based on Figure 2 The illustrated embodiment presents a flowchart of a method for constructing a multimodal knowledge graph;

[0058] Figure 4 This is a schematic diagram illustrating a first knowledge graph according to an exemplary embodiment;

[0059] Figure 5 It is based on Figure 4 The illustrated embodiment presents a schematic diagram of a second knowledge graph;

[0060] Figure 6 It is based on Figure 5 The illustrated embodiment presents a schematic diagram of a multimodal knowledge graph;

[0061] Figure 7 It is based on Figure 2 The illustrated embodiment presents a flowchart of a medical image classification method;

[0062] Figure 8 This is a schematic diagram illustrating an architecture for medical image classification based on a multimodal knowledge graph, according to an exemplary embodiment.

[0063] Figure 9 This is a block diagram illustrating a medical image classification device according to an exemplary embodiment;

[0064] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0065] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0066] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0067] This disclosure is mainly applied to the analysis of medical images. The related technology mainly involves fusing the imaging results of different examination techniques (such as X-ray computed tomography, positron emission tomography, magnetic resonance imaging, and functional magnetic resonance imaging), and then classifying the medical images based on the fusion results using a machine learning model. This classification method is based on the image features of medical images, which are then used to perform image recognition and classify the medical images according to the recognition results. However, the image features of medical images can only represent one dimension of the feature information of medical images and cannot represent other dimensions of the feature information of medical images, which will affect the accuracy of medical image classification.

[0068] To address the aforementioned problems, this disclosure provides a medical image classification method, apparatus, storage medium, and electronic device. The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0069] Figure 1This is a flowchart illustrating a medical image classification method according to an exemplary embodiment, such as... Figure 1 As shown, the method includes the following steps:

[0070] In step S11, the metadata information corresponding to the medical images to be classified is obtained.

[0071] The medical image may include images in DICOM (Digital Imaging and Communications in Medicine) format, such as CT images and MRI images. The metadata information records the examination equipment, examination method (generally referring to imaging technology), and basic user information (such as user ID, gender, and age) for the corresponding DICOM image.

[0072] In medical image classification tasks, making full use of the rich information implicit in the corresponding metadata can enhance the image representation capabilities, especially for long-tail problems with indistinct features, thereby improving the accuracy of image recognition.

[0073] In step S12, the medical text information corresponding to the medical image is obtained based on the metadata information. The medical text information includes clinical diagnosis and treatment information and medical prior knowledge.

[0074] The clinical diagnosis and treatment information may include the user's clinical and biological information, such as the user's name, gender, age, symptoms, medical auxiliary examinations, physical examination indicators, and suspected diagnostic results. The medical prior knowledge is typically based on medical knowledge and a large amount of clinical data, summarizing medical diagnosis and treatment standards. This prior knowledge defines the names of various diseases, the symptoms of each disease, the required examinations, the examination procedures for each examination, and the interpretation procedures for examination results, among other medical-related knowledge. This medical prior knowledge can be obtained from a pre-set prior knowledge base, such as the SNOMED (Systematized Nomenclature of Medicine) prior knowledge base, the ICD (International Classification of Diseases) prior knowledge base, or the BMJ (British Medical Journal) prior knowledge base.

[0075] The metadata information includes the user identification information of the target user corresponding to the medical image. In this step, the clinical diagnosis and treatment information of the target user can be determined based on the user identification information; then, the medical prior knowledge associated with the target user can be determined based on the clinical diagnosis and treatment information. For example, medical prior knowledge related to a suspected diagnosis can be associated based on a suspected diagnosis in the clinical diagnosis and treatment information, or medical prior knowledge related to a symptom can be associated based on the symptoms in the clinical diagnosis and treatment information, etc.

[0076] In step S13, the medical image is classified using a pre-trained target classification model based on the medical image and the medical text information.

[0077] The target classification model can be a pre-trained deep learning model.

[0078] Using the above method, medical images can be classified based on several different modalities of data, including the user's medical image results, clinical diagnosis and treatment information associated with the medical image, and medical prior knowledge. Since the semantic information contained in different modalities of data is different, this disclosure classifies medical images based on multimodal medical data. Compared with the method of classifying medical images based solely on one-dimensional image features, this method can significantly improve the accuracy of medical image classification results.

[0079] Figure 2 It is based on Figure 1 The illustrated embodiment presents a flowchart of a medical image classification method, as shown below. Figure 2 As shown, step S13 includes the following sub-steps:

[0080] In step S131, a multimodal knowledge graph is constructed based on the medical image, the clinical diagnosis and treatment information, and the medical prior knowledge. The multimodal knowledge graph represents the image features of the medical image and the text features of the medical text information.

[0081] Figure 3 It is based on Figure 2 The illustrated embodiment presents a flowchart of a method for constructing a multimodal knowledge graph, as shown in the following example. Figure 3 As shown, step S131 includes the following sub-steps:

[0082] In step S1311, a first knowledge graph corresponding to the target user is constructed based on the clinical diagnosis and treatment information. The first knowledge graph includes a central node and multiple first nodes connected to the central node. The central node is the node corresponding to the target user, and the first nodes represent the diagnosis and treatment information of the target user.

[0083] In one possible implementation of this step, the user ID of the target user can be used as the central node, and various diagnostic and treatment information corresponding to the target user (such as gender, age, symptoms, medical auxiliary examinations, physical examination indicators, suspected diagnosis results, etc.) can be used as the first node connected to the central node. Different diagnostic and treatment information corresponds to different first nodes. After pointing from the central node to each first node, the first knowledge graph is constructed.

[0084] For example, Figure 4 This is a schematic diagram illustrating a first knowledge graph according to an exemplary embodiment, such as... Figure 4 As shown, the central node of this first knowledge graph corresponds to "User ID". Starting from this central node, it points to multiple first nodes, which correspond to clinical diagnosis and treatment information such as "female", "36", "drowsiness", "dizziness", "limb weakness", and "185 / 105". Figure 4 The first knowledge graph shown is merely an exemplary representation, and this disclosure does not limit the specific information corresponding to each node in the first knowledge graph.

[0085] In step S1312, a second knowledge graph is constructed based on the first knowledge graph according to the medical prior knowledge. The second knowledge graph includes the central node, a plurality of first nodes connected to the central node, and a plurality of second nodes connected to the first nodes. The second nodes represent the medical prior knowledge associated with the target user.

[0086] In this step, a second node can be connected to the first node of the first knowledge graph, and the first node points to the second node. Figure 5 It is based on Figure 4 The illustrated embodiment presents a schematic diagram of a second knowledge graph. For example, for Figure 4 Taking the first node, "diabetes," as an example, all prior medical knowledge related to "diabetes" can be obtained from the prior knowledge base, and then it can be used as follows: Figure 5 As shown, the second node "Medical Prior Knowledge of Diabetes", which is connected to the first node "Diabetes", records all medical prior knowledge related to "Diabetes".

[0087] In step S1313, the multimodal knowledge graph is generated by adding a preset virtual node corresponding to the medical image to the second knowledge graph, and the preset virtual node is connected to the central node.

[0088] In one possible implementation, the preset virtual node can be used to add nodes corresponding to medical images to the second knowledge graph in the form of (user ID, imaging examination, medical image) triple data. For example... Figure 6 It is based on Figure 5 The illustrated embodiment presents a schematic diagram of a multimodal knowledge graph, as shown below. Figure 6 As shown, the gray-filled nodes are the preset virtual nodes, the central node points to the preset virtual node, and the preset virtual node is the node corresponding to the medical image.

[0089] based on Figure 3 The steps shown can be used to construct a multimodal knowledge graph that includes medical images, clinical diagnosis and treatment information, and medical prior knowledge.

[0090] In step S132, the medical image is classified according to the multimodal knowledge graph and the target classification model.

[0091] The target classification model includes a medical image vector representation model, a knowledge graph vector representation model connected to the medical image vector representation model, and a classifier connected to the knowledge graph vector representation model.

[0092] Figure 7 It is based on Figure 2 The illustrated embodiment presents a flowchart of a medical image classification method, as shown below. Figure 7 As shown, step S132 includes the following sub-steps:

[0093] In step S1321, the first vector representation of the medical image is determined by the medical image vector representation model based on the metadata information. The first vector representation represents the semantic features and sequence features of the medical image.

[0094] The medical image vector representation model includes a semantic feature extraction model and a sequence feature extraction model connected to the semantic feature extraction model. The semantic feature extraction model may include, for example, a CNN (Convolutional Neural Networks) model, and the sequence feature extraction model may include, for example, an LSTM (Long Short-Term Memory) model.

[0095] In this step, the first vector representation of the medical image can be determined in the following way:

[0096] The image acquisition sequence information corresponding to the medical image is read from the metadata information. This image acquisition sequence information may include, for example, any sequence information such as acquisition time, user posture (e.g., lateral, supine, prone), and acquisition position based on the same reference point. Then, the medical image is divided into multiple sequence images according to the image acquisition sequence information. For each sequence image, the semantic features of the sequence image are extracted using the semantic feature extraction model. Based on the semantic features corresponding to each sequence image, the first vector representation of the medical image is determined using the sequence feature extraction model.

[0097] Typically, DICOM format medical images have temporal characteristics. Taking CT images as an example, the final printed CT image is composed of multiple sequence images. Different sequence images may have different acquisition times, different user postures, or different acquisition positions. Therefore, in this step, the medical image can be divided into multiple sequence images based on the acquisition sequence information of any image.

[0098] For example, Figure 8 This is a schematic diagram illustrating an architecture for medical image classification based on a multimodal knowledge graph, according to an exemplary embodiment. Figure 8 As shown, the CT examination result DICOM image can first be divided into three sequence images: image 1, image 2, and image 3, based on the image acquisition sequence information. For each sequence image, the semantic features of each sequence image can be extracted by a CNN convolutional neural network and converted into the corresponding feature vector x. The specific calculation process is shown in formula (1):

[0099]

[0100] Where * represents the convolution operation, i.e., multiplying by elements and then adding them together, X in w represents one of the input image sequences. i is the convolution kernel, and i is the number of channels in the convolution kernel. For example, a convolution kernel with dimensions (3, 3, 3) can be selected. maxPooling is the maximum pooling operation, which takes the maximum value in the feature map corresponding to the filter as the output of the pooling layer. Then, the feature maps of the three output channels can be added bit by bit to obtain the final feature map. Then, the flatten function is used to concatenate and flatten the output feature map matrix along the 0th dimension to obtain the vector representation x of one of the sequence images.

[0101] Then, based on the semantic features corresponding to each image in the sequence, the first vector representation of the medical image can be determined using the sequence feature extraction model. One possible implementation is to arrange the semantic features extracted by the CNN convolutional neural network into a vector sequence {x1, x2, x3} in sequence order, such as... Figure 8 As shown, the vector sequence can be input into an LSTM network to extract its sequence features, and then the first vector representation h of the corresponding CT image with sequence features can be output. c The specific calculation process is shown in formula (2):

[0102] h i =tanh(U·x i +W·h i-1 (2)

[0103] Where U and W are learnable weight parameters, h i Let h0 be the hidden layer vector of the i-th layer of the LSTM network. Assume the LSTM network has 3 hidden layers, i = 1, 2, 3, and h0 is the initialization vector of the hidden layer of the LSTM network; tanh is the non-linear activation function. In this disclosure, the hidden layer vector h3 of the last layer of the LSTM network can be taken as the first vector representation h of the CT image. c The above examples are merely illustrative and are not intended to limit the scope of this disclosure.

[0104] It should be noted that the above example only shows the division of medical images into multiple sequence images based on one image acquisition sequence information. In another possible implementation of this disclosure, if there are multiple such image acquisition sequence information, the vector representation of the medical image can be obtained first for the multiple sequence images obtained by dividing each image acquisition sequence information, and then each vector representation can be fused (such as by weighted summation) to obtain the final first vector representation of the medical image.

[0105] In step S1322, the second vector representation of each node in the multimodal knowledge graph is determined by the knowledge graph vector representation model.

[0106] like Figure 8 As shown, the knowledge graph vector representation model can be, for example, a GAT (Graph Attention Network) network, which can include multiple network layers connected in sequence.

[0107] In this step, the constructed, such as Figure 6 The multimodal knowledge graph shown is input into the GAT network. For each layer of the GAT network, the correlation coefficient between two nodes in a pair of associated nodes in the multimodal knowledge graph is calculated. This pair of associated nodes consists of two nodes connected by an edge, such as... Figure 6As shown, the associated node pair can include (node ​​"User ID" → node "Diabetes"), (node ​​"User ID" → node "Female"), (node ​​"User ID" → node "Sleepiness"), etc. For the third node in each associated node pair, based on the correlation coefficient of the associated node pair, the vector representation of the third node obtained from the network layer above the network layer, and the vector representation of the fourth node obtained from the network layer above the network layer, the vector representation of the third node corresponding to the third node in the network layer is determined. The third node is one node in the associated node pair, and the fourth node is the other node in the associated node pair. Taking the associated node pair (node ​​"User ID" → node "Sleepiness") as an example, the third node can be node "User ID", and the fourth node can be node "Sleepiness". In this way, when the network layer is the last layer of the GAT, the vector representation of the third node in the network layer can be used as the second vector representation of the third node.

[0108] For example, in the process of encoding a multimodal knowledge graph, this disclosure can go through two stages: "original encoding - fine-tuning encoding". In the original encoding stage, the multimodal knowledge graph (the preset virtual nodes in the graph are only simple symbol nodes representing "medical images") can be input into the GAT network. Then, through each network layer of the GAT network, the correlation coefficient between the two nodes in each pair of related nodes in the multimodal knowledge graph is calculated based on formula (3):

[0109]

[0110] Where, α ij Let W represent the correlation coefficient between two nodes (i.e., node i and node j) in a pair of related nodes, W be the learnable parameter matrix, a be the learnable vector, and h be the learnable parameter vector. i N represents the vector of node i in the graph. i Let represent all first-order neighbor nodes of node i. Here, the first-order neighbor matrix refers to the nodes directly connected to node i through edges. || is the concatenation operation, which can concatenate two vectors in the last dimension. LeakyReLU represents the non-linear activation function.

[0111] Then, for the third node in each pair of associated nodes, the vector representation of the third node in each network layer can be calculated using the following formula (4):

[0112]

[0113] Among them, h i ′ represents the vector representation of the third node i calculated by the current network layer; h i h represents the vector representation of the third node i in the layer above the current network layer. jThis represents the vector representation of the fourth node j in the layer above the current network layer, where K represents the number of heads used by the GAT network, and W represents the vector representation of the fourth node j in the current network layer. k Let be the learnable parameter matrix of the k-th head, and ReLU be the non-linear activation function.

[0114] Thus, when the current network layer is determined to be the last layer of GAT, the vector representation h corresponding to the third node i in the current network layer can be represented as follows: i ′ is the second vector representation of the third node i.

[0115] The above examples are merely illustrative and are not intended to limit the scope of this disclosure.

[0116] It should be noted that, for each layer of the GAT network, assuming that the layer is not the last layer of the GAT network, the correlation coefficients corresponding to each target-related node pair in the multimodal knowledge graph calculated by that layer can be obtained. Each target-related node pair includes the central node and the nodes connected to that central node; for example, Figure 6 The "User ID" in the table is the central node. Each first node directly connected to the "User ID" forms a target associated node pair with the central node of that "User ID". Then, the correlation coefficient of a specified node pair in the target associated node pair is updated according to the correlation coefficient corresponding to each target associated node pair. One node in the specified node pair is the central node, and the other node in the specified node pair is a preset virtual node connected to the central node. Figure 6 As shown, the "Medical Imaging" node is the preset virtual node, and the specified node pair is... Figure 6 In the section (“User ID” → “Medical Imaging”), the correlation coefficient of the specified node pair can be updated in the following ways:

[0117] The maximum correlation coefficient is determined from the correlation coefficients corresponding to each target-related node pair. A preset multiple of the maximum correlation coefficient is used as the target correlation coefficient, which is greater than or equal to a preset value. For example, the preset value can be 1 and the preset multiple can be 1.5 times. The correlation coefficient of the specified node pair is then updated to the target correlation coefficient. After that, the correlation coefficients corresponding to each target-related node pair can be normalized. This can significantly improve the contribution rate of image features (i.e., semantic features and sequence features) corresponding to the "medical images" node in the multimodal knowledge graph in the classification task, thereby further improving the classification accuracy.

[0118] Next, the vector representation of each node can be calculated through the next network layer of the network layer based on the updated correlation coefficient (for example, it can be calculated using Equation 4), and if the next network layer is the last network layer of the GAT, the vector representation of each node is used as the second vector representation.

[0119] In step S1323, the target vector representation of the multimodal knowledge graph is determined based on the first vector representation and the second vector representation.

[0120] Based on the method provided in step S1322, the second vector representation of each node in the multimodal knowledge graph can be determined. It should be noted that the preset virtual node representing "medical image" in the multimodal knowledge graph will also obtain a corresponding vector representation. However, since this node is only a symbol node, in this step, the second vector representation corresponding to the preset virtual node can be replaced with the first vector representation of the medical image obtained in step S1321 to obtain the updated second vector representation of the preset virtual node (i.e., fine-tuning stage). After that, the target vector representation can be determined according to the preset weights corresponding to the preset virtual node and the second vector representation of each node.

[0121] For example, the target vector representation of this multimodal knowledge graph can be calculated using the following formula (5):

[0122]

[0123] Among them, h G For the target vector representation of a multimodal knowledge graph, h i h is the updated second vector representation of the image node "medical image". i Let be the second vector representation of the other nodes in the graph, and G be the set of nodes in the graph. β is a hyperparameter, which can be set to, for example, 70%.

[0124] In step S1324, the medical image is classified by the classifier according to the target vector representation.

[0125] The classifier may include, for example, DNN (Deep Neural Networks).

[0126] In this step, such as Figure 8 As shown, the target vector representation of the multimodal knowledge graph (i.e., the global representation of the multimodal knowledge graph) can be input into the DNN deep neural network, and then the classification result of the medical image can be output through the DNN deep neural network.

[0127] In one possible implementation, the DNN deep neural network can be designed as a 10-layer network, and the model can be trained using the cross-entropy loss function. The specific calculation process is shown in Equations (6) and (7):

[0128] h pred =W 9 (…ReLU(W 1 ·ReLU(W′ 0 ·h G (6)

[0129] loss = CrossEntropy(h pred y label (7)

[0130] Among them, h pred Here, y represents the predicted value output by the DNN, ReLU is the non-linear activation function, W is the learnable parameter matrix, loss is the loss function value, and CrossEntropy is the cross-entropy loss function. label These are the training label values.

[0131] Using the above method, medical images can be classified based on several different modalities of data, including the user's medical image results, clinical diagnosis and treatment information associated with the medical image, and medical prior knowledge. Since the semantic information contained in different modalities of data is different, this disclosure classifies medical images based on multimodal medical data. Compared with the method of classifying medical images based solely on one-dimensional image features, this method can significantly improve the accuracy of medical image classification results.

[0132] Figure 9 This is a block diagram illustrating a medical image classification device according to an exemplary embodiment, such as... Figure 9 As shown, the device includes:

[0133] The first acquisition module 901 is used to acquire metadata information corresponding to the medical images to be classified.

[0134] The second acquisition module 902 is used to acquire medical text information corresponding to the medical image based on the metadata information, wherein the medical text information includes clinical diagnosis and treatment information and medical prior knowledge.

[0135] The classification module 903 is used to classify the medical images based on the medical images and the medical text information using a pre-trained target classification model.

[0136] Optionally, the metadata information includes user identification information of the target user corresponding to the medical image. The second acquisition module 902 is used to determine the clinical diagnosis and treatment information of the target user based on the user identification information; and to determine the medical prior knowledge associated with the target user based on the clinical diagnosis and treatment information.

[0137] Optionally, the classification module 903 is used to construct a multimodal knowledge graph based on the medical images, the clinical diagnosis and treatment information, and the medical prior knowledge, wherein the multimodal knowledge graph represents the image features of the medical images and the text features of the medical text information; and classifies the medical images according to the target classification model based on the multimodal knowledge graph.

[0138] Optionally, the classification module 903 is used to construct a first knowledge graph corresponding to the target user based on the clinical diagnosis and treatment information. The first knowledge graph includes a central node and a plurality of first nodes respectively connected to the central node. The central node is a node corresponding to the target user, and the first nodes represent the diagnosis and treatment information of the target user.

[0139] Based on the medical prior knowledge, a second knowledge graph is constructed on the basis of the first knowledge graph. The second knowledge graph includes the central node, a plurality of first nodes connected to the central node, and a plurality of second nodes connected to the first nodes. The second nodes represent the medical prior knowledge associated with the target user.

[0140] The multimodal knowledge graph is generated by adding preset virtual nodes corresponding to the medical images to the second knowledge graph, and the preset virtual nodes are connected to the central node.

[0141] Optionally, the target classification model includes a medical image vector representation model, a knowledge graph vector representation model connected to the medical image vector representation model, and a classifier connected to the knowledge graph vector representation model. The classification module 903 is used to determine a first vector representation of the medical image based on the metadata information through the medical image vector representation model, wherein the first vector representation represents the semantic and sequence features of the medical image; determine a second vector representation of each node in the multimodal knowledge graph through the knowledge graph vector representation model; determine a target vector representation of the multimodal knowledge graph based on the first vector representation and the second vector representation; and classify the medical image based on the target vector representation using the classifier.

[0142] Optionally, the medical image vector representation model includes a semantic feature extraction model and a sequence feature extraction model connected to the semantic feature extraction model. The classification module 903 is used to read the image acquisition sequence information corresponding to the medical image from the metadata information; divide the medical image into multiple sequence images according to the image acquisition sequence information; extract the semantic features of the sequence image through the semantic feature extraction model for each sequence image; and determine the first vector representation of the medical image through the sequence feature extraction model based on the semantic features corresponding to each sequence image.

[0143] Optionally, the knowledge graph vector representation model includes a graph attention network (GAT), which comprises multiple sequentially connected network layers. The classification module 903 is used to calculate the correlation coefficient between two nodes in a pair of associated nodes in the multimodal knowledge graph for each network layer of the GAT. The pair of associated nodes is two nodes connected by an edge. For the third node in each pair of associated nodes, the vector representation of the third node in the network layer is determined based on the correlation coefficient of the pair, the vector representation of the third node obtained in the network layer above the network layer, and the vector representation of the fourth node obtained in the network layer above the network layer. The third node is one node in the pair of associated nodes, and the fourth node is the other node in the pair of associated nodes. When the network layer is the last layer of the GAT, the vector representation of the third node in the network layer is used as the second vector representation of the third node.

[0144] Optionally, the classification module 903 is configured to, when the network layer is not the last layer of the GAT, obtain the correlation coefficients corresponding to each target-related node pair in the multimodal knowledge graph calculated by the network layer, wherein the target-related node pair includes a central node and a node connected to the central node; update the correlation coefficients of a specified node pair in the target-related node pair according to the correlation coefficients corresponding to each target-related node pair, wherein one node in the specified node pair is the central node, and the other node in the specified node pair is a preset virtual node connected to the central node; calculate the vector representation of each node through the next network layer of the network layer according to the updated correlation coefficients, and, when the next network layer is the last layer of the GAT, use the vector representation of each node as the second vector representation.

[0145] Optionally, the classification module 903 is configured to determine the maximum correlation coefficient from the correlation coefficients corresponding to each of the target associated node pairs; use a preset multiple of the maximum correlation coefficient as the target correlation coefficient, wherein the preset multiple is greater than or equal to a preset value; and update the correlation coefficient of the specified node pair to the target correlation coefficient.

[0146] Optionally, the classification module 903 is used to replace the second vector representation corresponding to the preset virtual node with the first vector representation to obtain the updated second vector representation of the preset virtual node; and to determine the target vector representation based on the preset weights corresponding to the preset virtual node and the second vector representation of each node.

[0147] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0148] Figure 10 This is a block diagram illustrating an electronic device 1000 according to an exemplary embodiment. For example... Figure 10 As shown, the electronic device 1000 may include: a processor 1001 and a memory 1002. The electronic device 1000 may also include one or more of a multimedia component 1003, an input / output (I / O) interface 1004, and a communication component 1005.

[0149] The processor 1001 controls the overall operation of the electronic device 1000 to complete all or part of the steps in the aforementioned medical image classification method. The memory 1002 stores various types of data to support the operation of the electronic device 1000. This data may include, for example, instructions for any application or method operating on the electronic device 1000, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 1003 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 1002 or transmitted via communication component 1005. The audio component also includes at least one speaker for outputting audio signals. I / O interface 1004 provides an interface between processor 1001 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 1005 is used for wired or wireless communication between the electronic device 1000 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 1005 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0150] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the medical image classification method described above.

[0151] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the medical image classification method described above. For example, the computer-readable storage medium may be the memory 1002 including the program instructions described above, which may be executed by the processor 1001 of the electronic device 1000 to complete the medical image classification method described above.

[0152] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described medical image classification method when executed by the programmable device.

[0153] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0154] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0155] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A medical image classification method, characterized by, The method comprises: acquiring metadata information corresponding to a medical image to be classified; acquiring medical text information corresponding to the medical image according to the metadata information, the medical text information including clinical diagnosis and treatment information and medical prior knowledge; constructing a first knowledge graph corresponding to a target user according to the clinical diagnosis and treatment information, the first knowledge graph including a center node and a plurality of first nodes connected to the center node, the center node being a node corresponding to the target user, and the first nodes representing diagnosis and treatment information of the target user; constructing a second knowledge graph on the basis of the first knowledge graph according to the medical prior knowledge, the second knowledge graph including the center node, the plurality of first nodes connected to the center node, and a plurality of second nodes connected to the first nodes, the second nodes representing the medical prior knowledge associated with the target user; generating a multi-modal knowledge graph by adding a preset virtual node corresponding to the medical image to the second knowledge graph, the preset virtual node being connected to the center node, the multi-modal knowledge graph representing image features of the medical image and text features of the medical text information; determining a first vector representation of the medical image according to the metadata information by a medical image vector representation model in a target classification model pre-trained, the first vector representation representing semantic features and sequence features of the medical image, wherein the target classification model includes the medical image vector representation model, a knowledge graph vector representation model connected to the medical image vector representation model, and a classifier connected to the knowledge graph vector representation model; determining a second vector representation of each node in the multi-modal knowledge graph by the knowledge graph vector representation model; determining a target vector representation of the multi-modal knowledge graph according to the first vector representation and the second vector representation; classifying the medical image according to the target vector representation by the classifier.

2. The method of claim 1, wherein, The metadata information includes user identification information of a target user corresponding to the medical image, and the medical text information corresponding to the medical image is acquired according to the metadata information, comprising: determining the clinical diagnosis and treatment information of the target user according to the user identification information; determining the medical prior knowledge associated with the target user according to the clinical diagnosis and treatment information.

3. The method of claim 1, wherein, The medical image vector representation model includes a semantic feature extraction model and a sequence feature extraction model connected to the semantic feature extraction model, and the first vector representation of the medical image is determined according to the metadata information by the medical image vector representation model in the target classification model pre-trained, comprising: reading image acquisition sequence information corresponding to the medical image from the metadata information; dividing the medical image into a plurality of sequence pictures according to the image acquisition sequence information; extracting semantic features of each sequence picture by the semantic feature extraction model; According to the semantic features corresponding to each of the sequence pictures, the first vector representation of the medical image is determined by the sequence feature extraction model.

4. The method of claim 1, wherein, The knowledge graph vector representation model comprises a graph attention network (GAT), and the GAT comprises a plurality of network layers connected in sequence. The determination of the second vector representation of each node in the multi-modal knowledge graph by the knowledge graph vector representation model comprises: For each network layer of the GAT, the correlation coefficient of two nodes in an associated node pair in the multi-modal knowledge graph is calculated by the network layer; the associated node pair is two nodes connected by an edge; For a third node in each associated node pair, the vector representation of the third node in the network layer is determined according to the correlation coefficient of the associated node pair, the vector representation of the third node obtained by a previous network layer of the network layer, and the vector representation of a fourth node obtained by the previous network layer, the third node being one node in the associated node pair, and the fourth node being another node in the associated node pair; In the case where the network layer is the last network layer of the GAT, the vector representation of the third node in the network layer is taken as the second vector representation of the third node.

5. The method of claim 4, wherein, The method further comprises: In the case where the network layer is not the last network layer of the GAT, the correlation coefficient corresponding to each target associated node pair in the multi-modal knowledge graph calculated by the network layer is obtained, the target associated node pair comprising a center node and a node connected to the center node; The correlation coefficient of a specified node pair in the target associated node pair is updated according to the correlation coefficient corresponding to each target associated node pair, one node in the specified node pair being the center node, and another node in the specified node pair being a preset virtual node connected to the center node; The vector representation of each node is calculated by a next network layer of the network layer according to the updated correlation coefficient, and in the case where the next network layer is the last network layer of the GAT, the vector representation of each node is taken as the second vector representation.

6. The method of claim 5, wherein, The updating of the correlation coefficient of the specified node pair in the target associated node pair according to the correlation coefficient corresponding to each target associated node pair comprises: A maximum correlation coefficient is determined from the correlation coefficient corresponding to each target associated node pair; A preset multiple of the maximum correlation coefficient is taken as a target correlation coefficient, the preset multiple being greater than or equal to a preset value; The correlation coefficient of the specified node pair is updated to the target correlation coefficient.

7. The method of claim 1, wherein, The determination of the target vector representation of the multi-modal knowledge graph according to the first vector representation and the second vector representation comprises: The second vector representation corresponding to the preset virtual node is replaced by the first vector representation to obtain an updated second vector representation of the preset virtual node; The target vector representation is determined according to a preset weight corresponding to the preset virtual node and the second vector representation of each node.

8. A medical image classification apparatus characterized by comprising: The device comprises: The first acquisition module is configured to acquire metadata information corresponding to a medical image to be classified. The second acquisition module is configured to acquire medical text information corresponding to the medical image according to the metadata information, wherein the medical text information comprises clinical diagnosis and treatment information and medical prior knowledge. The classification module is configured to construct a first knowledge graph corresponding to a target user according to the clinical diagnosis and treatment information, wherein the first knowledge graph comprises a center node and a plurality of first nodes connected to the center node, the center node is a node corresponding to the target user, and the first nodes represent diagnosis and treatment information of the target user. A second knowledge graph is constructed on the basis of the first knowledge graph according to the medical prior knowledge, wherein the second knowledge graph comprises the center node, the plurality of first nodes connected to the center node, and a plurality of second nodes connected to the first nodes, and the second nodes represent the medical prior knowledge associated with the target user. A multi-modal knowledge graph is generated by adding a preset virtual node corresponding to the medical image to the second knowledge graph, wherein the preset virtual node is connected to the center node, and the multi-modal knowledge graph represents image features of the medical image and text features of the medical text information. A first vector representation of the medical image is determined by a medical image vector representation model in a target classification model according to the metadata information, wherein the first vector representation represents semantic features and sequence features of the medical image, and the target classification model comprises the medical image vector representation model, a knowledge graph vector representation model connected to the medical image vector representation model, and a classifier connected to the knowledge graph vector representation model. A second vector representation of each node in the multi-modal knowledge graph is determined by the knowledge graph vector representation model. A target vector representation of the multi-modal knowledge graph is determined according to the first vector representation and the second vector representation. The medical image is classified by the classifier according to the target vector representation.

9. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-7.

10. An electronic device, comprising: The program is executed by the processor to implement the steps of the method of any one of claims 1-7. The program is executed by the processor to implement the steps of the method of any one of claims 1-7. ​

Images