A method of medical image classification of a graph convolution network

Abstract

The application provides a medical image classification method based on graph convolution network. The traditional neural network and the graph convolution network are combined: the classic neural network is used to extract image features; each feature map is regarded as a node, an adjacency matrix is constructed by using the similarity between the feature maps, and a graph representation is obtained; then, the multi-scale graph attention convolution network is used to update the node information, learn the representation of the graph, and predict the disease classification of the medical image. Compared with the traditional convolutional neural network, the application has low parameter quantity and low calculation amount, but high classification accuracy. The application selects a graph structure method to model the structural continuity and interaction between different feature maps, the features of the graph structure are diverse, and a multi-scale graph attention network is invented to learn more complete high-level features of the graph, so that the expression ability of the network is improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a method for medical image classification using graph convolutional networks, which can be used for disease classification of medical images. Background Technology

[0002] Deep learning, belonging to the fields of artificial intelligence and machine learning, solves real-world problems by constructing neural network structures, extracting features, and making regression decisions. Due to its powerful performance, deep learning is increasingly popular in applications such as lesion segmentation, biomarker segmentation, disease diagnosis, and image synthesis. However, existing deep learning networks are mainly convolutional neural networks (CNNs), which have the following main problems: 1. As performance increases, the width, depth, and complexity of CNNs also increase, leading to a rapid increase in the number of network parameters and computational load, which is detrimental to device performance and real-time processing; 2. CNNs often have requirements for the size or resolution of input images, resulting in poor transferability of medical data generated by different instruments and equipment. Therefore, this invention proposes a method for medical image classification using graph convolutional networks. It utilizes a classic CNN structure as the backbone network, then uses the feature layer output of the backbone network as the basis for building graph convolutional data. After obtaining the graph data, a multi-scale graph attention structure is used for feature transfer and aggregation, ultimately yielding the task result output.

[0003] Graphs exhibit diverse topological structures, and medical image classification methods based on graph convolutional networks can learn the features of each feature map and model the structural continuity and interactions between different feature maps. Therefore, utilizing graph neural networks is significant for learning multi-class classification of fundus images. Summary of the Invention

[0004] To address the aforementioned problems, this paper presents a medical image classification method using graph convolutional networks. This method utilizes traditional neural networks to extract features, converts the extracted feature maps into a graph data structure, updates node information in the graph convolutional network, and learns the graph representation. Compared to traditional convolutional neural networks, it has lower parameter and computational costs while achieving higher classification accuracy. This invention chooses a graph structure approach to model the structural continuity and interactions between different feature maps, thus resolving the problems existing in the prior art.

[0005] This invention combines traditional neural networks and graph convolutional networks for the field of medical imaging. It utilizes the similarity between feature maps to construct an adjacency matrix, transmits graph data to a multi-scale feature fusion attention network structure for training, and predicts disease classification in medical images. The features of graph structures are diverse, and the multi-scale network structure facilitates learning more complete high-level graph features, improving the network's expressive power.

[0006] The technical solution adopted in this invention is a medical image classification method using graph convolutional networks, comprising the following steps:

[0007] S1, using a medical classification dataset to train a classic deep learning network to obtain a pre-trained model;

[0008] S2, extract medical image feature maps through a pre-trained model. After obtaining the feature map corresponding to each image, treat each feature map as a node and construct an adjacency matrix by calculating the similarity between each feature map.

[0009] S3, Load the graph structure data corresponding to the preprocessed medical image;

[0010] S4 utilizes a multi-scale attention graph convolutional neural network to transfer and aggregate features from graph data in medical images. By using multi-scale feature fusion, it increases the diversity of node features, learns graph representations, and finally outputs classification results.

[0011] Furthermore, in step S1, a classification model is trained using classic convolutional neural networks such as ResNet and VGG as the backbone network of the CNN, using medical images and their corresponding labels, and the trained model is saved.

[0012] Furthermore, step S2 specifically includes:

[0013] S21. Use the trained CNN model to extract feature maps corresponding to medical images. When constructing the graph structure, the topology of the graph is constructed based on the feature outputs of the intermediate layers of the backbone network, with dimensions of N×H×W, where N is the number of layers, H is the feature map height, and W is the feature map width.

[0014] S22, treat each feature map (H×W) as a node. Connect each two-dimensional feature map end-to-end in rows to form a one-dimensional node feature vector. Then, combine the node feature vectors formed by the N feature maps to form an N-row two-dimensional graph feature matrix X = [x0… ... N-1 ]∈R N×(H×W) A central node was added during graph construction, and its features were initialized with the value zero. The features of the i-th node are shown in equation (1-1):

[0015] x i =flatten(map) i )∈R 1×(H×W) , i∈{0,…,N-1} (1-1)

[0016] Here, `flatten(·)` represents the flattening operation, flattening the two-dimensional feature map into a one-dimensional one; `map`... i This represents the (i+1)th feature map.

[0017] S23, when constructing the adjacency matrix A of the graph, the central node is connected to all nodes. This helps the graph neural network process and integrate information from different feature maps during graph classification, and ensures that the central node considers information from all other nodes when updating vertex features. For the adjacency information between other nodes, the similarity matrix S∈R of all nodes is obtained by calculating the difference in the L2 norm between every two feature maps. N×N As shown in equations (1-2) and (1-3):

[0018]

[0019] S i，j =|L2 i -L2 j |,i,j∈{0,…,N-1} (1-3)

[0020] L2 i Let f represent the L2 norm of the i-th feature map. a S represents each value in the feature map; i，j S represents the similarity between the i-th and j-th feature maps. i，j =S j，i ;|·| indicates taking the absolute value.

[0021] Set the threshold to the average difference between the maximum and minimum values ​​in the similarity matrix S, and transform the similarity matrix into an adjacency matrix A. The formula for the threshold is shown in equation (1-4):

[0022] av = (S max -S min ) / 2 (1-4)

[0023] Furthermore, step S3 specifically includes:

[0024] Each medical image is represented as a graph G = (X, A), and the graph structure information consists of the graph's feature matrix and adjacency matrix. The graph structure data corresponding to each medical image is loaded and preprocessed.

[0025] Furthermore, step S4 is specifically performed according to the following steps:

[0026] S41, In the feature extraction stage, this invention employs a multi-channel mechanism to generate multi-scale features. Features are extracted using a Gaussian Atlas (GAT) in each channel. A set of node features input to the GAT is represented as H = {h1, h2, ..., h...} N}∈R N×d h i ∈R 1×dGAT aggregates neighborhood information and updates node features by assigning different weights to neighbors. The new node feature generated by the i-th channel is represented as follows: Where k i This represents the feature scale extracted from the i-th channel. The features output by GAT are shown in equation (1-5):

[0027]

[0028] S42, to reduce model parameters and improve model speed and accuracy, this invention adds a 2D convolution with a kernel size of 1 after the GAT in each feature extraction layer. BatchNorm is added for normalization, and the ReLU activation function is combined with BN to enhance the non-linear characteristics of the entire network and prevent gradient explosion or decay. The features after convolution are shown in equation (1-6):

[0029]

[0030] This invention adds 2D convolutions with a kernel size of 3 to the last two channels to increase the receptive field, capture more useful information, and enhance the network's expressive power. The final output features of the last two channels are shown in equation (1-7):

[0031]

[0032] S43, In the feature aggregation stage, this invention first fuses the generated multi-scale features using a concat method, so that each node obtains features from different scales. The concatenated features are shown in equation (1-8):

[0033]

[0034] Finally, the multi-scale features are passed to the information aggregator for learning, increasing feature diversity. This invention does not specify an information aggregation method; the final output features are shown in equations (1-9).

[0035] H″=aggregate(reshape(C))∈R N×d′ (1-9)

[0036] The reshape operation converts the features into a data type that the GNN model can accept; d′ is the number of node features in the final output of the GNN structure.

[0037] S44: Based on the final graph representation output by S43, perform prediction and classification, and output the classification result. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a structural diagram of a graph convolutional network used for classifying medical images according to an embodiment of the present invention.

[0040] Figure 2 This is a flowchart illustrating how medical images are classified based on graph convolutional networks according to an embodiment of the present invention.

[0041] Figure 3 It is a diagram of a multi-scale graph attention network structure. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] This embodiment is based on an ophthalmic medical dataset. Fundus examination is a diagnostic procedure used to examine the biological structures and abnormalities of the eye. Glaucoma, diabetic retinopathy, cataracts, and other eye diseases are leading causes of vision impairment worldwide. Ocular Disease Intelligent Recognition (ODIR) is a benchmark structured fundus image dataset used by researchers for multi-label, multi-disease classification of fundus images. Currently, using fundus images for early screening of eye diseases has significant clinical implications.

[0044] This invention provides a method for medical image classification based on graph convolutional networks, such as... Figure 1 As shown, a CNN network structure is first trained using fundus images, and the outputs of the intermediate layers of the CNN are used to construct graph data corresponding to each image. This example selects the output features of the sixth layer of the CNN to construct the graph data, and the L2 norm is chosen when constructing the adjacency matrix to calculate the similarity between feature maps; however, the invention is not limited to this. In the information aggregation stage, any graph convolutional layer can be used instead. This example selects a multilayer perceptron (MLP) for information aggregation. MLPs share weights among all nodes, enabling pre-classification. However, the invention is not limited to this.

[0045] Assumptions: The output of the last layer of the CNN is [512, 1, 1], and the output of the sixth layer is [512, 28, 28]. The dataset is ODIR. The GNN network structure extracts multi-scale features of three scales: 512×2, 512×10, and 512×30 during the feature extraction stage. In the information aggregation stage, MLP is used to extract features, and the aggregated feature scale is N×8 (class = 8).

[0046] This invention provides a method for medical image classification using graph convolutional networks, such as... Figure 1 As shown, the specific steps are as follows:

[0047] S1, use the ODIR dataset to train a CNN network and save the trained model.

[0048] S21, Prepare graph structure data. Use a trained CNN network to construct the graph structure information corresponding to all fundus images. Select the output of the sixth layer of the CNN, with an output dimension of [512, 28, 28], resulting in 512 feature maps. Treat each feature map as a node, and flatten each feature map to obtain the features corresponding to a node. Concatenate the features of all nodes to obtain the feature matrix X, with the features of the central node all set to zero.

[0049] x i =flatten(map) i )∈R 1×(28×28) , i∈{0,…,511} (1)

[0050] S22: Construct an adjacency matrix using the output features of the sixth layer of the CNN. Similar to step S21, treat each feature map as a node. First, calculate the L2 norm of each feature map, then construct a similarity matrix S∈R using the absolute value of the difference in norms between any two feature maps. 512×512 Finally, using the set threshold, the similarity matrix is ​​converted into an adjacency matrix, and a center node is added that is connected to all nodes.

[0051]

[0052] S i，j =|L2 i -L2 j |,i,j∈{0,…,511} (3)

[0053] S41 uses a multi-scale feature fusion network structure to process the constructed graph structure data, thereby better learning the graph representation. In this example, during the feature extraction stage, GAT is used to generate features at three scales: 512×2, 512×10, and 512×30 in the three channels, respectively.

[0054]

[0055] S42 adds a 2D convolution with kernel 1 to each channel to reduce computation, and adds a 2D convolution with kernel 3 to the last two channels to increase the receptive field.

[0056]

[0057]

[0058] In step S43, the multi-scale features generated in step S42 are fused using concat. Then, an MLP is used for information aggregation to learn the graph representation; in this example, the MLP output features are 512×8. Finally, the learned graph features are used for prediction and classification.

[0059]

[0060] H″=MLP(reshape(C))∈R 512×8 (8)

[0061] This example was trained on the Ocular Disease Intelligent Recognition (ODIR) dataset, which contains the ages of 5000 patients, color fundus photographs of their left and right eyes, and physician diagnostic keywords. This dataset is a collection of "real-life" patient information gathered by Shanggong Medical Technology Co., Ltd. from various hospitals / medical centers. In these institutions, fundus images were captured by various cameras available on the market, resulting in different image resolutions. Annotations were added by trained human readers under quality control. Patients were categorized into eight labels: Normal (N), Diabetes (D), Glaucoma (G), Cataract (C), Age-related Macular Degeneration (A), Hypertension (H), Pathological Myopia (M), and Other Diseases / Abnormalities (O).

[0062] This invention utilizes graph convolutional networks for fundus image classification. First, it constructs graph structure information using feature maps output from the intermediate layers of a pre-trained traditional neural network. Then, it trains the graph neural network using the graph data corresponding to each fundus image. This invention can be applied to the classification of other medical images, such as two-dimensional images like lung CT scans, breast ultrasounds, and abdominal CT scans; it can also be used with common classification datasets, such as the CIFAR-10 and COCO datasets.

[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A medical image classification method using graph convolutional networks, characterized in that, Includes the following steps: S1, using a medical classification dataset to train a classic deep learning network to obtain a pre-trained model; S2, extract medical image feature maps through a pre-trained model. After obtaining the feature map corresponding to each image, treat each feature map as a node and construct an adjacency matrix by calculating the similarity between each feature map. S3, Load the graph structure data corresponding to the preprocessed medical image; S4 utilizes a multi-scale attention map convolutional neural network to transfer and aggregate features of graph data in medical images. It increases the diversity of node features through multi-scale feature fusion, learns graph representation, and finally outputs classification results. In step S2, specifically: S21. Use the trained CNN model to extract the feature map corresponding to the medical image. When constructing the graph structure, construct the graph topology based on the feature output of the intermediate layer of the backbone network, with a size of N×H×W, where N is the number of layers, H is the feature map height, and W is the feature map width. S22, treat each feature map (H×W) as a node. Connect each two-dimensional feature map end-to-end in rows to form a one-dimensional node feature vector. Then, combine the node feature vectors formed by the N feature maps to form an N-row two-dimensional graph feature matrix X = [x0… ... N-1 ]∈R N×(H×W) A central node was added during graph construction, and its features were initialized with the value zero. The features of the i-th node are shown in equation (1-1). x i =flatten(map i )∈R 1×(H×W) ,i∈{0,…,N-1} (1-1) Here, `flatten(·)` represents the flattening operation, flattening the two-dimensional feature map into a one-dimensional one; `map`... i This represents the (i+1)th feature map; S23, when constructing the adjacency matrix A of the graph, the central node is connected to all nodes. This helps the graph neural network process and integrate information from different feature maps during graph classification, and ensures that the central node considers information from all other nodes when updating vertex features. For the adjacency information between other nodes, the similarity matrix S∈R of all nodes is obtained by calculating the difference of the L2 norm between every two feature maps. N×N As shown in equations (1-2) and (1-3): S i,j =|L2 i -L2 j |,i,j∈{0,…,N-1} (1-3) L2 i Let f represent the L2 norm of the i-th feature map. a S represents each value in the feature map; i，j S represents the similarity between the i-th and j-th feature maps. i，j =S j,i ;|·| indicates taking the absolute value; Set the threshold to the average of the differences between the maximum and minimum values ​​in the similarity matrix S, and convert the similarity matrix into an adjacency matrix A; the formula for the threshold is shown in equation (1-4): off=(S max -S min ) / 2(1-4) Step S4 is specifically performed according to the following steps: S41, In the feature extraction stage, a multi-channel mechanism is used to generate multi-scale features; features are extracted using GAT in each channel; a set of node features input to GAT is represented as H = {h1, h2, ..., h...} N }∈R N×d h i ∈R 1×d GAT aggregates neighborhood information and updates node features by assigning different weights to neighbors. The new node feature generated by the i-th channel is represented as follows: Where ki represents the feature scale extracted from the i-th channel; the features output by GAT are shown in equation (1-5): S42, to reduce model parameters and improve model speed and accuracy, a two-dimensional convolution with a kernel size of 1 is added after the GAT of each feature extraction layer; and BatchNorm is added for normalization, combining the ReLU activation function with BN to enhance the nonlinear characteristics of the entire network and prevent gradient explosion or decay; the features after convolution are shown in Equation (1-6): Two-dimensional convolutions with a kernel size of 3 were added to the last two channels, and the final output features of the last two channels are shown in Equation (1-7): S43, In the feature aggregation stage, the generated multi-scale features are first fused using the concat method, so that each node obtains features from different scales; the concatenated features are shown in Equation (1-8): Finally, the multi-scale features are passed to the information aggregator for learning to increase feature diversity; without specifying an information aggregation method, the final output features are shown in equation (1-9): H″=aggregate(reshape(C))∈R N×d′ (1-9) The reshape operation converts the features into a data type that the GNN model can accept; d′ is the number of node features in the final output of the GNN structure. S44: Based on the final graph representation output by S43, perform prediction and classification, and output the classification result.

Images