A cross-modal medical image domain adaptation classification method based on graph neural network

Abstract

The present invention discloses a cross-modal medical image domain adaptive classification method based on graph neural network. Since cross-modal medical images show large intra-class differences and small inter-class differences, which affects the collection and labeling of cross-modal medical image data, the graph neural network module of the present invention can well distinguish medical images in different fields, which can significantly reduce model oscillation and improve classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image classification, and in particular to a cross-modal medical image domain adaptive classification method based on graph neural network. Background Art

[0002] Cancer remains one of the leading causes of death in humans, and timely and accurate grading of the disease plays a vital role in subsequent treatment. Although medical imaging technology has made rapid progress in recent years, computer-aided diagnosis (CAD) systems usually rely on training machine learning models (such as deep convolutional neural networks) and a large number of correctly labeled samples, and the lack of large training data sets poses a major challenge to computer-aided diagnosis (CAD) systems. First, in order to collect data, experienced medical imaging experts are required to carefully examine hundreds to thousands of gigabytes of images. This annotation process is very time-consuming and labor-intensive. Secondly, medical image annotation is difficult. For the same type of medical imaging data, there are still large differences between medical images collected by different equipment and imaging conditions. The deep learning model trained on a certain dataset cannot be effectively migrated and reused.

[0003] In recent years, a variety of domain adaptation methods have been developed to address the above challenges in medical image analysis. For example, Lafarge et al. first proposed a system solution based on domain adversarial neural networks (DANN) (Maxime W Lafarge, Josien PW Pluim, Koen AJ Eppenhof, Pim Moeskops, and Mitko Veta, "Domain adversarial neural networks to address the appearance variability of histopathology images," in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 83–91. Springer, 2017) to address the differences in mitosis detection in breast cancer histopathology images. Subsequently, Wollmann et al. used cycle-consistent generative adversarial networks (CycleGAN) to develop a domain adaptation method (Thomas Wollmann, CS Eijkman, and Karl Rohr, "Adversarial domain adaptation to improve automatic breast cancer grading in lymph nodes," in IEEE International Symposium on Biomedical Imaging. IEEE, 2018, pp. 582–585) to classify full-slice images and grade the disease level of breast cancer patients.Recently, Zhang et al. proposed a deep unsupervised domain adaptation method (Yifan Zhang, Hanbo Chen, Ying Wei, Peilin Zhao, Jiezhang Cao, Xinjuan Fan, Xiaoying Lou, Hailing Liu, Jinlong Hou, Xiao Han, et al., "From whole slide imaging to microscopy: Deep microscopy adaptation network for histopathology cancer image classification," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2019, pp. 360–368) to adapt the deep model trained on the labeled whole slide image domain to the unlabeled microscopy image domain.

[0004] The above introduction to existing methods shows that the domain adaptation methods currently designed for medical image analysis are mainly based on the framework of adversarial learning, which is inefficient and often leads to model oscillation and difficulty in convergence. Therefore, there is a need for an image recognition method that is effective for images in different domains, has good performance and minimizes model oscillation. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a cross-modal medical image domain adaptive classification method based on graph neural network.

[0006] To achieve the above object, the present invention provides the following technical solution: a cross-modal medical image domain adaptive classification method based on graph neural network, comprising the following steps:

[0007] S1 obtains a cross-modal medical imaging dataset, and classifies the data of the cross-modal medical imaging dataset, selects image datasets of a source domain and a target domain from the classified cross-modal medical imaging dataset, and labels the image data in the source domain and the target domain;

[0008] S2 preprocesses the image datasets of the source domain and the target domain;

[0009] S3 uses a deep neural network to build a model backbone network, inputs the image data of the source domain and the target domain preprocessed in step S2 into the model backbone network, extracts the features of the image data, and outputs the feature vectors of the source domain and the target domain;

[0010] S4 calculates the maximum average difference between the source domain and the target domain feature vectors and the loss function of the contrast feature constraint;

[0011] S5 passes the feature vectors of the source domain and the target domain as nodes into the high-order graph neural network, generates an output vector that integrates the information of the two domains in the high-order graph neural network, realizes the transfer of the label information of the source domain to the target domain, and then passes the output vector into the softmax layer to generate predicted labels for the image data in the source domain.

[0012] S6 calculates the total loss function, repeats steps S3 to S5, and trains the constructed model by minimizing the total loss function until the model converges;

[0013] S7 inputs the target domain image data preprocessed in step S2 into the convergence model obtained in step S6, and the model outputs the predicted predicted label of the category to which the target domain image data belongs.

[0014] Furthermore, in step S1, a cross-modal medical image dataset is obtained, and a labeled cross-modal medical image dataset is determined as an image dataset of the source domain. The cross-modal medical image dataset to be classified is used as the image dataset of the target domain

[0015] in, is the i-th image data in the source domain image dataset, N s is the number of image data in the source domain image dataset; is the jth image data in the target domain image dataset, N t is the number of image data in the target domain image dataset, X s With X t They are the collections of source domain and target domain image data respectively.

[0016] Furthermore, in step S1, the cross-modal medical image dataset includes pathological images of different tissues under different staining conditions, skin disease images collected by different dermatoscope devices, and diabetic retinopathy images collected by different fundus cameras.

[0017] Furthermore, in step S2, the preprocessing is to crop the images in the image datasets of the selected source domain and the target domain, the cropping size is n×n, and determine the category set C into which the image data in the image dataset is to be divided, wherein the category set C includes two categories: positive samples and negative samples.

[0018] Furthermore, in step S3, the specific steps of obtaining the feature vector are:

[0019] a. Use a pre-trained deep neural network as the backbone network of the model;

[0020] b. Randomly select the source domain and target domain image data preprocessed in step S2 and input them into the backbone network;

[0021] c. In the backbone network, set the threshold T for feature space mapping and generate the corresponding feature vector according to the threshold T.

[0022] Furthermore, step S4 specifically includes the following steps:

[0023] a. Calculate the maximum average difference loss function and minimize the maximum average difference loss function so that similar features are extracted from the image data in the source domain and the target domain:

[0024]

[0025] Among them, L mmd represents the maximum average discrepancy loss function; Represents the i-th image data in the source domain image dataset; is the jth image data in the target domain image dataset; φ(x) represents the feature vector of x generated by the nonlinear mapping in the deep neural network in step S3; H k represents the reproducing kernel Hilbert space associated with the nonlinear mapping φ(·) of the deep neural network in step S3;

[0026] b. Calculate the contrast feature constraint loss function and minimize the contrast feature constraint loss function to reduce the distance between image data of the same category in the source domain and the target domain, and the distance between image data of different categories is greater than d:

[0027]

[0028] Among them, L g is the loss function for contrast feature constraints; d is the predefined boundary value, l i With l j represent the true categories of the i-th and j-th image data in the source domain and target domain image datasets, respectively, and φ(·) is the nonlinear mapping generated by the deep neural network in step S3.

[0029] Furthermore, in step S5, a graph neural network is used to bring similar image data in the source domain and the target domain closer together, thereby transferring the label information of the source domain image data to the image data in the target domain.

[0030] Furthermore, step S6 specifically includes the following steps:

[0031] a. Calculate the total loss function: L = L mmd +L g +L c

[0032] Among them, L mmd Maximum average discrepancy loss function; L g To compare the loss function of feature constraints, is the cross entropy loss between the predicted label and the true label of the image data in the source domain output by the model, where y is the true label, is the predicted label output by the model described in step S5, The labels representing the output of the model are expressed as probabilities The information entropy when predicted as a certain class, The labels representing the output of the model are expressed as probabilities Information entropy when predicted as another class;

[0033] b. Update the model parameters by minimizing the total loss function until the total loss function of the model converges.

[0034] Furthermore, when the category prediction vector output by the softmax layer of the convergence model is greater than the vector of the set threshold ∈, the class corresponding to the category prediction vector is the prediction label of the category to which the target domain image data belongs.

[0035] Compared with the prior art, the present invention has at least the following beneficial effects:

[0036] The present invention proposes a cross-modal medical image domain adaptive classification method based on graph neural network. Since cross-modal medical images show large intra-class differences and small inter-class differences, which affects the collection and labeling of cross-modal medical image data, the present invention uses a deep neural network as the model backbone network, combines the graph neural network module and uses the softmax layer to classify and predict the image data of the target domain, which can well distinguish medical images in different fields, greatly reduce model oscillation, and improve the classification accuracy of cross-modal medical images.

[0037] The classification method of the present invention has a short training time and a fast model convergence speed. Currently, the training time for hundreds of epochs of deep learning models is generally one day or even several days, while this model only needs a few hours.

[0038] Compared with previous adversarial algorithms, the classification method provided by the present invention has good adaptability to different medical imaging data sets and is more stable.

[0039] The loss function designed by the present invention not only aligns the feature distribution differences of the image data sets in two fields, but also takes into account the category differences, so that better classification effects are achieved in more difficult classification scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1It is a flow chart of the medical image classification method shown in the present invention.

[0041] Figure 2 It is an image data in the CAMELYON16 medical pathology image data set used in the experiment of the embodiment of the present invention.

[0042] Figure 3 It is an image data in the medical pathology image BreakHis dataset used in the experiment of the embodiment of the present invention.

[0043] Figure 4 This is the effect of feature clustering of data from 0 to 100 different training periods when the CAMELYON16 image dataset is used as the source domain and the BreakHis image dataset is used as the target domain.

[0044] The technical solutions in the embodiments of the present invention are described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0045] like Figure 1 As shown, this embodiment discloses a cross-modal medical image domain adaptive classification method based on graph neural network, comprising the following steps:

[0046] 1) Obtain cross-modal medical image datasets collected by different imaging conditions and different devices, such as pathological images of different tissues under different staining conditions, skin disease images collected by different dermatoscope devices, diabetic retinopathy images collected by different fundus cameras, etc., and classify the data in the image datasets;

[0047] Specifically: Based on the diseases involved in the obtained medical image dataset, the samples in the image dataset are rated considering their unique levels.

[0048] 2) Select the image datasets as the source domain and the image datasets as the target domain in the experiment, label the image data of the source domain and the target domain, and then preprocess the image data in the source domain and the target domain image datasets;

[0049] Specifically: the images in the selected image dataset are cropped to a size of n×n, where images containing tumors or lesions are positive samples, and images otherwise are negative samples. Secondly, a sample set with readable labels during training is determined as the image dataset of the source domain. The sample set to be classified is used as the target domain image dataset

[0050] in, is the i-th image data in the source domain image dataset, N s is the number of image data in the source domain image dataset; is the jth image data in the target domain image dataset, N t is the number of image data in the target domain image dataset, X s With X t They are the collections of source domain and target domain image data respectively.

[0051] 3) Use the widely used deep neural network, such as VGG, ResNet and AlexNet, to build the backbone network of the model, input the image data of the source domain and the target domain preprocessed in step 2) into the backbone network, extract the features of the image data, and output the feature vector;

[0052] Specifically: use a pre-trained deep neural network as the backbone network of the model, randomly select several source domain and target domain images cropped in step 2), input the images into the backbone network of the model, and in the backbone network of the model, for example, in the ResNet-18 model, set the threshold T for feature space mapping, and generate the corresponding feature vector according to the threshold T.

[0053] 4) Using the feature vectors of the source domain and the target domain extracted in step 3), the maximum mean discrepancy (MMD) loss function is calculated to reduce the difference between the features of the source domain and the target domain image data, and a loss function is designed using the contrast feature constraint to improve the discriminability of the learned features;

[0054] Specifically: by minimizing the Maximum Mean Discrepancy (MMD) loss function, similar features are extracted from the source domain and the target domain. At the same time, a loss function is constructed using contrast feature constraints to minimize the distance between samples of the same category and ensure that the distance between samples of different categories is at least greater than the predefined boundary value d.

[0055] a. Calculate the maximum average difference loss function and minimize the maximum average difference loss function so that similar features are extracted from the image data in the source domain and the target domain:

[0056]

[0057] Among them, L mmd represents the maximum average discrepancy loss function; Represents the i-th image data in the source domain image dataset; is the jth image data in the target domain image dataset; φ(x) represents the feature vector of x generated by the nonlinear mapping in the deep neural network in step S3; H k represents the reproducing kernel Hilbert space associated with the nonlinear mapping φ(·) of the deep neural network in step S3;

[0058] b. Calculate the loss function of the contrast feature constraint and minimize the contrast feature constraint loss function to reduce the distance between the image data of the same category in the source domain and the target domain, and the distance between the image data of different categories is greater than d:

[0059]

[0060] Among them, L g is the loss function for contrast feature constraints; d is the predefined boundary value, l i With l j represent the true categories of the i-th and j-th image data in the source domain and target domain image datasets, respectively, and φ(·) is the nonlinear mapping generated by the deep neural network in step S3.

[0061] 5) The feature vectors of the two domains generated in step 3) are used as nodes in a graph, and the nodes are passed into a high-order graph neural network. The feature vectors of the source domain and the target domain are fused in the high-order graph neural network, so that the label information of the source domain is transferred to the target domain; the high-order graph neural network generates an output vector that fuses the source domain and target domain information and inputs the output vector into the softmax layer. The model generates a predicted label for the image data of the source domain.

[0062] Specifically: In a graph neural network, the features of a node are related to the features of its adjacent neighbor nodes. Therefore, a graph neural network is used to bring similar image data in the source domain and the target domain closer together, thereby transferring the label information of the source domain image data to the image data in the target domain.

[0063] 6) Calculate the total loss function, repeat steps 3) to 5), train the constructed model by minimizing the total loss function, and continuously update and iterate until the model converges;

[0064] Specifically: the total loss function is calculated as: L = L mmd +L g +L c .in, It is the cross entropy loss between the predicted label and the true label of the image data in the source domain output by the model. By minimizing the total loss function, the model parameters are updated until the total loss function of the model converges.

[0065] Among them, L mmd Maximum average discrepancy loss function; L g is the loss function for contrast feature constraints, y is the true label of the image data in the source domain, is the predicted label output by the model for the image data in the source domain, The labels representing the output of the model are expressed as probabilities The information entropy when predicted as a certain class, The labels representing the output of the model are expressed as probabilities The information entropy when predicted as another class.

[0066] 7) inputting the target domain image data preprocessed in step 2) into the convergence model in step 6) for testing, and the model outputs a prediction of the category to which the image data in the target domain belongs;

[0067] Specifically, the last softmax layer in the model outputs a category prediction vector, and the class corresponding to the vector greater than the set threshold ∈ in the category prediction vector is used as the predicted label of the category to which the target domain image data belongs.

[0068] The present invention designs multiple sets of domain adaptation experiments during the experiment, and the medical image datasets used in the three embodiments are CAMELYON16, BreakHis, GlaS and DigestPath. These datasets were originally provided by different institutions and released for different practical applications. Therefore, the diverse appearances presented between the datasets form multi-modal medical image datasets.

[0069] The CAMELYON16 image dataset was provided by two medical centers in the Netherlands. This dataset contains whole-slice images of lymph nodes and was originally used to automatically detect metastatic lesions in images.

[0070] The BreakHis image dataset was collected by a laboratory in Brazil. The dataset contains microscopic image data of breast cancer tissue and is mainly used to classify different tumor types.

[0071] The GlaS image dataset was provided by a hospital and a team of pathologists in the UK. The dataset contains benign and malignant colon cancer image data and was originally used for segmentation tasks.

[0072] The DigestPath image dataset was jointly produced by several hospitals and tissue pathology diagnosis centers in China. The dataset contains full-slice tissue images of colon cancer and aims to automatically segment and classify the lesion areas in the entire tissue.

[0073] Example 1

[0074] The model is trained using the preprocessed open source image dataset CAMELYON16 as the source domain and the BreakHis dataset as the target domain, and the image data in the BreakHis dataset is used as the test set to test the training results.

[0075] Example 2

[0076] The model is trained using the preprocessed open source image dataset CAMELYON16 as the source domain and the GlaS dataset as the target domain, and the image data in the GlaS dataset is used as the test set to test the training results.

[0077] Example 3

[0078] The model is trained using the preprocessed open source image dataset CAMELYON16 as the source domain and the DigestPath dataset as the target domain, and the image data in the DigestPath dataset is used as the test set to test the training results.

[0079] The classification accuracy of multiple groups of domain adaptation experiments in the embodiment is calculated, and the calculation steps of the classification accuracy are as follows:

[0080] 1) Record the number M of image data contained in the test set;

[0081] 2) Compare the predicted labels of the image data in the test set with the real labels obtained after preprocessing, and record the number N of predicted labels equal to the real labels;

[0082] 3) Divide the number of records N in step 2) by the number of records M in step 1) to obtain the classification accuracy R during testing, that is,

[0083] The calculation results are shown in Table 1:

[0084] Table 1 shows the experimental results of multiple domain adaptation experiments designed for multiple medical pathology image datasets.

[0085]

[0086] In Example 1, the open-source tissue pathology image datasets CAMELYON16 and BreakHis are used as the source domain and target domain, respectively. Figure 2 and Figure 3 It can be clearly seen that the appearance of the data in the two datasets is very different, and it is difficult to perform linear classification, so it is very challenging to achieve domain adaptation on datasets in these two fields to classify medical images.

[0087] This embodiment also visualizes the experimental process during the experiment, where: Figure 4 The CAMELYON16 image dataset is used as the source domain, the BreakHis image dataset is used as the target domain, and the feature clustering effect of the data in different training periods from 0 to 100 is calculated according to Figure 4It can be clearly seen that from left to right, the clustering effect of the two category features in the two domains gradually improves. In the early stage of training, that is, the 0th training period, the features of the two categories in CAMELYON16 (source domain) and BreakHis (target domain) are mixed with each other. As the training progresses, by the 100th training period, the features of the two categories in the two domains are clearly distinguished and separated from each other. In addition, the features of the same category in the source domain and the target domain are close to each other, and the features of different categories are far away from each other, which is helpful for the classification task.

[0088] And it can be seen from Table 1 that, since the CAMELYON16 and BreakHis datasets in Example 1 are both breast cancer pathology images, the difference in data distribution between the two domains is the smallest among the three groups of experiments, and the accuracy is the highest among the three groups of experiments. At the same time, in the other two groups of experiments, due to the large difference in data distribution between the two domains and the high difficulty of classification, the accuracy is slightly reduced, but still maintains a high level, which fully proves the feasibility of this method in cross-modal medical image domain adaptation classification tasks with large intra-class differences and small inter-class differences.

[0089] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A cross-modal medical image domain adaptation classification method based on graph neural network, characterized in that: The following steps are involved: S1 obtains a cross-modal medical image dataset, classifies the cross-modal medical image dataset, selects image datasets of a source domain and a target domain from the classified cross-modal medical image dataset, and labels the image data in the source domain and the target domain; S2 preprocesses the image datasets of the source domain and the target domain; S3 uses a deep neural network to build a model backbone network, inputs the image data of the source domain and the target domain preprocessed in step S2 into the model backbone network, extracts the features of the image data, and outputs the feature vectors of the source domain and the target domain; S4 calculates the maximum average difference between the source domain and the target domain feature vectors and the loss function of the contrast feature constraint; S5 passes the feature vectors of the source domain and the target domain as nodes into the high-order graph neural network, generates an output vector that integrates the information of the two domains in the high-order graph neural network, and passes the output vector into the softmax layer to generate a predicted label for the image data of the source domain; S6 calculates the total loss function, repeats steps S3 to S5, and trains the constructed model by minimizing the total loss function until the model converges; S7 inputs the target domain image data preprocessed in step S2 into the convergence model obtained in step S6, and the model outputs a predicted label of the category to which the target domain image data belongs; In step S1, a cross-modal medical image dataset is obtained, and a labeled cross-modal medical image dataset is determined as an image dataset of the source domain. , the cross-modal medical image dataset to be classified is used as the image dataset of the target domain ; in, It is the source domain image dataset. Image data, is the number of image data in the source domain image dataset; It is the first Image data, is the number of image data in the target domain image dataset, and They are the collections of source domain and target domain image data respectively; In step S3, the specific steps of obtaining the feature vector are: a. Use a pre-trained deep neural network as the backbone network of the model; b. Randomly select the source domain and target domain image data preprocessed in step S2 and input them into the backbone network; c. In the backbone network, set the threshold T for feature space mapping, and generate the corresponding feature vector according to the threshold T; In step S4, the maximum average difference loss function is calculated and minimized so that similar features are extracted from the image data in the source domain and the target domain: in, represents the maximum average discrepancy loss function; Represents the source domain image dataset Image data; It is the first Image data; Represents the nonlinear mapping generated by the deep neural network in step S3 The eigenvector of Represents the nonlinear mapping with the deep neural network in step S3 the associated reproducing kernel Hilbert space; In step S4, the contrast feature constraint loss function is calculated and minimized to reduce the distance between the image data of the same category in the source domain and the target domain, and the distance between the image data of different categories is greater than : in, is the loss function for contrast feature constraints; are predefined boundary values, and Represents the source domain and target domain image datasets. The first The true category of the image data, is a nonlinear mapping generated by the deep neural network in step S3; In step S5, a graph neural network is used to bring similar image data in the source domain and the target domain closer together, thereby transferring label information of the source domain image data to the target domain image data; Step S6 specifically includes the following steps: a. Calculate the total loss function as: in, Maximum mean discrepancy loss function; To compare the loss function of feature constraints, is the cross entropy loss between the predicted labels and the true labels of the image data in the source domain output by the model, where is the true label, is the predicted label output by the model described in step S5, The labels representing the output of the model are expressed as probabilities The information entropy when predicted as a certain class, The labels representing the output of the model are expressed as probabilities Information entropy when predicted as another class; b. Update the parameters of the model by minimizing the total loss function until the total loss function of the model converges; In step S7, when the category prediction vector output by the softmax layer of the converged model is greater than the set threshold When , the class corresponding to the class prediction vector is the predicted label of the class to which the target domain image data belongs.

2. According to claim 1, a cross-modal medical image domain adaptive classification method based on graph neural network is characterized in that: In step S1, the cross-modal medical image dataset includes pathological images of different tissues under different staining conditions, skin disease images collected by different dermatoscopes, and diabetic retinopathy images collected by different fundus cameras.

3. According to claim 1, a cross-modal medical image domain adaptive classification method based on graph neural network is characterized in that: In step S2, the preprocessing is to crop the images in the image datasets of the selected source domain and target domain, and the cropping size is , determine the category set C into which the image data in the image dataset is to be divided, where the category set C includes two categories: positive samples and negative samples.

Images