Multi-label gastroscopy image classification method with multiple image inputs based on graph neural network

Abstract

The present application relates to a multi-label gastroscope image classification method based on multi-image input of a graph neural network, which belongs to the technical field of intelligent processing of medical images. The method comprises: preprocessing multiple gastroscope images in sample data to obtain preprocessed gastroscope images; obtaining a co-occurrence matrix between overall label results; inputting the preprocessed gastroscope images and the co-occurrence matrix into a preset network model for training to obtain a classification model to classify at least one input image; the BiT‑ResNet in the network model is used to extract image features of the preprocessed gastroscope images; the Attention layer is used to calculate the weights of each part of the image features to obtain updated image features; the graph neural network is used to output network results according to the co-occurrence matrix, and the network results and the updated image features are used to generate classification results; compared with using multiple single-image single-label image classification methods, better classification effects can be obtained.

Description

[Technical field]

[0001] The present application relates to a multi-label gastroscopy image classification method based on multi-image input of a graph neural network, which belongs to the technical field of medical image intelligent processing. [Background technology]

[0002] With the development and application of new technologies such as cloud services, big data, and artificial intelligence, deep learning networks are gradually being applied to the fields of image classification and image recognition. The combination of deep learning networks and medical care to assist doctors in diagnosis and treatment has also become a hot research field.

[0003] Existing image recognition is usually used to determine whether a single image belongs to a certain label category.

[0004] However, in the real world, a picture may contain multiple objects, and more than one picture needs to be judged at a time. For example, a gastroscopy corresponds to the input of multiple gastroscopy pictures and the output of multiple disease labels. Existing image recognition technology can be applied to this problem, but it ignores the relationship between labels and cannot find the key points in multiple pictures well, which affects the performance of image classification. [Summary of the invention]

[0005] This application provides a multi-label gastroscopy image classification method based on multi-image input of graph neural network, which can fully capture the key points between images and the connection between labels, and improve the accuracy of automatic recognition of gastroscopy samples. This application provides the following technical solutions:

[0006] Acquire multiple groups of sample data, each group of sample data includes multiple gastroscopic images and overall label results corresponding to the multiple gastroscopic images;

[0007] Preprocessing the multiple gastroscopic images to obtain preprocessed gastroscopic images;

[0008] Obtaining a co-occurrence matrix between the overall label results;

[0009] Inputting the preprocessed gastroscope image and the co-occurrence matrix into a preset network model, and training the network model using a preset loss function and the overall label result to obtain a classification model, wherein the classification model is used to classify at least one input image;

[0010] Among them, the preset network model includes BiT-ResNet, an attention layer connected to the BiT-ResNet, and a graph neural network connected to the Attention layer; the BiT-ResNet is used to extract the image features of the preprocessed gastroscopy images; the Attention layer is used to calculate the weights of each part of the image features to obtain updated image features; the graph neural network is used to output network results according to the co-occurrence matrix, and the network results and the updated image features are used to generate classification results.

[0011] Optionally, preprocessing the plurality of gastroscopic images to obtain preprocessed gastroscopic images includes:

[0012] Scaling the multiple gastroscope images to obtain multiple images of preset sizes;

[0013] Combine multiple pictures of preset sizes;

[0014] The spliced ​​image data is standardized to obtain the preprocessed gastroscopy image.

[0015] Optionally, before inputting the preprocessed gastroscopy images and the co-occurrence matrix into a preset network model, and training the network model using a preset loss function and the overall label result to obtain a classification model, the method further includes:

[0016] Initialize parameters through the BiT-ResNet pre-trained model.

[0017] Optionally, the preset network model is also used to multiply the network result as a classifier vector with the updated image feature to obtain the probability of image classification; and obtain the classification result according to the probability of image classification.

[0018] Optionally, the network result is multiplied by the updated image feature as a classifier vector, and the probability of image classification is obtained by activating the sigmoid function.

[0019] Optionally, the preset loss function is expressed by the following formula:

[0020]

[0021] Among them, y i is the true label distribution of sample i, x i is the predicted label distribution of the model for sample i.

[0022] Optionally, the method further comprises:

[0023] After the classification model is trained, the classification model is tested using a test set to determine the performance of the classification model.

[0024] Optionally, the method further comprises:

[0025] When using the classification model, a plurality of gastroscopic images to be classified are preprocessed;

[0026] The preprocessed gastroscopic images are input into the classification model to obtain an overall classification result of the multiple gastroscopic images to be classified.

[0027] Optionally, the overall label result includes one or more of the following: healthy stomach, gastric polyp, gastric ulcer, gastric cancer, gastritis and bile reflux.

[0028] The beneficial effects of the present application include at least: by collecting and organizing the images and results of gastroscopy, each sample contains multiple gastroscopy images and an inspection report, corresponding to a gastroscopy. Then, the sample images are labeled according to the inspection report, and are divided into 6 categories, namely: healthy stomach, gastric polyps, gastric ulcers, gastric cancer, gastritis and bile reflux, and each sample can contain one or more labels. Then, the images in each sample are preprocessed and spliced, and input into the BiT-ResNet model and the subsequent Attention layer to obtain a feature representation of a unified dimension. At the same time, a graph neural network between labels is constructed through label co-occurrence information, and a classifier is finally trained to classify the feature representation using the classifier, and the labels contained in the sample can be determined. Since each sample contains multiple images, and not every image contributes to the label of the sample, the addition of the Attention layer can perceive the key points in multiple images, and the classifier trained by the graph neural network can make full use of the connection between labels. Therefore, compared with the use of multiple traditional single-image single-label image classification methods, better classification results can be obtained by this method.

[0029] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0030] Figure 1 This is a schematic diagram of the network structure of BiT-ResNet provided by an embodiment of the present application;

[0031] Figure 2 This is a flowchart of a multi-label gastroscopy image classification method based on multi-image input of a graph neural network provided by an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of the training process of the classification model provided by an embodiment of the present application. [Specific implementation method]

[0033] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0034] First, several terms involved in this application are introduced.

[0035] BigTransfer (BiT) model: is a pre-trained residual network (ResNet) that can be used as a starting point for any visual task. The pre-trained ResNet uses the classic ResNet architecture to increase the model capacity and pre-train on larger datasets, which can improve network performance by increasing the computational budget and training time. Fine-tuning the pre-trained BiT model for downstream tasks, BiT only needs to be pre-trained once, and then the cost of fine-tuning for downstream tasks is very low. Fine-tuning involves many hyperparameters, such as learning rate, weight decay, etc. Based on this, a hyperparameter tuning (BiT-HyperRule) method is proposed. This method is only based on high-level dataset features, such as image resolution and number of annotated samples. In other words, BiT does not perform hyperparameter adjustments for each downstream task. This can effectively reduce the cost of task adaptation.

[0036] refer to Figure 1 The transfer learning method shown in the figure is based on the ResNet model framework and uses the BiT pre-training model for training to obtain the BiT-ResNet network structure diagram. Figure 1 It can be seen that BiT-ResNet is based on ResNet-50, that is, BiT-ResNet consists of 50 weighted layers, and the convolution kernels used are 1*1, 3*3, and 1*1. ResNet-50 is mainly divided into 6 parts, namely conv1, conv2_x, conv3_x, conv4_x, conv5_x, and the final global average pooling layer and fully connected layer. The global average pooling layer is used to map the two-dimensional feature map into a vector and input it to the fully connected layer. The fully connected layer consists of two layers. The first fully connected layer contains 1000 neurons, and the second fully connected layer contains 100 neurons, and the final output representation corresponding to the input image is obtained.

[0037] It should be noted that in actual implementation, the network structure of BiT-ResNet may be slightly different, for example, the number of weighted layers in the residual network is different, or the number of neurons in the fully connected layer is different. This embodiment does not limit the network structure of BiT-ResNet.

[0038] Graph Neural Networks (GNN): The data used in traditional machine learning is in Euclidean Domain. The most notable feature of data in Euclidean Domain is that it has a regular spatial structure. For example, pictures are regular square grids, and voice data is a one-dimensional sequence. These data can be represented by a one-dimensional or two-dimensional matrix. At the same time, there is a core assumption: samples are independent of each other. However, in real life, many data do not have a regular spatial structure, that is, data in non-Euclidean space, such as graphs abstracted from electronic transactions and recommendation systems. The connection between each node and other nodes in the graph is not fixed. Graph neural networks can model data in non-Euclidean space and capture the internal dependencies of the data. Graph neural networks are irregular and disordered.

[0039] Since the characteristics of graphs can well characterize the interdependence between labels, we can choose to model and learn based on graph neural networks, so as to use them in image classification networks.

[0040] A typical graph neural network structure includes an input layer, a hidden layer, a rectified linear unit (ReLU) layer, and an output layer. Among them, the hidden layer and the ReLU layer can be repeated multiple times. In actual implementation, the graph neural network can also be other structures, and this embodiment does not limit the implementation method of the graph neural network.

[0041] The input layer includes the feature description matrix H(n*d) and the correlation coefficient matrix A(n*n), where n is the number of nodes, that is, the number of all labels, and d is the dimension of the feature. The feature dimension is determined by the structure of the neural network used. Among them, the correlation coefficient matrix is ​​a matrix that represents the label relationship in the GCN network.

[0042] Schematically, the correlation coefficient matrix A is initialized by mining the co-occurrence of tags in the dataset. i is the number of times label i appears in the data, M ij is the probability that label i and label j appear together. Dividing the two can get the conditional probability matrix P between different labels.

[0043] P ij =M ij / N i

[0044] The hidden layer is used to learn the relevant information between nodes and update the node representation. Each Hidden Layer can be represented by a nonlinear function.

[0045] Co-occurrence matrix: It can count the number of times that classification labels appear at the same time. The basic steps to construct a co-occurrence matrix are: first separate the labels of each text and convert them into a list, and at the same time build a dictionary about labels, build an empty matrix to store the co-occurrence matrix of labels, and then calculate the co-occurrence frequency between labels.

[0046] Attention Mechanism: enables a neural network to focus on a subset of its input (or features).

[0047] Optionally, the present application takes the execution subject of each embodiment as an electronic device with image processing capability as an example for illustration. The electronic device may be a terminal or a server. The terminal may be a computer, a laptop computer, a tablet computer, a medical diagnostic device, etc. This embodiment does not limit the type of terminal and the type of electronic device.

[0048] Figure 2 This is a flowchart of a multi-label gastroscopy image classification method based on multi-image input of a graph neural network provided by an embodiment of the present application. The method includes at least the following steps:

[0049] Step 201, obtaining multiple groups of sample data, each group of sample data includes multiple gastroscopic images and overall label results corresponding to the multiple gastroscopic images.

[0050] Optionally, the overall label result includes one or more of the following: healthy stomach, gastric polyp, gastric ulcer, gastric cancer, gastritis and bile reflux.

[0051] For example, 2,400 gastroscopy samples are collected, each containing 3 to 99 gastroscopy images and an examination report. The examination report in each sample will give the overall results of several images in the sample, and the results will contain one or more of the following six labels: healthy stomach, gastric polyps, gastric ulcers, gastric cancer, gastritis, and bile reflux. The 2,400 samples are divided into training sets, validation sets, and test sets in a ratio of 7:2:1, and a multi-label image classification network with multiple image inputs based on a graph neural network is trained to finally obtain an automatic classification model.

[0052] Step 202, preprocessing the plurality of gastroscope images to obtain preprocessed gastroscope images.

[0053] Multiple gastroscopy images are preprocessed to obtain input suitable for the network model.

[0054] In one example, a plurality of gastroscopic images are preprocessed to obtain preprocessed gastroscopic images, including: scaling the plurality of gastroscopic images to obtain a plurality of images of preset sizes; splicing the plurality of images of preset sizes; and standardizing the spliced ​​image data to obtain preprocessed gastroscopic images.

[0055] The preset size may be 3*224*224, or other sizes. This embodiment does not limit the value of the preset size. Accordingly, n pictures of the preset size are spliced ​​to obtain a spliced ​​image in a new dimension, such as n*3*224*224. n is a positive integer.

[0056] Step 203, obtaining the co-occurrence matrix between the overall label results.

[0057] Optionally, the co-occurrence matrix may be obtained through an automatic statistical tool or manual statistical analysis. This embodiment does not limit the method for obtaining the co-occurrence matrix.

[0058] Step 204, input the preprocessed gastroscope image and the co-occurrence matrix into a preset network model, and use the preset loss function and the overall label result to train the network model to obtain a classification model, which is used to classify at least one input image; wherein the preset network model includes BiT-ResNet, an attention layer connected to BiT-ResNet, and a graph neural network connected to the Attention layer; BiT-ResNet is used to extract image features of the preprocessed gastroscope image; the Attention layer is used to calculate the weights of each part of the image features to obtain updated image features; the graph neural network is used to output network results according to the co-occurrence matrix, and the network results and the updated image features are used to generate classification results.

[0059] In this embodiment, before training, that is, before this step, it is necessary to initialize parameters through the BiT-ResNet pre-training model.

[0060] During training, an initial learning rate (such as 0.0001) is used, a stochastic gradient descent (SGD) optimizer is used to optimize the network parameters of the network model, and a multi-label one-versus-all loss based on maximum entropy is used as the loss function for training.

[0061] Among them, the preset loss function is expressed by the following formula:

[0062]

[0063] y iis the true label distribution of sample i, x i is the predicted label distribution of the model for sample i.

[0064] Optionally, in order to accommodate image inputs of different dimensions, the batch size is set to 1.

[0065] Optionally, set the learning rate to decrease exponentially with a value of 0.99.

[0066] The preset network model is also used to multiply the network result as a classifier vector with the updated image features to obtain the probability of image classification; and obtain the classification result according to the probability of image classification. Specifically, the network result is multiplied as a classifier vector with the updated image features, and the probability of image classification is obtained by activating the sigmoid function.

[0067] Schematically, the probability p of each label is i If it is greater than 0.5, the sample is judged to contain this label.

[0068] Among them, the Attention layer can calculate the weight of each part of the input feature, which is the key to identifying the lesion part of the image. The Attention layer built using Attention technology will assign different weights to each area of ​​the image feature. The sum of all weights is 1, and the size of the weight represents the importance of the area. And finally, the image features of different dimensions in a sample are converted into a unified size of 2048.

[0069] One of the most advanced multi-label classification methods is used when building the graph neural network, which effectively captures the distribution of labels and the relationship between labels. The graph neural network can well preserve various information in the logical graph. By building the graph neural network according to the co-occurrence relationship between labels, the relationship between labels can be preserved.

[0070] refer to Figure 3 The preset network model shown (the same as the network structure of the classification model) includes BiT-ResNet, an attention layer connected to BiT-ResNet, and a graph neural network connected to the Attention layer; BiT-ResNet is used to extract image features of preprocessed gastroscopy images; the Attention layer is used to calculate the weights of each part of the image features to obtain updated image features; the graph neural network is used to output network results according to the co-occurrence matrix, and the network results and the updated image features are multiplied and activated by the sigmoid function to obtain the classification results.

[0071] Optionally, after the classification model is trained, the classification model is tested using a test set to determine the performance of the classification model.

[0072] During the test, the image is still scaled, spliced ​​and standardized, converted into n*3*224*224 image features, and input into the trained classification model to obtain the final classification result. The whole process does not require human participation, which reduces human factors and can provide doctors with a basis for auxiliary diagnosis more efficiently.

[0073] Table 1 compares the performance of different methods on the test set. Since the training set only has an overall annotation for multiple images in each sample and cannot correspond to a specific image, the traditional single-image single-label classification method cannot be applied here. The model structure of the method without the attention layer and the method with the attention layer both use graph neural networks to capture the connection between labels, and the only difference is the presence or absence of attention. It can be seen that the addition of the attention layer greatly improves the performance of the model, and all indicators have been improved by more than double. The huge performance difference highlights the importance of the attention layer. Combined with the capture of the connection between labels by the graph neural network, the micro-average F1 value of this method has reached 75%, which can be fully applied to the auxiliary diagnosis and treatment of gastroscopy.

[0074] Table 1

[0075] Using the Model Accuracy Micro-average F1 value Macro average F1 value No Attention layer 0.100 0.442 0.120 With Attention layer 0.525 0.750 0.577

[0076] Among them, the accuracy examines whether the predicted distribution of each sample is completely consistent with the true distribution, the micro-average F1 value mainly examines the classification correctness of labels with a small sample size, and the macro-average F1 value mainly examines the classification correctness of labels with a large sample size.

[0077] Optionally, when using the classification model, multiple gastroscopic images to be classified are preprocessed; the preprocessed gastroscopic images are input into the classification model to obtain the overall classification results of the multiple gastroscopic images to be classified. In this case, no matter how many images are input, the model can be used for prediction.

[0078] In summary, the multi-label gastroscopy image classification method based on multi-image input of the graph neural network provided in this embodiment collects and organizes the images and results of gastroscopy examinations. Each sample contains multiple gastroscopy images and an examination report, corresponding to a gastroscopy examination. Then, the sample images are labeled according to the examination report, and are divided into 6 categories, namely: healthy stomach, gastric polyps, gastric ulcers, gastric cancer, gastritis, and bile reflux. Each sample can contain one or more labels. Then, the images in each sample are preprocessed and spliced, and input into the BiT-ResNet model and the subsequent Attention layer to obtain a feature representation of a unified dimension. At the same time, a graph neural network between labels is constructed through label co-occurrence information, and a classifier is finally trained. The classifier is used to classify the feature representation to determine which labels the sample contains. Since each sample contains multiple images, and not every image contributes to the label of the sample, the addition of the Attention layer can perceive the key points in multiple images, and the classifier trained by the graph neural network can make full use of the connection between labels. Therefore, compared with using multiple traditional single-image single-label image classification methods, better classification results can be achieved through this method.

[0079] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the multi-label gastroscopy image classification method based on multi-image input of a graph neural network in the above-mentioned method embodiment.

[0080] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the multi-label gastroscopy image classification method based on multi-image input of a graph neural network in the above-mentioned method embodiment.

[0081] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-label gastroscopy image classification method based on multi-image input of graph neural network, characterized in that: The method comprises: Acquire multiple groups of sample data, each group of sample data includes multiple gastroscopic images and overall label results corresponding to the multiple gastroscopic images; Preprocessing the multiple gastroscopic images to obtain preprocessed gastroscopic images; Obtaining a co-occurrence matrix between the overall label results; Inputting the preprocessed gastroscope image and the co-occurrence matrix into a preset network model, and training the network model using a preset loss function and the overall label result to obtain a classification model, wherein the classification model is used to classify at least one input image; The preset network model includes BiT-ResNet, an attention layer connected to the BiT-ResNet, and a graph neural network connected to the Attention layer; the BiT-ResNet is used to extract the image features of the preprocessed gastroscopy image; the Attention layer is used to calculate the weights of each part of the image features to obtain updated image features; the graph neural network is used to output network results according to the co-occurrence matrix, and the network results and the updated image features are used to generate classification results; The BiT-ResNet adopts a transfer learning method based on the ResNet model framework and uses a BiT pre-training model for training to obtain the network structure of BiT-ResNet.

2. The method according to claim 1, characterized in that The preprocessing of the plurality of gastroscopic images to obtain preprocessed gastroscopic images includes: Scaling the multiple gastroscope images to obtain multiple images of preset sizes; Combine multiple pictures of preset sizes; The spliced ​​image data is standardized to obtain the preprocessed gastroscopy image.

3. The method according to claim 1, characterized in that The method further includes: inputting the preprocessed gastroscope image and the co-occurrence matrix into a preset network model, and training the network model using a preset loss function and the overall label result to obtain a classification model. Initialize parameters through the BiT-ResNet pre-trained model.

4. The method according to claim 1, characterized in that The preset network model is also used to multiply the network result as a classifier vector with the updated image feature to obtain the probability of image classification; and obtain the classification result according to the probability of image classification.

5. The method according to claim 4, characterized in that The network result is used as a classifier vector and multiplied by the updated image feature, and the probability of image classification is obtained by activating the sigmoid function.

6. The method according to claim 1, characterized in that The preset loss function is expressed by the following formula: Among them, y i is the true label distribution of sample i, x i is the predicted label distribution of the model for sample i.

7. The method according to claim 1, characterized in that The method further comprises: After the classification model is trained, the classification model is tested using a test set to determine the performance of the classification model.

8. The method according to claim 1, characterized in that The method further comprises: When using the classification model, a plurality of gastroscopic images to be classified are preprocessed; The preprocessed gastroscopic images are input into the classification model to obtain the overall classification results of the multiple gastroscopic images to be classified.

9. The method according to claim 1, characterized in that: The overall label result includes one or more of the following: healthy stomach, gastric polyp, gastric ulcer, gastric cancer, gastritis and bile reflux.

Images