Method for performing disease classification diagnosis on CT image of liver based on neural network

Abstract

A method for automatically judging which of three common diseases, namely hepatocellular carcinoma (HCC), intrahepatic cholangiocellular carcinoma (ICC) and fatty liver (FAT), the liver belongs to is based on a neural network introduced with r-IBS, and the method comprises the following steps: firstly, constructing a training classifier based on r-IBS, and judging which of the diseases belongs to each liver CT by using the classifier; and collecting CT images of the liver, carrying out image processing by using a snake model, inputting the images into the trained classification neural network model, and outputting a classification result of the diseases from the neural network.

Description

Technical Field

[0001] The present invention relates to deep learning, convolutional neural network, information similarity, statistical analysis theory, category memory, in the classification direction, K nearest neighbor algorithm (KNN), information based similarity (IBS), and application of contrastive learning. Background Art

[0002] Liver cancer is a serious health problem worldwide. According to data released by the International Agency for Research on Cancer, there will be nearly 2.5 million new cases of liver cancer in 2022, accounting for about 12% of all new cancer cases worldwide, ranking relatively high.

[0003] Nowadays, imaging examinations also play an important role in the diagnosis of liver diseases. Imaging technologies such as ultrasound, CT, and MRI can clearly show the morphology and structural changes of the liver, which helps doctors to accurately diagnose and stage liver diseases. In addition, liver elastic imaging technology can also determine the status of liver fibrosis, providing an important basis for early intervention and treatment of liver diseases.

[0004] As artificial intelligence has made rapid progress. In the field of computer vision, convolutional neural networks have become more powerful, still dominate, and have achieved great success. However, liver imaging datasets are different from most commonly used classification datasets and have many unique characteristics. They often have small data volumes, contain a lot of noise, are difficult to understand, and have uneven data distribution. Therefore, existing network architectures suitable for conventional classification datasets may not be suitable for helping diagnose liver diseases and abnormalities.

[0005] At present, CT images are widely used in liver detection tasks. However, since medical images have high requirements on the network's ability to discover feature specificity and feature granularity, we not only created a new algorithm based on the concept of information similarity, which we call the r-IBS (reflection IBS) algorithm, but also implemented KNN online learning for natural animal image classification tasks. We also applied the network to medical CT image datasets to explore and verify the classification effect of the network on this task. In addition, traditional manual classification methods are time-consuming and labor-intensive, resulting in increasing work pressure on doctors and cannot meet the application scenarios of large-scale medical images.

[0006] If we can use computers to automatically classify the types of diseases in the data set by putting CT images into the network, doctors can conduct a slight verification without having to put in as much effort as before, which can greatly reduce the workload of medical staff.

[0007] KNN Online Learning: The KNN algorithm can easily handle new data. When a new training sample arrives, it only needs to be added to the training data set without retraining the entire model. This makes the KNN algorithm very suitable for online learning scenarios. When new data arrives, the algorithm can immediately use this new data for prediction without the need for complex adjustments to the model. Unlike many other machine learning algorithms. This means that in an online learning environment, when new data arrives, the KNN algorithm can immediately start using this data for prediction without waiting for model training to complete.

[0008] The present invention uses a new neural network model based on neural network to classify disease types using liver CT images. The new network structure mainly includes four parts: a traditional convolutional neural network feature extraction module, a new r-IBS (reflection IBS) algorithm module inspired by information similarity, an online learning KNN classification module, and a loss function. The extraction module mainly uses the CNN model. CNN is a machine learning method that is often used in image analysis. Its inventor was originally inspired by the structure of the visual cortex of the human brain. It is generally used for image recognition and classification problems. It has achieved good results in these tasks and has achieved excellent results in various competitions. The memory processing module is a memory network module based on similarity. This module mainly simulates the network module constructed by the human memory storage and update mechanism. It is responsible for generating memory during training, updating memory, and modifying input images. The KNN classification module is capable of realizing KNN online learning, which has the advantages of rapid adaptation to new data, real-time feedback and adjustment, processing data streams, and high resource utilization. The loss function will be introduced later. Summary of the invention

[0009] The present invention solves the above-mentioned problem: the network model for predicting common liver diseases of patients based on CT images proposed in the present invention can help patients reduce the greater damage to the liver caused by X-ray examinations during examinations, and reduce the workload of doctors. At the same time, the prediction accuracy of this network model is also excellent.

[0010] The present invention solves the above-mentioned problems: the neural network-based algorithm model proposed in the present invention is based on similarity calculation and category memory and is based on a new classification network of an online KNN classifier to help automatically diagnose common liver diseases, greatly reducing the workload of medical staff, while improving the accuracy of prediction of common liver diseases, so as to achieve early detection and early treatment for patients and reduce greater damage to the liver caused by late detection.

[0011] The present invention solves the above-mentioned problem: the category memory storage unit and the online KNN classifier proposed in the present invention can continuously modify the memory according to the input image during training. On the one hand, this method ensures the continuous updating of the category memory, and each memory continuously absorbs the characteristics of the specific category image input to the network. On the other hand, through KNN classification, there are k outputs, that is, the calculated k largest IBSsub are put into the KNN classifier, and the probability values ​​of the three categories are re-weighted and calculated according to different categories, which can continuously improve the prediction accuracy of the network, thereby using computers to solve the problem that the diagnosis of less experienced doctors is not as accurate as that of experienced doctors.

[0012] The present invention solves the above-mentioned problems: the neural network-based algorithm model proposed in the present invention references memory to help the network classify CT images. In order to increase resource utilization, we introduce KNN online learning. The present invention solves the problem of improving efficiency. The online KNN algorithm usually improves efficiency through some optimization strategies, such as maintaining a data structure to quickly find the nearest neighbor. The neural network-based method of the present invention for classifying three common diseases, namely hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC) and fatty liver (FAT), using liver CT images includes the following steps:

[0013] Step 1: CT images of the liver were collected. The 163 subjects included 90 cases of hepatocellular carcinoma and 73 cases of intrahepatic cholangiocarcinoma. In addition, it is very important to classify whether the patient has liver cancer in this study, so the study also included 70 cases of fatty liver to form a control group with liver cancer cases.

[0014] Step 2: Divide the obtained liver CT images. Most of the images are used as network training sets, and a small number of images are used as network validation sets. The original CT images in the data set are whole abdominal images. In order to make the training objectives clear, we segment and remove other pixels that are not related to the liver. We use a semi-automatic snake model method combined with the Unet network to automatically segment the data. The initial model used by snake is provided by the radiologist of the cooperative hospital.

[0015] Step 3: Before the image is input into the network, it needs to be uniformly adjusted to a consistent size to ensure the correct input size. Of course, it also needs to be standardized according to the overall mean and standard deviation of the data set, converting the image data to between [0, 1], converting the mean of the data distribution to 0, and converting the standard deviation to 1, so that the data conforms to the standard normal distribution, which can speed up the convergence of the model and the training accuracy.

[0016] Step 4: Establish a CT image feature extraction module based on a convolutional neural network, input the training set of liver CT images into the model, fix the output dimension of the feature extraction network to H (default H = 2048), output the features extracted from the image, and pass them to the next part of the network.

[0017] Step 5, establish a network classifier module (B*C*M) based on r-IBS. There are as many r-IBS similarity calculation modules as there are classes in the network classification module. In this invention, we take three types of diseases as an example and therefore consider that there are three r-IBS similarity calculation modules, where B is the batch size, M is the size of each memory set (the size of the memory dimension) (default value = 16), and C is the number of categories we expect to classify. In this task, we consider C to be 3. The extracted image feature data module and the pre-stored memory module inspired by information similarity are used for r-IBS similarity calculation to help the memory to be updated and modified.

[0018] Step 6: Construct a classifier to determine the category of liver disease in the input image. The classifier is composed of the KNN classification combination of the IBSsub value.

[0019] Step 7: Complete the training and fine-tuning of the overall module, and then perform an evaluation test on the test set. After the test meets the standards, the network module is applied to more predictive diagnosis of liver CT images.

[0020] Furthermore, in step 5, the r-IBS module can be divided into several steps: x and y represent two systems. In this network, x represents the features extracted from the input image, which is generally 2048-dimensional, and y represents the initialized memory stored in the network itself. Then, x and y are mapped, encoded and translated using the same Encoder. After translation, they are mapped to a fixed-dimensional channel, and the value on the channel represents the performance of the input on the channel. The similarity and difference are measured by measuring the energy size and direction of the channels of the two systems on the mapping plane. We believe that if the two systems are completely similar, then the energy values ​​of their projected channels should be exactly the same, and the projection feature directions should be exactly the same; when the directions are completely opposite and the absolute value and energy value are higher, the two systems are more different.

[0021] Rx=Encoder(x)

[0022] Ry=Encoder(y)

[0023] The specific r-IBS calculation process is shown in the following formula:

[0024]

[0025]

[0026] Px_log=-Px·logPx

[0027] Py_log=-Py·logPy

[0028]

[0029] IBSsub=∑|Rx-Ry|·Fk

[0030] IBSplus=Σ|Rx+Ry|·Fk

[0031] Where R represents the projection vector obtained after the input passes through the unified encoder. The positive and negative values ​​of R represent the feature directions on a certain plane after all features are projected onto the plane. If they belong to the same category, the two directions should be consistent, and if they belong to different categories, they should be opposite. The absolute value of R represents the energy size. P represents the energy size ratio of each channel of the projection vector, which characterizes the information density of the projection vector. Fk is the ratio of the sum of the information entropy of the two projection vectors on each channel. The smaller the information entropy, the more stable the system is, and the lower the uncertainty of the channel value after projection. Finally, we subtract the R values ​​of the two systems and calculate IBSsub through the formula to represent the similarity between the two systems. We add the R values ​​of the two systems and calculate IBSplus through the formula to represent the difference between the two systems. Finally, in practical applications, we use IBSsub as the similarity value r-NorIBS, and use the difference value IBSplus to normalize IBSsub to obtain r-IBS. The value range of r-IBS is 0 to 1.

[0032] The loss function mainly consists of two parts: Losssoftmax and LossIBS, where LossIBS is:

[0033]

[0034] Loss = Losssoftmax + LossIBS

[0035] The total loss function is the cross entropy loss and the IBS loss. The IBS loss is to constrain the memory maintained by each IGM unit to the image of its corresponding category, so that the memory maintained by each IGM is of the same category as itself without crosstalk. Losssoftmax is the traditional classification loss. After using r-IBS as the output probability, the two Losses have different meanings, but they are essentially the same in form.

[0036] Step 6, let's take a closer look at the construction of similarity calculation. After obtaining the IBSsub similarity, we borrow the idea of ​​the K nearest neighbor algorithm to select the k with the largest IBSsub value. The selection of k is crucial. We use a pseudo code to illustrate the idea of ​​KNN classification:

[0037]

[0038] Compared with the existing technology, the present invention has the following beneficial effects: extracting image features through convolutional neural network, introducing category memory and online learning KNN algorithm, helping the network to extract image features of CT data set, and helping the network to perform classification prediction, solving the problems of small amount of CT image data, large noise, and image feature extraction, and being able to efficiently predict diseases. Inviting experienced doctors to meet the data set standards, training the network can make the network prediction achieve excellent classification effect, solving the problems of large workload, slow speed, and inconsistent accuracy of doctors' diagnosis.

[0039] Now, when there is a new data point that needs to be classified, the model will calculate the distance between this data point and all samples in the window and select the K samples with the closest distance (K is another preset constant). Then, the model will look at the categories of these K samples and select the category with the most occurrences as the predicted category of the new data point, which greatly improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly and easily explain the embodiments of the present invention or the technical solutions in the prior art, I will now briefly introduce the drawings required to be used in the description of the embodiments or the prior art.

[0041] Figure 1 The overall model architecture diagram of the neural network of the KNN classification algorithm based on the memory network provided by the embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the r-IBS calculation process; DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is clearly and detailed described below with reference to the accompanying drawings and specific examples.

[0044] The solution adopted by the present invention to solve the problem is divided into 4 parts:

[0045] In the first part, a classifier is constructed to identify the liver disease type on each CT image, whether it is hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC), or fatty liver (FAT). The liver CT images that have been segmented and confirmed by professional radiologists are labeled to mark the disease type on the CT images. Each liver CT image is standardized to unify the image size and grayscale range. The processed liver CT image dataset is used as the input of the neural network training model, and the corresponding labels are used as the output of the neural network to train the neural network model. The schematic diagram of the network model is shown below. Figure 1 As shown, the trained neural network model with high classification accuracy is used as a classifier to determine the type of liver disease of the patient on CT, and is deployed on a computer host or server.

[0046] In the second part, a conventional dose of CT is used to scan the liver, and a complete CT image of the liver of the patient to be tested is collected. The CT image is then segmented by a professional radiologist to include only the liver area. The captured image is then preprocessed for the training set during training, i.e., the CT image is completed and scaled, and the image size and grayscale range are unified.

[0047] In the third part, each liver CT image is used as the input of the neural network, and the output of the network is used as the judgment result to determine which type of disease the CT image belongs to.

[0048] It should be understood that although this specification is described in terms of implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only to describe an example more clearly. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0049] The detailed descriptions of a series of ideas and methods listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. The scope of protection of the present invention should include this innovative way of solving problems using the methods and network structures of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for classifying and diagnosing diseases in liver CT images based on a neural network, characterized in that The following steps are involved: Step 1: Collect a CT image of the abdomen and ask a radiologist to segment the CT image of the liver. Another doctor will then confirm the accuracy of the segmentation, thereby obtaining a reliable CT image of the liver. Step 2: The obtained liver CT images are divided into two groups, with most of the images used as network training sets and a small number of images used as network validation sets; Step 3: Preprocess the images of the training set and the validation set; Step 4, establishing a dental CT image feature extraction module based on convolutional neural network; Step 5, establish a network classifier module (B*C*M) based on r-IBS, and there are as many r-IBS similarity calculation modules as there are classes in the network classification module. Step 6: Construct a classifier to determine the category of liver disease in the input image. The classifier is composed of the KNN classification combination of the IBSsub value. Step 7: Use the total loss function to complete the training and fine-tuning of the overall module, and then perform an evaluation test on the test set. After the test meets the standards, the network module is applied to more predictive diagnosis of liver CT images.

2. According to step 1 of claim 1, CT images of the liver were collected, and the 163 research subjects included 90 cases of hepatocellular carcinoma and 73 cases of intrahepatic cholangiocarcinoma. In addition, it is very important to classify whether the patient has liver cancer in this study, so the study also included 70 cases of fatty liver to form a control group with liver cancer cases.

3. According to step 2 of claim 1, the obtained liver CT images are divided, most of the images are used as network training sets, and a small number of images are used as network verification sets, and the original CT images in the data set are whole abdominal images.

4. According to step 3 of claim 1, in order to make the training goal clear, we segmented and removed the pixels of other parts of the liver that were not related to the liver, and used a semi-automatic snake model method combined with a Unet network to automatically segment the data. The initial model used by the snake was provided by the radiologist of the cooperative hospital.

5. According to step 4 of claim 1, a CT image feature extraction module based on a convolutional neural network is established, the training set of liver CT images is input into the model, the output dimension of the feature extraction network is fixed to H (default H = 2048), the features extracted from the output images are passed to the next part of the network.

6. The r-IBS module of step 5 according to claim 1 can be divided into several steps: x and y represent two systems. In this network, x represents the features extracted from the input image, which is generally 2048-dimensional. y represents the initialized memory stored in the network itself. Then, x and y are mapped, encoded and translated using the same Encoder. After translation, they are mapped to a fixed-dimensional channel. The value on the channel represents the performance of the input on the channel. The similarity and difference are measured by measuring the energy size and direction of the channels of the two systems on the mapping plane. We believe that if the two systems are completely similar, then the energy values ​​of their projected channels should be completely consistent, and the projection feature directions should be exactly the same; when the directions are completely opposite and the absolute value and energy value are higher, the two systems are more different. Rx=Encoder(x) Ry=Encoder(y) The specific r-IBS calculation process is shown in the following formula: P x_logx =-p x logp x P y_logy =-p y logp y IBSsub=∑|Rx-Ry|·Fk IBSsub=Σ|Rx-Ry|·Fk Among them, R represents the projection vector obtained after the input passes through the unified encoder. The positive and negative values ​​of R represent the feature directions on a certain plane after all features are projected onto the plane. If they belong to the same category, the two directions should be consistent, and if they belong to different categories, they should be opposite. The absolute value of R represents the energy size. P represents the energy size ratio of each channel of the projection vector, which characterizes the information density of the projection vector. Fk is the ratio of the sum of the information entropy of the two projection vectors on each channel. The smaller the information entropy, the more stable the system is, and the lower the uncertainty of the channel value after projection. Finally, we subtract the R values ​​of the two systems and calculate IBSsub through the formula to represent the similarity between the two systems. We add the R values ​​of the two systems and calculate IBSplus through the formula to represent the difference between the two systems. Finally, in practical applications, we use IBSsub as the similarity value r-NorIBS, and use the difference value IBSplus to normalize IBSsub to obtain r-IBS. The value range of r-IBS is 0 to 1. The loss function mainly consists of two parts: Losssoftmax and LossIBS, where LossIBS is: Loss=Losssoftmax+LossIBS.

7. According to claim 1, after obtaining the IBSsub similarity in step 6, we draw on the idea of ​​the K nearest neighbor algorithm to select k with the largest IBSsub value, where the selection of k is crucial. Compared with the prior art, the present invention has the following beneficial effects: extracting image features through a convolutional neural network, introducing category memory and online learning KNN algorithm, helping the network to extract image features of the CT data set, and helping the network to perform classification prediction, solving the problems of small CT image data volume, large noise, and image feature extraction, and being able to efficiently predict diseases. Experienced doctors are invited to meet the data set standards, and training the network can enable the network prediction to achieve excellent classification results, solving the problems of large workload, slow speed, and inconsistent accuracy of doctors' diagnosis. Now, when there is a new data point that needs to be classified, the model will calculate the distance between this data point and all samples in the window and select the K samples with the closest distance (K is another preset constant). Then, the model will look at the categories of these K samples and select the category with the most occurrences as the predicted category of the new data point, which greatly improves efficiency. Let's use a pseudo code to illustrate the idea of ​​KNN classification: 。 8. According to claim 1, the total loss function in step 7 is a cross entropy loss and an IBS loss, wherein the IBS loss is to constrain the memory maintained by each IGM unit to be constrained by the image of its corresponding category, so that the memory maintained by each IGM is of the same category as itself without crosstalk, and Losssoftmax is a traditional classification loss. Since r-IBS is used as the output probability, the two Losses are essentially consistent in form although their meanings are inconsistent.

Images