A multi-level medical image classification model establishment method based on a convolutional neural network

Abstract

The present application relates to the technical field of medical data analysis, in particular to a multi-level medical image classification model based on convolutional neural network, the present application first distinguishes the main class of images in the data set; after obtaining the main class, the original residual characteristics of the image are analyzed, and the tensor from the fully connected layer is connected; then the output program passes through the CNN attention stack area program block, and then the subclass thereof is distinguished. Through this process, the present application can more effectively give the main class and subclass of the medical condition. By borrowing the deep learning technology and applying the transfer learning technology to image data, the limitations of the current diagnosis system in the Internet of Things medical field, such as low prediction accuracy and complex implementation, are solved. In the field of medical Internet of Things, the model can also provide better support for medical workers when diagnosing medical conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data analysis, in particular to a multi-level medical image classification model establishment method based on a convolutional neural network. BACKGROUND

[0002] Correct disease diagnosis is crucial for timely treatment and recovery of patients, and Internet of Medical Things (IoMT) can collect relevant pathological data of patients through various devices connected by network media for analysis of diseases and timely provision of prognosis, diagnosis, treatment, logistics and other services. With the rapid development of AI technology, modern IoMT is also more intelligent in connecting medical personnel, patients and devices. With the development of Internet of Things (IoT) technology, the medical industry is also gradually realizing intelligentization. Internet of Medical Things (IoMT) collects relevant data through various devices connected by network media for analysis and timely provision of prognosis, diagnosis, treatment, logistics and other services. The successful application of AI technology makes IoMT more intelligent in connecting medical personnel, patients and devices.

[0003] Currently, AI technology has been applied to the diagnosis of diseases, and AI-based CAD system models can diagnose diseases more accurately than professional medical personnel, because humans cannot always accurately interpret medical data such as chest x-rays, CT scans and MRIs, and provide timely treatment in the early stages of diseases. It can also provide remote monitoring and other services to enable medical institutions to obtain real-time medical information about the patient's condition.

[0004] There are many cases of using classified medical images to diagnose diseases at present, such as DL technology and TL technology are often used to classify medical images and diagnose diseases. Among them, the CNN model is mainly used for medical image classification and disease diagnosis, which can directly extract deep features from data. Therefore, in training, the CNN model is used to automatically extract clinical features that are not obvious to humans, which is more forward-looking in medical image analysis, including but not limited to pancreatic segmentation, brain tumor segmentation and carotid intima-media segmentation using computed tomography images. However, the existing diagnosis model has low accuracy, which means that the data for training these methods is insufficient, i.e. more templates are needed to address the data challenge, thereby improving the prediction accuracy of the classification model. SUMMARY

[0005] The technical problem of the present application is that the existing diagnosis model has low accuracy, and the training data is insufficient, which affects the prediction accuracy of the classification model.

[0006] The basic scheme provided by the present application is a multi-level medical image classification model establishment method based on a convolutional neural network, comprising the following steps:

[0007] S1: establishing an initial image classification model;

[0008] S2: improving the classification performance of the image classification model through transfer learning;

[0009] S3: preprocessing and expanding the data;

[0010] S4: establishing a model evaluation standard;

[0011] S5: generating an actual image classification model.

[0012] Further, the S2 includes the steps of:

[0013] S21: pre-training the ResNet50 architecture;

[0014] S22: initializing the same network structure with the upper layer parameters obtained from the pre-trained model, and then using the ResNet50 model to transfer the upper layer parameters from the pre-trained model to the fourth convolutional layer;

[0015] S23: data preprocessing and discriminative feature learning;

[0016] S24: main class prediction, inputting the medical image into the ResNet50 model to extract high-level features;

[0017] S25: sub-class prediction, concatenating the fully connected layer features of the main class prediction with the original ResNet50 feature level and performing feature extraction and recognition;

[0018] S26: transmitting the extracted feature weights to the initial image classification model.

[0019] Further, the S21 includes the steps of:

[0020] S211: pre-training, extracting high-level features from images, and capturing deeper information through convolutional layers with self-attention mechanisms;

[0021] S212: data preprocessing, normalizing and enhancing adjustments such as rotating, flipping, scaling, and blurring the image to improve the quality of the input image;

[0022] S213: fine-tuning, fine-tuning through a small data set to adapt to specific tasks.

[0023] Further, the S24 includes the steps of:

[0024] S241: inputting the medical image into the ResNet50 model to extract high-level features;

[0025] S242: feeding the features into a convolutional layer with self-attention mechanisms and global average pooling to extract discriminative features and reduce spatial dimensions;

[0026] S243: feeding the connected features into a fully connected layer and a normalized exponential function for activation to obtain the probability of each main class.

[0027] Further, the S25 includes the following steps:

[0028] S251: concatenating and parallel feature extraction identification of the fully connected layer features from the main class prediction with the original ResNet50 feature level;

[0029] S252: further feature extraction identification using a convolutional layer with a self-attention mechanism and global average pooling;

[0030] S253: feeding the features into a fully connected layer and a normalized exponential function for activation to obtain the probability of each sub-class within the selected main class.

[0031] Further, the image classification model adopts a two-dimensional CNN model, including a plurality of convolutional layers, pooling layers and fully connected layers.

[0032] Further, during the convolutional forward propagation process, the filter slides on the input, and the activation shown is calculated by calculating the point-by-point product of each element and adding them to obtain the activation of the point.

[0033] Further, the image classification model uses an activation function to introduce nonlinearity, uses a sampling layer to generate summary statistics of the input and reduce the dimension, and the output of the image classification model is flattened into a single vector.

[0034] Further, a data augmentation technique involving random transformation is adopted in the S3 to augment the actual data set.

[0035] Further, the model evaluation criteria in the S4 include four indicators of accuracy, specificity, sensitivity and AUC-ROC.

[0036] The principles and advantages of the present application are:

[0037] The present application first distinguishes the main class of the images in the data set; after obtaining the main class of the image, the original residual feature of the image is analyzed, and the tensor from the fully connected layer is connected; then the output program passes through the CNN attention stack area program block, and then the sub-class of the image is distinguished. Through this process, the present application can more effectively give the main class and sub-class of the medical condition. By borrowing the deep learning technology and applying the transfer learning technology to image data, the limitations of the current diagnostic system in the Internet of Things medical field, such as low prediction accuracy and complex implementation, are solved. In the field of medical Internet of Things, the model can also provide better support for medical workers when diagnosing medical conditions. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A step flowchart of an embodiment of a method for establishing a multi-level medical image classification model based on a convolutional neural network.

[0039] Figure 2 A schematic diagram of an embodiment of a method for establishing a multi-level medical image classification model based on a convolutional neural network.

[0040] Figure 3 A pseudo-code diagram of an embodiment of a method for establishing a multi-level medical image classification model based on a convolutional neural network. DETAILED DESCRIPTION

[0041] The following is further described in detail through specific embodiments:

[0042] The specific implementation process is as follows:

[0043] Embodiment One

[0044] Embodiment One is basically as shown in the accompanying drawings, a method for establishing a multi-level medical image classification model based on a convolutional neural network, comprising the steps of: Figure 1 S1: Establishing an initial image classification model;

[0045] S2: Improving the classification performance of the image classification model through transfer learning;

[0046] S3: Preprocessing and expanding the data;

[0047] S4: Establishing a model evaluation standard;

[0048] S5: Generating an actual image classification model.

[0049] Specifically, in S1, the image classification model of the present scheme is a two-dimensional CNN model, and CNN is a feedforward neural network, known for its application in developing deep learning models. Convolution helps to capture translational invariance, as the filter is independent of position, which can significantly reduce the number of parameters. Convolutional layers, pooling layers, and fully connected layers are the three types of layers in CNN models, which handle tasks such as dimensionality reduction, feature extraction, and classification.

[0050] In the present scheme, during the convolution forward propagation process, the filter slides on the input, and the activation shown is calculated by calculating the point-by-point product of each element and adding them to obtain the activation of that point, so that the sliding filter is realized using convolution. Because this operation is linear, it can be effectively expressed using the dot product. Suppose there is an input x' and a kernel function w, and the convolution (x' x w) at time t (a) can be formulated in all dimensions, as shown in Equation 1:

[0051]

[0052] ​(x' x w)(a) = ∑x'(t)w(a - t)dt (1)

[0053] (where a belongs to Rn and n > 1)

[0054] Since parameters are usually discrete, the discrete convolution process can be defined as equation 2:

[0055] (x x w)(a) = ∑ a x(t)w(t - a) (2)

[0056] In the two-dimensional CNN model in this solution, the two-dimensional image I will input the two-dimensional kernel K, and then the convolution operation can be expressed as equation 3:

[0057] (I x K)(i,j) = ∑ m ∑ n I(m,n)K(i - m,j - n) (3)

[0058] Nonlinearity is introduced by using 2 activation functions (such as sigmoid and ReLU). The sigmoid nonlinear function is suitable for the case where the output needs to be contained in the range of [0, 1]. In addition, this means that the S-shaped function is monotonically increasing, which can cause the problem of gradient disappearance: the neuron saturates, causing the gradient of a given input x to be approximately zero, while θ(x) tends to infinity, making the subsequent optimization operation difficult. On the other hand, compared with the S-shaped nonlinearity, the gradient of ReLU(x) = 1 for x > 0 and ReLU'(x) = 0 for x < 0 will therefore converge faster.

[0059] Therefore, the model uses a sampling layer to generate summary statistics of its input and reduce the dimension. Considering each rectangular point around the data of a given 2-D point of each input feature, the maximum pool is used to return the maximum value. The average pool calculates the average of the data around the point. Considering each rectangular point around the data of a given 2-D point (i,j) of each input feature, the maximum sample is used to calculate the maximum value. The average sample calculates the average of the data around the point.

[0060] In addition, the model uses a fully connected layer with an input size of n and an output size of m. Given an input vector x ∈ R n , the output of a fully connected layer (FC) with an activation function f can be expressed as equation 4:

[0061] FC(x) := f(W x +b) ∈ R m (4)

[0062] (where W x represents matrix multiplication, and the function f is used by component.)

[0063] To achieve the attachment of fully continuous layers on top of the CNN, the output of the 2D CNN model is flattened into a single vector.

[0064] Due to the small amount of data sets related to medical image analysis, it is difficult to extract and identify image features when training the model. Therefore, in order to improve the training process of the 2D CNN model, the pre-trained VGG 19 and ResNet-50 models are also used as the feature extractor of the 2D CNN model. The problem of small data set is overcome by pre-training model technology, and pre-trained model is also widely used for more specific classification, such as classification of breast histopathology images in this embodiment.

[0065] Further, as shown in Figure 2 S2 includes the following steps:

[0066] S21: pre-training ResNet50 architecture;

[0067] S22: initializing the same network structure with the upper layer parameters obtained by the pre-trained model, and then using the ResNet50 model to transfer the upper layer parameters from the pre-trained model to the fourth convolutional layer;

[0068] S23: data preprocessing and discriminative feature learning;

[0069] S24: main class prediction, inputting the medical image into the ResNet50 model to extract high-level features;

[0070] S25: sub-class prediction, concatenating the full connection layer features of the main class prediction with the original ResNet50 feature level and performing feature extraction and identification;

[0071] S26: transmitting the extracted feature weights to the initial image classification model.

[0072] The feature weights extracted from these models are transmitted to the 2D CNN model separately, and fine-tuned using the breast cancer BHI data set. It is effectively trained through the transfer learning process, and the prediction performance of the model is improved.

[0073] Further, S21 includes the following steps:

[0074] S211: pre-training, extracting high-level features from images, and capturing deeper information through convolutional layers with self-attention mechanism;

[0075] S212: data preprocessing, normalizing and enhancing adjustment such as rotating, flipping, scaling and blurring the image to improve the quality of the input image;

[0076] S213: fine-tuning, fine-tuning through a small data set to adapt to specific tasks.

[0077] This approach first pre-trains the ResNet50 architecture with a large image dataset, and then fine-tunes the network with a small dataset of labeled images specific to the target task. The pre-trained network is to extract high-level features from images, capturing deeper information through a series of convolutional layers with self-attention mechanisms. The attention mechanism effectively improves the ability to find relevant graphical content and reduce noise in the network. By pre-training with a large image dataset, this method can be applied to image feature extraction under various conditions; while fine-tuning with a small dataset can better adapt to certain specific tasks; in addition, the data preprocessing step helps to reduce the impact of noise or changes in the input image set.

[0078] In addition to pre-training and fine-tuning, we also perform a data preprocessing step to normalize and enhance the images by rotating, flipping, scaling, and blurring them, to improve the quality of the input images. This step increases the total number of instances in the dataset, which helps improve the accuracy of the classification model.

[0079] After that, the upper layer parameters obtained by the pre-trained model are used to initialize the same network structure, and the ResNet50 model is transferred from the pre-trained model to the upper layer parameters up to the fourth convolutional layer.

[0080] Further, in S23, in order to use a new dataset containing fewer image instances, fine-tuning is required. First, data is pre-processed through data standardization and various transformations to increase the number of variations in the dataset. These transformations include random rotation, horizontal and vertical flipping, random cropping, elastic deformation, color jittering, and adding Gaussian noise. By random rotation, the image is rotated by a random angle to simulate the angle variation of the tissue sample; horizontal and vertical flipping also changes the direction of the tissue sample; random cropping changes the size and position of the tissue in the image; elastic deformation is applied to the image to simulate the deformation of the tissue caused by physical factors such as compression or stretching; color jittering allows the image color to be randomly adjusted to simulate changes in lighting conditions or staining techniques used during sample preparation. Finally, Gaussian noise is added to the image to simulate the noise present in real-world images. Then the pre-processed images are used to fine-tune the model, allowing it to learn task-specific discriminative features.

[0081] Further, in S24, the present solution demonstrates a new method for image main class prediction using deep learning. The method involves an improved ResNet50 architecture that incorporates a self-attention mechanism to capture deeper meaning in the image. We first extract the high-level features from the image using ResNet50, accompanied by a series of convolutional layers with self-attention mechanisms; the self-attention mechanisms improve the ability of the model to filter the image.

[0082] The output of the final convolutional layer is concatenated with the ResNet50 features and passes through a series of fully connected layers to predict the main class of the image. The fully connected layers are activated using a normalized exponential function (softmax) that outputs a probability distribution over the main classes {MC-1, …, MC-N}, where MC represents the main class and N denotes the total number of classes. The present solution demonstrates the effectiveness of the self-attention mechanism in capturing the details of the image.

[0083] Specifically, in the present embodiment, for a breast histopathology dataset with multi-level labels, such as the Breast Cancer Histopathological Image Database (BreakHis), the features extracted by ResNet50 that are fine-tuned on the dataset are passed to a convolutional attention stack block, as shown in Figure 2 The kernel of the block is a Conv-3x3—BatchNorm—ReLU convolutional layer. One layer employs a self-attention mechanism on the output of the previous layer, which can help the model focus on the most important parts of the image. Another similar Conv-3x3—BatchNorm—ReLU convolutional layer is used to extract more complex features from the self-attention output. The last self-attention layer of the block is used to pass its output again to the final Conv-3x3—BatchNorm—ReLU convolutional layer. The last layer of the block is a global average pooling convolutional layer that calculates the average of the feature maps of each convolutional layer before.

[0084] To classify the features from the CNN attention stack block, we use a fully connected layer with 2 output units and a normalized exponential function for activation, which makes the prediction present two results: benign and malignant.

[0085] Compared with the prior art, the present solution has the advantage that by incorporating the self-attention mechanism, the method can capture more detailed patterns in the image, thereby improving the classification accuracy.

[0086] Further, in S25, the present scheme provides a new sub-class prediction method for image classification using deep learning. This step is based on the previously built main class prediction model, and a set of additional CNN attention stack areas and fully connected layers are merged to predict the sub-class of the image. The output procedure of the fully connected layer in the main class prediction is concatenated with the original ResNet50 feature level and passed through the CNN attention stack area and fully connected layer.

[0087] The fully connected layer used for sub-class prediction is activated using a normalized exponential function, which outputs a probability distribution over the sub-classes {SC-1, …, SC-N}, where SC represents a sub-class and N is the total number of sub-classes.

[0088] Specifically, in this embodiment, the sub-classes of the secondary data set are the primary classes, and the benign classes are adenosis, fibroadenoma, phyllodes tumor, and tubular adenoma, while the malignant classes are ductal carcinoma, lobular carcinoma, mucinous carcinoma, and papillary carcinoma. To propagate information from the main class, the output from the main classification fully connected layer is fused by concatenating with the original features from the fourth layer of the ResNet50 network. The resulting output procedure is passed to the CNN attention stack area block for more complex feature extraction. Then a fully connected layer with 4 output units and a normalized exponential function for activation are used to predict the sub-class of the image.

[0089] Compared with the prior art, the present scheme has the advantages that by predicting the main class and sub-class of the image, the present scheme has a more detailed understanding of the deep information of the image content, which can also be used in many other fields.

[0090] Further, in S3, the present scheme uses the BHI image data set, which includes 277524 breast tissue image blocks, 78786 IDC(+) images, and 198738 non-IDC(-) images. These data come from H and E staining of 162 breast tissue pathological patients, and the purpose is to distinguish between cancerous (IDC: invasive ductal carcinoma) and non-IDC images. The above data sample size indicates that the data distribution is uneven, which may cause the problem of model overfitting.

[0091] At the same time, in order to improve the generalization ability of the model, a data augmentation technique involving random transformation is used to expand the actual data set, and additional data is generated to train the model and improve the model performance to support the data augmentation results.

[0092] Finally, the model evaluation criteria established in the present scheme include accuracy, specificity, sensitivity, and AUC-ROC four indicators. The performance of the model is evaluated by the four evaluation indicators, which makes it easier to be included in the Internet of Things healthcare system.

[0093] In addition, the embodiment also uses data enhancement technology, and the combination of the enhanced data and transfer learning also significantly improves the prediction ability of the 2DCNN model. The pseudo code flow of the proposed model is as shown in Figure 3

[0094] In summary, the disease diagnosis method proposed in the scheme uses deep learning technology and transfer learning technology applied to image data, and solves the limitations of the current diagnosis system in the Internet of Things medical field, such as low prediction accuracy and complex implementation. In addition to the residual network used for transfer learning, the application also uses a program block composed of a group of convolutional neural networks and a self-attention stack area. The program block enables the application to further extract and distinguish features. The application first distinguishes the main class of images in the data set; after obtaining the main class, it analyzes the original residual features of the image and connects them with the tensors from the fully connected layer; then the output program passes through the CNN attention stack area program block, and then distinguishes its subclass. Through this process, the application can more effectively give the main class and subclass of the medical condition. In the field of medical Internet of Things, the model provides better support for medical workers when diagnosing medical conditions.

[0095] The above is only an embodiment of the application, and the common knowledge of the specific structure and characteristics in the scheme is not described in detail here. The person skilled in the art knows all the common technical knowledge in the field of the application before the application date or the priority date, can know all the prior art in the field, and has the ability to apply conventional experimental means before that date. The person skilled in the art can improve and implement the scheme based on their own ability under the guidance of this application, and some typical known structures or known methods should not be an obstacle for the person skilled in the art to implement the application. It should be noted that for those skilled in the art, without departing from the structure of the application, a number of modifications and improvements can be made, which should also be considered as the protection scope of the application, and these will not affect the effect and practicality of the application. The protection scope of the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.​

Claims

1. A method for establishing a multi-level medical image classification model based on a convolutional neural network, characterized in that, The method comprises the steps of: S1: establishing an initial image classification model; S2: improving the classification performance of the image classification model through transfer learning; S3: preprocessing and expanding the data; S4: establishing a model evaluation standard; S5: generating an actual image classification model; The S2 comprises the steps of: S21: pre-training the ResNet50 architecture; S22: initializing the same network structure with the upper layer parameters obtained by the pre-trained model, and then using the ResNet50 model to transfer the upper layer parameters from the pre-trained model to the fourth convolutional layer; S23: data preprocessing and discriminative feature learning; S24: main class prediction, inputting the medical image into the ResNet50 model to extract high-level features; S25: sub-class prediction, concatenating the full connection layer features of the main class prediction with the original ResNet50 feature level and performing feature extraction and recognition; S26: transmitting the extracted feature weights to the initial image classification model; The S24 comprises the steps of: S241: inputting the medical image into the ResNet50 model to extract high-level features; S242: feeding the features into the convolutional layer with self-attention mechanism and global average pooling to extract discriminative features and reduce spatial dimension; S243: feeding the connected features into the full connection layer and the normalized exponential function to activate to obtain the probability of each main class; The S25 comprises the steps of: S251: concatenating the full connection layer features from the main class prediction with the original ResNet50 feature level and performing parallel feature extraction and recognition; S252: using the convolutional layer with self-attention mechanism and global average pooling for further feature extraction and recognition; S253: feeding the features into the full connection layer and the normalized exponential function to activate to obtain the probability of each sub-class within the selected main class. 2.The method of claim 1, wherein, The S21 comprises the steps of: S211: pre-training, extracting high-level features from images, and capturing deeper information through convolutional layers with self-attention mechanism; S212: data preprocessing, normalizing and enhancing adjustment of images including rotation, flipping, scaling and blurring to improve the quality of input images; S213: fine-tuning, fine-tuning through a small data set to adapt to specific tasks. 3.The method of claim 2, wherein the method further comprises: training the multi-level medical image classification model based on the convolutional neural network. The image classification model adopts a two-dimensional CNN model, which includes multiple convolutional layers, pooling layers and fully connected layers.

4. The method of claim 3, wherein the method further comprises: In the convolutional forward propagation process of the image classification model, the filter slides on the input, and the activation shown is calculated by calculating the point-by-point product of each element and adding them to obtain the activation of that point.

5. The method of claim 4, wherein the method further comprises: training the multi-level medical image classification model based on the convolutional neural network. The image classification model uses an activation function to introduce nonlinearity, uses a sampling layer to generate summary statistics of the input and reduce dimensionality, and the output of the image classification model is flattened into a single vector. 6.The method of claim 5, wherein the method further comprises: In the S3, a data expansion technique involving random transformation is used to expand the actual data set.

7. The method of claim 6, wherein the method further comprises: training the multi-level medical image classification model based on the convolutional neural network. The model evaluation standard in the S4 includes accuracy, specificity, sensitivity and AUC-ROC four indicators.

Images