An intelligent recognition method for skin lesions based on deep Bayesian distillation network
Through deep Bayesian distillation network and knowledge distillation technology, the problem of CNN model not being able to quantify uncertainty in skin lesions recognition is solved, the recognition accuracy and efficiency are improved, the computing resource occupation is reduced, and more efficient skin lesions are achieved Identify.
Patent Information
- Application Number
- CN202210788978.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-06
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-07-06
AI Technical Summary
Existing CNN models cannot effectively quantify model uncertainty in skin lesions recognition, resulting in insufficient recognition accuracy, and the optimization of hyperparameters or structural methods has problems such as large computing resource occupancy and poor generalization ability.
Using a deep Bayesian distillation network method, the output of the teacher network is fitted by constructing a student network model, combining knowledge distillation technology, the model parameter quantity and training time are optimized, and the accident and cognitive uncertainty are accurately modeled.
It improves the accuracy of skin lesions recognition, reduces model scale and deployment time, reduces training and operation overhead, and improves the generalization ability and efficiency of the model.
Smart Images

Figure CN115147376B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and smart medicine, and in particular to a method for intelligently identifying skin lesions based on a deep Bayesian distillation network. Background Art
[0002] Skin lesions are widely distributed in the population and at a wide age range. 30%-70% of the world's population has skin-related health problems. Malignant skin cancers such as melanoma deteriorate quickly and have a high mortality rate, posing a great threat to human life safety. Early diagnosis and timely treatment of melanoma can increase the survival rate of patients to 90%. Therefore, improving the accuracy and efficiency of skin lesion recognition is of great practical significance. With the help of Convolutional Neural Networks (CNN), automatic recognition and classification of skin diseases based on images can be achieved, improving the efficiency and accuracy of diagnosis.
[0003] The main reason why the current mainstream CNN architecture has insufficient accuracy in skin lesion recognition is that the CNN design is flawed and cannot model the uncertainty associated with prediction. The output of the model is usually the prediction probability. The prediction probability represents the confidence of the model, but the confidence is difficult to objectively reflect the uncertainty of the model. Therefore, the current mainstream CNN model can only give the deviation and confidence of the fit between the test data and the training data, but cannot further improve the prediction accuracy of CNN. Deep Bayes can quantify the uncertainty of the model, so it is necessary to introduce deep Bayes to improve the prediction accuracy of CNN.
[0004] Currently, CNN-assisted identification of skin lesion categories, especially malignant skin lesion identification, combined with Bayesian deep learning, is mainly divided into hyperparameter optimization methods and structure optimization methods.
[0005] Methods for optimizing hyperparameters: Although this type of method alleviates the problem of precision loss caused by the defect of CNN's inability to quantify uncertainty to a certain extent, and uses Bayesian optimization of the model's hyperparameters to improve detection accuracy, it does not fundamentally solve the structural defects of CNN. The method of optimizing parameters has great limitations, requires a lot of computing resources for training, has poor model generalization ability, and has high actual application costs.
[0006] Methods for optimizing structure: This type of method shows that applying deep Bayesian models to CNN can significantly improve model accuracy, optimize CNN structural defects, and improve the model's ability to quantify uncertainty. However, the problems with the above research schemes are that the model is large in scale, the training takes up a lot of computing resources and time, the model deployment takes up a lot of resources, the operation waiting time is long, and the actual application value is low. Summary of the invention
[0007] To solve the above problems, the present invention proposes a skin lesion recognition method based on a deep Bayesian distillation network. Compared with mainstream CNNs that cannot quantify model uncertainty, in order to improve the accuracy of diagnostic results, a deep Bayesian network is constructed. Bayesian deep learning is used to quantify the aleatoric uncertainty and epistemic uncertainty of the model, and a more accurate model of uncertainty is established. Knowledge distillation is introduced to optimize the model parameters and time. A student network model is constructed to fit the output of the teacher network, and the output of the student network is trained with the parameters of the teacher network and the true value labels, thereby reducing the model size and waiting time.
[0008] The intelligent skin lesion recognition method based on a deep Bayesian distillation network according to the present invention includes the following steps:
[0009] Step 1: Collect skin lesion images as samples, label the skin lesion samples, and construct a skin lesion sample set;
[0010] Step 2: Use the skin lesion samples as inputs to construct a deep Bayesian network model based on a deep residual pyramid multi-scale encoding network; characterize the uncertainty of the deep Bayesian network model through a prior distribution, and at the same time model and deduce the uncertainty based on a likelihood function to deduce a posterior distribution;
[0011] Step 3: The deep Bayesian network model includes a teacher model and a student model. Compress the deep Bayesian network model based on knowledge distillation, and construct a student model to fit the output of the teacher model.
[0012] Further, the deep Bayesian network model based on a deep residual pyramid multi-scale encoding network includes an input layer, a convolutional layer, an activation layer, a pooling layer, a bottleneck layer, and an output layer; the uncertainties targeted by the model include aleatoric uncertainty and epistemic uncertainty. For aleatoric uncertainty, the k-fold cross-validation method is used to train the data; for epistemic uncertainty, a deep Bayesian distillation network is used to fit the training data model.
[0013] Further, the specific steps of using the k-fold cross-validation method to train the data are as follows:
[0014] Step 2-1-1: Preprocess the lesion sample D, and at the same time set the number of times k for dividing the cross-validation sub-samples, set the number of loops N and initialize N = 0;
[0015] Step 2-1-2: At the beginning of this round of iteration, when N < k, it means that the cross-validation has not been completed, and iterative cross-validation is performed, otherwise the iteration is terminated;
[0016] Step 2-1-3: Use a deep Bayesian network model. Take the Nth subsample as the validation set and the remaining N-1 samples as the training set for model training, and output the evaluation metric J N ,;
[0017] Step 2-1-4: Iteration count k = k + 1;
[0018] Step 2-1-5: At the end of this iteration, determine whether the iteration stop condition is met. If it is met, stop the iteration; otherwise, repeat steps 2-1-2 to 2-1-4;
[0019] Step 2-1-6: Output the average value of the evaluation results N represents the loop counter, and k represents the cross-validation counter.
[0020] Furthermore, the specific steps for using a deep Bayesian distillation network to fit the training data model are as follows:
[0021] Step 2-2-1: Preprocess the lesion samples D. At the same time, set the number of times k for cross-validation subsample division, set the number of loops N and initialize N = 0;
[0022] Step 2-2-2: At the start of this iteration, when N < k, it means that the cross-validation has not been completed, and perform iterative cross-validation; otherwise, jump out of the iteration;
[0023] Step 2-2-3: Use a deep Bayesian network model. Take the Nth subsample as the validation set and the remaining N-1 samples as the training set for model training, and output the evaluation metric J N , solve the uncertainty score I of the samples. First, take the average of multiple predictions for each sample and then calculate the entropy to obtain H[y|x,D]. First, calculate the entropy of multiple predictions for each sample and then take the average to obtain E θ~p(θ∣D) H[y|x,θ], adjust the training based on I, and update the training model; D is the labeled sample, H is the entropy of the current model, θ is the parameter of the target model to be solved, x is the network input, and y is the network output;
[0024] Step 2-2-4: Iteration count k = k + 1;
[0025] Step 2-2-5: At the end of this iteration, determine whether the iteration stop condition is met. If it is met, stop the iteration; otherwise, repeat step 2-4;
[0026] Step 2-2-6: Output the average value of the evaluation results N represents the loop counter, and k represents the cross-validation counter.
[0027] Furthermore, in step 2-2-3, the labeled sample D is trained by MC dropout to minimize the uncertainty of the parameters under the given labeled sample D, that is, to minimize the conditional entropy H, as shown in formula (1):
[0028] M=argmin D H[θ|D] (1)
[0029] argmin D Indicates the value of variable D when entropy takes the minimum value;
[0030] In actual training, the greedy method is used to find a sample point z that can minimize the entropy of the current model, as shown in formula (2):
[0031]
[0032] argmax x Indicates the value of variable x when entropy reaches its maximum value;
[0033] Since the parameter dimension of the deep Bayesian network model is high, it is difficult to solve formula (2), so it is necessary to solve the equivalent problem of formula (2); formula (2) is equivalent to solving the mutual information I of the model under the conditions of given sample D and newly added sample point x, that is, to find the uncertainty score of the model, that is, formula (3); the maximum uncertainty score between the model prediction and the model parameter is denoted as I m , that is, formula (4):
[0034]
[0035]
[0036] Further, in step 3, the deep Bayesian network model generates class probabilities by using a softmax output layer, which is calculated by each class Logitz i Compare it with other logits to convert it into probability q i , the expression is shown in formula (5):
[0037]
[0038] Among them, z i is the student's logits, N is the total number of labels, logitz i is the result calculated by the current model. Logit is the most widely used classification and evaluation model. T is the temperature. The larger the T value is, the weaker the probability distribution is on the class. Therefore, when training the student model, a larger T is used to make the output of softmax closer to the output of the teacher model. After the training is completed, the value of T is reset to use the student model, as shown in formula (6):
[0039]
[0040] in, It is represented by the value of the teacher model on the i-th category through softmax output at temperature T; v i is the teacher's logits, N is the total number of labels, and T is the temperature. When training the student model, the output of the teacher model is called the weak target, and the true output is called the strong target. During training, the weak target and the strong target are used to train the student model. In this process, two functions are generated: the cross entropy between the student model and the weak target, and the cross entropy between the student model and the strong target. The weighted average of the two objective functions is used as the training target of the student model. The training target is to minimize the weighted average loss, as shown in Equation (7), Equation (8), and Equation (9):
[0041] L KD =αL soft +βL hard (7)
[0042]
[0043]
[0044] Among them, c j is the true value, represents the value of the teacher's softmax output on the jth class at temperature T, represents the value of the student's softmax output on the jth class at temperature T, L represents the loss Loss, L KD is the knowledge distillation loss, L Soft represents the cross entropy loss of the student model trained with the weak target, L Hard represents the cross entropy loss of the student model trained with a strong target; α, β are modulation factors that adjust the parameter ratio during training.
[0045] The beneficial effects of the present invention are as follows: the present invention constructs a deep Bayesian network based on ResNet and FPN (feature pyramid network), uses Bayesian deep learning to quantify the accidental uncertainty and cognitive uncertainty of the model, and performs multiple sampling distribution fitting training data models to more accurately model uncertainty; in order to solve the problems of large scale and long waiting time when the model is deployed, knowledge distillation is introduced to optimize the model parameter quantity and time, and a student network model is constructed to fit the output of the teacher network. The output of the student network is trained with the parameters of the teacher network and the true value labels, thereby reducing the scale and parameter quantity of the model, reducing the overhead of model training and deployment, and reducing the training time and running time. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 This is a schematic diagram of the skin disease intelligent identification system model;
[0047] Figure 2 Schematic diagram of the neural network structure;
[0048] Figure 3 It is a deep Bayesian distillation network model;
[0049] Figure 4 This is a schematic diagram of the training set pictures;
[0050] Figure 5 This is a schematic diagram of the test set picture;
[0051] Figure 6 A diagram of the confusion matrix of the traditional data augmentation ResNet method compared with the literature;
[0052] Figure 7 A diagram of the confusion matrix of the prediction results without a deep Bayesian network;
[0053] Figure 8 A diagram of the confusion matrix of the prediction results of the deep Bayesian distillation network;
[0054] Fig. 9 Schematic diagram of the accuracy index of different lesion categories in the final prediction results;
[0055] Fig.10 Schematic diagram for comparing the prediction accuracy of the method of the present invention with other models. DETAILED DESCRIPTION
[0056] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.
[0057] The present invention discloses a skin lesion recognition method based on a deep Bayesian distillation network, comprising the following steps:
[0058] S1. A skin lesion intelligent recognition model based on deep Bayesian knowledge distillation network was constructed. The model consists of four parts: user, diagnosis cloud, model training and sample source.
[0059] Users: The users considered in the present invention are patients with skin diseases and dermatologists. The present invention provides different levels of services for different patients, which are mainly divided into amateur collection users and professional dermatology collection users. Amateur collection users, such as general patients, can upload photos of skin lesions taken by cameras or mobile phones to the diagnostic cloud to obtain diagnostic results. Similarly, professional dermatology collection users, such as dermatologists, can upload dermatoscope images and obtain results, which will provide diagnostic assistance to dermatologists, thereby making more accurate diagnoses for patients. Amateur users mainly obtain initial diagnosis results after taking pictures of lesions with mobile devices (such as mobile phones). The accuracy of the service diagnosis results will be affected by factors such as lighting, angle and clarity taken by mobile devices, and it is used as a reference for preliminary screening and extraction; professional dermatology collection users obtain services through professional dermatology collection equipment (such as dermatoscopes) in hospitals, and collect and upload through professional dermatoscopes in dermatology to assist doctors in medical extraction, and accurate extraction results can be obtained.
[0060] Diagnostic cloud: The diagnostic cloud of the present invention is composed of cloud services, model storage and data cache, and mainly provides extraction and data storage and transmission services. Among them, the model storage uses the model storage of model training for diagnostic services, and the diagnosis is updated in time according to the model training results to improve the extraction accuracy; the data cache uses the user data cache for model training downloads, which are provided to the sample source for verifying the extraction results and expanding the sample set. When the diagnostic cloud receives the user's diagnostic request, it processes the collected skin lesion images and sends the diagnostic results to the user according to the trained model. At the same time, when the diagnostic cloud completes the diagnostic service, it will store this image with the labeled results to enrich the training data set.
[0061] Model training: Due to the limited capabilities of user acquisition equipment and diagnostic cloud computing, the model training of the present invention is based on the skin lesion intelligent recognition model of the deep Bayesian knowledge distillation network, and the model training requires a lot of computing power and storage. Therefore, the training of the neural network of the present invention is mainly completed on the GPU server. The training data comes from the sample source, and is trained in the training server in combination with the training samples provided by the data provider. The training server is composed of several GPUs and CPUs, and the skin lesion intelligent recognition model based on the deep Bayesian knowledge distillation network of the present invention and the training environment are deployed for model training, and the model parameters of the diagnostic cloud are updated.
[0062] Sample source: In data provision, the samples consist of open-source data provided by ISIC and samples collected by professional dermoscopy collectors in medical institutions and annotated by dermatologists; the dermatologists extract the collection data provided by the collectors and provide the extraction results, annotate the sample data, and provide it as training samples for model training; at the same time, the dermatologists download the extraction results of the diagnostic cloud from model training for result analysis, and the extracted and annotated data is used to expand the training samples; the dermatologists conduct consultations to solve difficult and complicated cases, review the comparison results of the diagnostic cloud, and evaluate the advantages and disadvantages of the model.
[0063] S2. The designed deep Bayesian network of the present invention is based on a deep residual pyramid multi-scale encoding network, including an input layer, a convolutional layer, an activation layer, a pooling layer, and a bottleneck layer, and a dropout layer is added for MC dropout training. When designing, the main uncertainties targeted are aleatoric uncertainty and epistemic uncertainty.
[0064] Aleatoric uncertainty: That is, data uncertainty, such as data loss and noise caused by factors such as hair, lighting, and skin damage. At the same time, there is also a small amount of adversarial noise added to the pictures and data cropping by humans. Although such uncertainties can be easily recognized by the human eye, they will cause different degrees of misjudgment in mainstream CNNs. Adding adversarial noise or introducing an adversarial network does not achieve an ideal effect, and aleatoric uncertainty will still affect the recognition accuracy of CNNs. Therefore, the present invention adopts the k-fold cross-validation method to avoid aleatoric errors.
[0065] The specific steps for training the data using the k-fold cross-validation method are as follows:
[0066] Step 2-1-1: Preprocess the lesion sample D, and at the same time set the number of divisions k of the cross-validation sub-samples, set the number of loops N and initialize N = 0;
[0067] Step 2-1-2: At the beginning of this round of iteration, when N < k, it means that the cross-validation has not been completed, and iterative cross-validation is performed, otherwise the iteration is exited;
[0068] Step 2-1-3: Use the deep Bayesian network model, use the Nth sub-sample as the validation set, and the remaining N - 1 samples as the training set for model training, and output the evaluation index J N ,;
[0069] Step 2-1-4: Iterative count k = k + 1;
[0070] Step 2-1-5: At the end of this round of iteration, judge whether the iteration stop condition is met. If it is met, stop the iteration, otherwise repeat steps 2-1-2 to 2-1-4;
[0071] Step 2-1-6: Output the average value of the evaluation results
[0072] Epistemic uncertainty: also known as model uncertainty, refers to the exponential increase in the number of layers and parameters of the model as the scale of the model continues to increase, resulting in limited training data and an even more limited amount of effective information in the training data, which leads to insufficient model training and difficulty in training the optimal model. The bottom line is that the effective information in the training data can hardly keep up with the substantial growth rate of the model size. Although the difference is not large when verifying the training set, the error of the test set is extremely large. If there is enough training data, the problems caused by epistemic uncertainty can be easily solved, but considering that the data set distribution of the present invention has a sample imbalance problem, and the data scale is far from sufficient, it is difficult to solve epistemic uncertainty by expanding the data sample.
[0073] In order to solve this problem, the present invention adopts a deep Bayesian distillation network to fit the training data model, and fits the training data model through multiple sampling distributions, such as Figure 2 As shown, uncertainty is modeled more accurately. Specifically, the model of the present invention describes model uncertainty through prior distribution, and at the same time models and infers data uncertainty based on likelihood function to derive posterior distribution.
[0074] During training, in order to obtain the bias and distribution of the network, it is necessary to obtain the posterior probability, that is, to obtain all the value spaces of the weights. The value space of a single weight of a neural network is a real number set R, so the space composed of weights between different layers is very complex and difficult to integrate. At the same time, the present invention adopts a multi-scale model. The difference between the present invention and the mainstream CNN is that the CNN of the present invention is based on ResNet. On the basis of ResNet, three 3*3 convolution kernels are used to replace the 7*7 convolution kernel, and two 3*3 convolution kernels are used to replace the 5*5 convolution kernel, so that the fully connected network of 5*5 convolution slides in the 5*5 area. Mainstream CNN is a single-scale network, and the present invention divides the bottleneck layer into a multi-scale coding network. Specifically, the bottleneck layer is composed of multiple bottleneck layers 1 and bottleneck layers 2, and bottleneck layers 1 and bottleneck layers 2 are composed of convolution blocks, batch normalization and linear rectification. Training and fitting bias and distribution based on the deep residual pyramid multi-scale coding network further increases the complexity of the weight composition space between different layers, so the present invention uses MC dropout to solve this problem. By adding a dropout layer, turning on dropout during testing and averaging the forward propagation calculations for the same output multiple times to fit the posterior distribution, a more accurate estimate of the prediction confidence is derived through multiple sampling, thereby improving the training effect of the network.
[0075] The MC dropout used in the present invention provides an explanation of the regularization technique of dropout. Variational inference is a Bayesian method for estimating the posterior distribution using an arbitrary distribution; but dropout is a regularization method for a neural network, in which neurons are randomly turned on or off during training to prevent the network from relying on a specific neuron. This is the core idea of MCdropout: dropout can be used to perform variational inference, where the variational distribution is a Bernoulli distribution with states of on and off. MC means that dropout samples in a manner similar to Monte Carlo. In the present invention, MCdropout is used to transform the traditional network into a Bayesian network, so that the research problem is equivalent to sampling from a Bernoulli distribution, providing a measure of model uncertainty, that is, the continuity of predictions made by multiple samplings. The present invention trains the labeled samples D through MC dropout, aiming to minimize the possible assumptions of the model parameters θ, that is, to minimize the uncertainty of the parameters under the given labeled samples D, that is, to minimize the conditional entropy H, that is, formula (1):
[0076] M=argmin D H[θ|D] (1)
[0077] In actual training, a greedy method is needed to find a sample point z that can minimize the entropy of the current model, that is, formula (2):
[0078]
[0079] Since the parameter dimension of the model of the present invention is high, it is difficult to solve formula (2), so it is necessary to solve the equivalent problem of formula (2). Formula (2) is equivalent to solving the mutual information of the model under the conditions of given sample D and new sample point x.
[81] (MutualInformation, MI), denoted as I, that is, to find the uncertainty score of the model, that is, formula (3). The maximum uncertainty score between the model prediction and the model parameters is denoted as I m , that is, formula (4):
[0080]
[0081]
[0082] The specific steps of using the deep Bayesian distillation network to fit the training data model are:
[0083] Step 2-2-1, pre-process the lesion sample D, set the number of cross-validation sub-sample divisions k, set the number of cycles N and initialize N = 0;
[0084] Step 2-2-2: At the beginning of this round of iteration, when N < k, it means that the cross-validation has not been completed, and iterative cross-validation is performed; otherwise, the iteration is terminated.
[0085] Step 2-2-3: Adopt a deep Bayesian network model. Use the Nth subsample as the validation set and the remaining N - 1 samples as the training set for model training, and output the evaluation metric J. N , solve the uncertainty score I of the sample. First, take the average of multiple predictions for each sample and then calculate the entropy to obtain H[y|x,D]. First, calculate the entropy of multiple predictions for each sample and then take the average to obtain E. θ~p(θ∣D) H[y|x,θ], adjust the training based on I, and update the training model.
[0086] Step 2-2-4: Iteration count k = k + 1.
[0087] Step 2-2-5: At the end of this round of iteration, determine whether the iteration stop condition is met. If it is met, stop the iteration; otherwise, repeat Step 2-4.
[0088] Step 2-2-6: Output the average value of the evaluation results.
[0089] S3. The model compression method of the present invention is designed based on knowledge distillation. During the model compression process, two models are involved, namely the teacher model and the student model. The teacher model is a pre-trained large model with high prediction accuracy but large scale and high resource consumption; the student model is a small model to be trained, which is the final model with small scale and low resource consumption that is expected to be obtained. Knowledge distillation deploys useful knowledge such as the parameters of the teacher model to the student model to be trained through training and learning, and at the same time transfers the generalization ability of the teacher model to the student model. This is achieved by using the class probabilities generated by the teacher model as the training target for the student model. Distillation extracts knowledge from a large model with good generalization ability and transfers the knowledge to a small model. The student model learns useful knowledge from the teacher model to achieve performance approaching that of the teacher model. Training the student model using the output of the teacher model can transfer the knowledge of the teacher model to the student model. The output of the teacher model is called the soft target, while the expected data is called the hard target. The student model is trained using both the soft target and the hard target, and the influence of the teacher model on the student model is adjusted by adjusting the proportion of the soft target and the hard target. The loss of training the student model using the soft target is called the distillation loss, and the loss of training the student model using the expected data is called the student loss. During the training process, the sum of the distillation loss and the student loss is used as the total loss of the student model, and the training objective is to minimize the total loss of the student model. Knowledge distillation has the following advantages: The student model can learn the knowledge of the teacher model, and with a relatively small-scale model, it can obtain relatively good model performance; training the student model using both the soft target and the hard target improves the generalization ability of the student model.
[0090] The neural network of the present invention generates class probabilities by using a softmax output layer, which is calculated by each class The logitz i Compare it with other logits to convert it into probability q i , the expression is shown in formula (5):
[0091]
[0092] Among them, z i is the student's logits, N is the total number of labels, and T is the temperature, which is usually set to 1. The larger the T value, the weaker the probability distribution on the class can be. Therefore, when training the student model, a larger T can be used to make the output of softmax closer to the output of the teacher model. After training, the value of T is reset (such as T = 1) to use the student model, as shown in formula (6).
[0093]
[0094] Among them, v i is the teacher's logits, N is the total number of labels, and T is the temperature, which is usually set to 1. When training the student model, the output of the teacher model is called the weak target, and the true output is called the strong target. During training, the student model is trained together with the weak target and the strong target. In this process, two functions are generated, the cross entropy between the student model and the weak target, and the cross entropy between the student model and the strong target. The weighted average of the two objective functions is used as the training target of the student model. The training target is to minimize the weighted average loss, as shown in Equation (7), Equation (8), and Equation (9):
[0095] L KD =αL soft +βL hard (7)
[0096]
[0097]
[0098] Among them, c j is the true value, represents the value of the teacher's softmax output on the jth class at temperature T, Represents the value of the student's softmax output on the jth class at temperature T.
[0099] like Figure 3As shown, the present invention uses a deep Bayesian network as a teacher network to construct a deep Bayesian distillation network. The training data is input into the teacher network and the student network at the same time. The student network is designed separately, and its structure is the same as the teacher network, but the scale and number of layers are less than the teacher network, thereby reducing the cost of model deployment and training time, and improving the efficiency of network training. The teacher output is obtained by training the teacher network through training data, and the output of the student training is combined with the real label and the output of the teacher network for knowledge distillation.
[0100] The present invention sets L hard To prevent the teacher model from making mistakes, L hard This can prevent the teacher’s mistakes from being passed on to students. soft and L hard The purpose of the weight is to adjust its impact on the model effect, because experience shows that L hard Smaller weights often bring better results. soft The gradient contribution is approximately 1 / T 2 Therefore, in order to ensure that the gradient contributions of the two losses are equal, the present invention sets L soft With T 2 Multiply. The most primitive softmax function is T=1. When T<1, the probability distribution is denser; when T tends to 0, the output value becomes hard weight-oriented; when T>1, the probability distribution will be smoother. As T increases, the probability distribution entropy will increase accordingly; when T tends to infinity, the softmax result is evenly distributed. The degree of change of T determines how much attention the student model pays to the negative category. When the temperature is very low, the model does not pay much attention to the negative category, especially those negative categories that are smaller than the mean; when the temperature is very high, the model pays more attention to the negative category. In fact, the negative category carries more information, especially the negative category greater than the mean. The choice of T is closely related to the size of the student model, and experience shows that when the student model is relatively small, T should not be set too large, because the student model is unable to learn all the knowledge of the teacher model, and some negative category information can be ignored. Therefore, T of the present invention is based on experimental selection, and the value of T is adjusted through multiple samplings, thereby reducing the accuracy loss of the model while improving the degree of knowledge distillation optimization.
[0101] The sample set of the present invention is a three-channel dermoscopic image of the RGB color system. The sample set is divided into a training set and a test set. It is composed of an ISIC open source skin disease sample set and samples collected by professional dermoscopic collectors in medical institutions and annotated by dermatologists. Each lesion image of the sample in this article includes a primary lesion, excluding smaller secondary lesions or other pigmented areas. The distribution of lesion samples is consistent with the actual situation, that is, there are more benign lesions than malignant lesions, but there are more types of malignant lesions. The training set of the present invention consists of 12,609 dermoscopic lesion images. The main lesion types are melanoma (MEL), melanocytic nevus (NV), basal cell carcinoma (BCC), actinic keratosis (AKIEC), benign keratosis (BKL), dermatofibroma (DF) and vascular lesions (VASC), etc. The training set is selected from ISIC2018 and desensitized sampling pictures of medical institutions, such as Figure 4 and Figure 5 As shown, the pictures do not contain any personal information of the patients, thereby protecting the privacy of the patients. At the same time, each picture corresponds to a label in csv format, and the label of the training set picture is MEL. In order to avoid the influence of factors such as hair, bubbles, and light on the experimental results in the collected training set samples, the present invention uses algorithms such as denoising and color constancy to pre-process the pictures, and at the same time translates, flips, stretches, and rotates the sample set to expand the training set to 20,000 pictures.
[0102] In order to further microscopically judge the model, let T be the correct judgment of the classifier (true), F be the wrong judgment of the classifier (false), P represent the disease result as positive, N represent the disease result as negative, then TP is true positive, TN is true negative, FP is false positive, and FN is false negative. The present invention adopts sensitivity as the standard for judging the accuracy of diagnosis, sensitivity = TP / P, and introduces specificity and precision to assist judgment. Specifity = TN / N, precision = TP / (TP+FP).
[0103] The output result after diagnosis by the model of the present invention is shown in the figure, where Figure 6The confusion matrix of Budhiman using traditional data augmentation ResNet (existing literature A. Budhiman, S. Suyanto, A. Arifianto. Melanoma cancer classification using ResNet with data augmentation [C]. Proceedings of International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), 2019: 17-20.), Figure 7 is the confusion matrix of the prediction result without deep Bayesian network of the present invention, Figure 8 It is the confusion matrix of the prediction results of the deep Bayesian distillation network of the present invention.
[0104] Figure 8 It is the accuracy index of the final prediction result of the present invention for different lesion categories. As can be seen from the figure, the recognition accuracy of NV and VASC and the specificity of all classification results of the present invention have reached a high level. Fig. 9 As shown in Table 1, although the sensitivity of AKIEC and DF is slightly lower due to sample imbalance, the average sensitivity, specificity and accuracy of the proposed model are improved compared with the mainstream model, and the classification results of each category have achieved higher sensitivity, specificity and accuracy.
[0105] Table 1 Accuracy index of different lesion categories in the final prediction results of the present invention
[0106]
[0107] like Fig.10 As shown in Table 2, the average prediction result of the deep Bayesian model of the present invention is better than that of the mainstream solution. The reason is that the mainstream solution and the deep residual pyramid model cannot model the uncertainty related to the prediction. The model output is the prediction probability, which is difficult to objectively reflect the uncertainty of the model. It can only give the deviation and confidence of the test data and the training data fitting. The deep Bayesian network of the present invention fits the training data model through multiple sampling distributions to model the uncertainty more accurately, combines the Bayesian uncertainty modeling with the deep residual pyramid model fitting ability, and gives play to their complementary advantages. When predicting, the results of the posterior distribution prediction model are integrated to improve the prediction ability of the model, so it has better prediction results. Compared with the models in other papers, the prediction results of the deep Bayesian distillation network model of the present invention are the highest in sensitivity, specificity and accuracy.
[0108] Table 2 Comparison of the prediction accuracy of the present invention and other models
[0109]
[0110] The deep Bayesian network of the present invention can fit the model of training data, fit the training data model through multiple sampling distributions, and model uncertainty more accurately. Specifically, the model of the present invention characterizes model uncertainty through prior distribution, and at the same time models and infers data uncertainty based on the likelihood function to derive the posterior distribution. The present invention combines Bayesian uncertainty modeling with the function fitting ability of traditional CNN to make up for the limitations of traditional CNN, give play to their complementary advantages, solve the overfitting problem, and integrate the results of the posterior distribution prediction model during prediction to improve the prediction ability of the model. The present invention uses MC dropout to solve this problem, adds a dropout layer, turns on dropout during testing, and averages the forward propagation calculations for the same output multiple times to fit the posterior distribution. A more accurate estimate of the prediction confidence is derived through multiple sampling, thereby improving the training effect of the network.
[0111] The present invention adopts a deep Bayesian network as a teacher network to construct a deep Bayesian distillation network, and the training data is simultaneously input into the teacher network and the student network. The student network is designed separately, and its structure is the same as that of the teacher network, but its scale and number of layers are less than that of the teacher network, thereby reducing the overhead of model deployment and training time, and improving the efficiency of network training. The teacher output is obtained by training the teacher network through training data, and the output of the student training is combined with the real label and the output of the teacher network for knowledge distillation. As shown in Table 3, knowledge distillation deploys useful knowledge such as the parameters of the teacher model to the student model to be trained through training and learning, and at the same time transfers the generalization ability of the teacher model to the student model, and is achieved by using the class probability generated by the teacher model as the training target of the student model. Table 3 shows the comparison of the parameter amount and running time of the deep Bayesian distillation network model of the present invention with the mainstream solution, which effectively simplifies the parameter amount and time when the model is running, and reduces the degree of reduction in model accuracy while compressing. Table 4 shows the model comparison of the deep Bayesian distillation network of the present invention and the knowledge-free distillation network of the present invention.
[0112] Table 3 Comparison of parameters and running time between the deep Bayesian distillation network of the present invention and the mainstream solution
[0113]
[0114] Table 4 Parameter changes after knowledge distillation of the deep Bayesian distillation network of the present invention
[0115]
[0116] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.
Claims
1. A method for intelligent recognition of skin lesions based on deep Bayesian distillation network, characterized in that: It includes the following steps: Step 1: Collect skin lesion images as samples, annotate the skin lesion samples, and construct a skin lesion sample set; Step 2: Use the skin lesion samples as inputs to construct a deep Bayesian network model based on a deep residual pyramid multi-scale encoding network; characterize the uncertainty of the deep Bayesian network model through a prior distribution, and at the same time model and deduce the uncertainty based on a likelihood function to deduce a posterior distribution; The uncertainty targeted by the deep Bayesian network model based on the deep residual pyramid multi-scale encoding network is epistemic uncertainty. A deep Bayesian distillation network is used to fit the training data model. The specific steps are as follows: Step 2-2-1: Preprocess the lesion sample D, and at the same time set the number of times k for dividing the cross-validation sub-samples, set the number of loops N and initialize N = 0; Step 2-2-2: At the beginning of this round of iteration, when N < k, it means that the cross-validation has not been completed, and iterative cross-validation is performed, otherwise the iteration is terminated; Step 2-2-3: Use the deep Bayesian network model, take the Nth subsample as the validation set, and the remaining N-1 samples as the training set, perform model training, and output the evaluation index J N , solve the uncertainty fraction I of the sample, first average the multiple predictions of each sample and then calculate the entropy to obtain H[y|x,D], first calculate the entropy and then average the multiple predictions of each sample to obtain E θ~p(θ∣D) H[y|x,θ], adjusts the training based on I and updates the training model; D is the labeled sample, H is the current model entropy, θ is the parameter to solve the target model, x is the network input, and y is the network output; Step 2-2-4: Iterative count k = k + 1; Step 2-2-5: At the end of this round of iteration, determine whether the iteration stop condition is satisfied. If it is satisfied, stop the iteration, otherwise repeat Step 2-4; Step 2-2-6: Output the average value of the evaluation results N represents the loop counter, k represents the cross-validation counter; Step 3: The deep Bayesian network model includes a teacher model and a student model. Compress the deep Bayesian network model based on knowledge distillation, and construct a student model to fit the output of the teacher model.
2. According to claim 1, a method for intelligent recognition of skin lesions based on deep Bayesian distillation network is characterized in that: The deep Bayesian network model based on the deep residual pyramid multi-scale encoding network includes an input layer, a convolutional layer, an activation layer, a pooling layer, a bottleneck layer, and an output layer; the uncertainty targeted by the model also includes aleatoric uncertainty. For aleatoric uncertainty, the k-fold cross-validation method is used to train the data.
3. The method for intelligent recognition of skin lesions based on deep Bayesian distillation network according to claim 2 is characterized in that: The specific steps for training the data using the k-fold cross-validation method are as follows: Step 2-1-1: Preprocess the lesion sample D, and at the same time set the number of times k for dividing the cross-validation sub-samples, set the number of loops N and initialize N = 0; Step 2-1-2: At the beginning of this round of iteration, when N < k, it means that the cross-validation has not been completed, and iterative cross-validation is performed, otherwise the iteration is terminated; Step 2-1-3: Use the deep Bayesian network model, take the Nth subsample as the validation set, and the remaining N-1 samples as the training set, perform model training, and output the evaluation index J N ; Step 2-1-4: Iterative count k = k + 1; Step 2-1-5: At the end of this round of iteration, determine whether the iteration stop condition is satisfied. If it is satisfied, stop the iteration, otherwise repeat Steps 2-1-2 to 2-1-4; Step 2-1-6: Output the average value of the evaluation results N represents the loop counter and k represents the cross validation counter.
4. The method for intelligent recognition of skin lesions based on deep Bayesian distillation network according to claim 1, characterized in that: In Step 2-2-3, the labeled sample D is trained through MC dropout to minimize the uncertainty of the parameters under the given labeled sample D, that is, to minimize the conditional entropy H, as shown in Equation (1): M=argmin D H[θ|D] (1) argmin D Indicates the value of variable D when entropy takes the minimum value; During actual training, the greedy method is used to solve, and a sample point z that can minimize the entropy of the current model to the greatest extent is obtained, as shown in Equation (2): z=argmax x [H[θ|D]-E y~p(y|x,D) [H[θ|D,x,y]]] (2) argmax x Indicates the value of variable x when entropy reaches its maximum value; Since the parameter dimension of the deep Bayesian network model is high, it is difficult to solve formula (2), so it is necessary to solve the equivalent problem of formula (2); formula (2) is equivalent to solving the mutual information I of the model under the conditions of given sample D and newly added sample point x, that is, to find the uncertainty score of the model, that is, formula (3); the maximum uncertainty score between the model prediction and the model parameter is denoted as I m , that is, formula (4): I=H[y|x,D]-E θ~p(θ∣D) H[y|x,θ] (3) I m =argmax x [H[y|x,D]-E θ~p(θ∣D) H[y|x,θ]] (4)。 5. The method for intelligent recognition of skin lesions based on deep Bayesian distillation network according to claim 1, characterized in that: In step 3, the deep Bayesian network model generates class probabilities by using a softmax output layer, which converts the logitz i Compare it with other logits to convert it into probability q i , the expression is shown in formula (5): Among them, z i is the student's logits, N is the total number of labels, logitz i The result calculated by the current model, logit It is the most widely used classification evaluation model. T is the temperature; the larger the value of T, the weaker the probability distribution generated on the class, Therefore, when training the student model, a larger T is used to make the output of softmax closer to the output of the teacher model; after the training is completed, the value of T is reset to use the student model, as shown in Equation (6): in, It is represented by the value of the teacher model on the i-th category through softmax output at temperature T; v i is the teacher's logits, N is the total number of labels, and T is the temperature. When training the student model, the output of the teacher model is called the weak target, and the true output is called the strong target. During training, the weak target and the strong target are used to train the student model. In this process, two functions are generated: the cross entropy between the student model and the weak target, and the cross entropy between the student model and the strong target. The weighted average of the two objective functions is used as the training target of the student model. The training target is to minimize the weighted average loss, as shown in Equation (7), Equation (8), and Equation (9): L KD =αL soft +βL hard (7) Among them, c j is the true value, represents the value of the teacher's softmax output on the jth class at temperature T, represents the value of the student's softmax output on the jth class at temperature T, L represents the loss Loss, L KD is the knowledge distillation loss, L Soft represents the cross entropy loss of the student model trained with the weak target, L Hard represents the cross entropy loss of the student model trained with a strong target; α, β are modulation factors that adjust the parameter ratio during training.
