A method and system for classifying and determining features of skin condition images

Abstract

The present invention discloses a method and system for class recognition and feature determination of skin state images, which relates to the field of image analysis, and includes: obtaining a skin state image to be identified; inputting the skin state image to be identified into a class recognition model to identify the class of the skin state image to be identified; the class recognition model is obtained by training a convolutional neural network based on historical skin state images; when the class of the skin state image to be identified is an unhealthy state, the Shapley value method is used to determine the features based on which the recognition result is based. The present invention can determine the features based on which the recognition result is based, and improves the recognition accuracy and interpretability of the skin state image class.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method and system for category recognition and feature determination of skin condition images. Background Art

[0002] Intelligent assisted diagnosis and treatment of skin diseases is an interdisciplinary field where digitalization, telemedicine and informatics are increasingly interacting, and skin condition images need to be classified. Compared with traditional statistical and machine learning methods, deep learning models can better learn nonlinear representations from complex high-dimensional data and capture complex structural information in the data. They have achieved excellent performance in image classification, semantic segmentation, target detection and target positioning. However, due to the complexity and low transparency of deep learning, the topic of "black box" has always been a controversy in the application of deep learning in the medical field. That is, deep learning can only identify the category of skin condition in skin condition images, but it is not interpretable, that is, there is no feature based on which the category of skin condition images is determined (the category includes a variety of skin disease states in healthy and unhealthy states), which makes it difficult for users to be convinced. For high-precision models, if it is impossible to determine whether the model has captured the "correct features" of skin disease images, it is difficult to be truly recognized. It can be said that the interpretability and accuracy of the model are equally important. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for category recognition and feature determination of skin condition images, which can determine the features based on which skin condition categories are identified, thereby improving the recognition accuracy and interpretability of skin condition image categories.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for classifying and determining features of a skin condition image, comprising:

[0006] Obtain a skin state image to be identified;

[0007] Inputting the skin state image to be identified into a category recognition model to identify the category of the skin state image to be identified; the category recognition model is obtained by training a convolutional neural network based on historical skin state images;

[0008] When the category of the skin state image to be identified is an unhealthy state, the features based on which the identification result is based are determined using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified.

[0009] Optionally, before obtaining the skin state map to be identified, the method further includes:

[0010] Get multiple historical skin status pictures;

[0011] Labeling the types of the plurality of historical skin condition images to obtain a plurality of historical skin condition labeled images as a training set;

[0012] The convolutional neural network is trained with the training set as input and the types of multiple pictures in the training set as output to obtain a category recognition model.

[0013] Optionally, the step of taking the training set as input and the types of the multiple pictures in the training set as output, training the convolutional neural network to obtain a category recognition model specifically includes:

[0014] Construct a convolutional neural network as a 0th-order category recognition model;

[0015] Let the number of iterations m be equal to 1;

[0016] Inputting the training set into an m-1th order category recognition model to obtain an mth order category recognition model and a plurality of historical skin state pseudo-labeled images;

[0017] According to the training set and the plurality of historical skin state pseudo-labeled images, the formula Determine the loss function for the mth iteration;

[0018] Determine whether the loss function of the mth iteration is less than the loss function threshold and obtain a determination result;

[0019] If the judgment result is no, the training set is updated according to the multiple historical skin state pseudo-labeled images, the value of the iteration number m is increased by 1, and the step of "inputting the training set into the m-1th order category recognition model to obtain the mth order category recognition model and multiple historical skin state pseudo-labeled images" is returned;

[0020] If the judgment result is yes, determining the m-th order category recognition model as the category recognition model;

[0021] Where Loss is the loss function; n is the number of samples; x is the image in the training set at the mth iteration; y is the annotation result of the image in the training set; a is the recognition result of the m-1th order category recognition model.

[0022] Optionally, updating the training set according to the plurality of historical skin state pseudo-annotated images specifically includes:

[0023] Determine a plurality of historical skin state pseudo-annotated images with prediction accuracy greater than a prediction accuracy threshold as a backup set;

[0024] Determine the number of images corresponding to each skin condition type in the training set;

[0025] Arrange the various skin condition categories in ascending order according to the number of corresponding pictures;

[0026] Determine the first preset number of skin condition types as the skin condition types to be updated;

[0027] A plurality of historical skin state pseudo-annotated images corresponding to each type of skin state to be updated are selected from the spare set and added to the training set.

[0028] Optionally, when the category of the skin state image to be identified is an unhealthy state, the features based on which the identification result is based are determined using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified, specifically including:

[0029] Construct weighted linear regression models and optimization functions;

[0030] According to the pixel value of each pixel point on the skin state map to be identified, the weighted linear regression model is optimized using the optimization function, and the estimation coefficient group corresponding to the minimum of the optimization function is determined to be the Shapley value of each pixel point on the skin state map to be identified;

[0031] Determine the pixel values ​​of pixels whose Shapley values ​​are not equal to 0 as the features based on the recognition results;

[0032] in,

[0033] The weighted linear regression model is:

[0034] The optimization function is

[0035] Among them, g(.) is the weighted linear regression model, that is, the explanatory model; z' is the indicative function; z'∈{0,1} M ; M is the size of the largest coalition; φ0 is the average of the Shapley values; φ j is the Shapley value of the jth pixel; z' j is the indicative function value of the jth pixel, z' j 1 means the pixel value of the jth pixel is greater than 0, z' j 0 means that the pixel value of the jth pixel is equal to 0; L(.) is the optimization function, and the estimated coefficient group of the optimization function is the corresponding multiple φ when j in g(.) is 1 to M j f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' to the pixel value of the pixel point x to be explained; π x (.) is the kernel function,

[0036] A system for classifying and determining features of skin condition images, comprising:

[0037] A module for obtaining a skin state image to be identified, used for obtaining a skin state image to be identified;

[0038] An image category recognition module, used for inputting the skin state image to be recognized into a category recognition model to recognize the category of the skin state image to be recognized; the category recognition model is obtained by training a convolutional neural network based on historical skin state images;

[0039] The feature determination module is used to determine the feature based on the recognition result by using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified when the category of the skin state image to be identified is an unhealthy state.

[0040] Optionally, the system further includes:

[0041] A historical skin state image acquisition module is used to acquire multiple historical skin state images;

[0042] A labeling module, used for labeling the types of the plurality of historical skin state images to obtain a plurality of historical skin state labeling images as a training set;

[0043] The category recognition model construction module is used to train the convolutional neural network with the training set as input and the types of multiple pictures in the training set as output to obtain a category recognition model.

[0044] Optionally, the category recognition model building module specifically includes:

[0045] A 0th order category recognition model building unit, used to build a convolutional neural network as a 0th order category recognition model;

[0046] An iteration number assignment unit, used to set the iteration number m to be equal to 1;

[0047] A training unit, used for inputting the training set into an m-1th order category recognition model to obtain an mth order category recognition model and a plurality of historical skin state pseudo-annotated images;

[0048] A loss function calculation unit is used to calculate the loss function according to the training set and the plurality of historical skin state pseudo-labeled images using the formula Determine the loss function for the mth iteration;

[0049] A judgment unit, used to judge whether the loss function of the mth iteration is less than the loss function threshold, and obtain a judgment result; if the judgment result is no, the training set updating unit is called; if the judgment result is yes, the category recognition model determination unit is called;

[0050] A training set updating unit, used for updating the training set according to the plurality of historical skin state pseudo-annotated images, increasing the value of the number of iterations m by 1, and calling the training unit;

[0051] A category recognition model determination unit, used to determine the m-th order category recognition model as the category recognition model;

[0052] Where Loss is the loss function; n is the number of samples; x is the image in the training set at the mth iteration; y is the annotation result of the image in the training set; a is the recognition result of the m-1th order category recognition model.

[0053] Optionally, the training set updating unit specifically includes:

[0054] A reserve set determination subunit, used for determining a plurality of historical skin state pseudo-annotated images with prediction accuracy greater than a prediction accuracy threshold as a reserve set;

[0055] A quantity determination subunit, used to determine the number of pictures corresponding to each skin condition type in the training set;

[0056] A sorting subunit, for sorting the multiple skin condition categories in ascending order according to the number of corresponding pictures;

[0057] A skin condition category determination subunit to be updated, used to determine a first preset number of skin condition categories as skin condition categories to be updated;

[0058] The training set updating subunit is used to select a plurality of historical skin state pseudo-annotated images corresponding to each type of skin state to be updated from the standby set and add them to the training set.

[0059] Optionally, the feature determination module specifically includes:

[0060] Model building unit, used to build weighted linear regression model and optimization function;

[0061] A Shapley value determination unit is used to optimize the weighted linear regression model using the optimization function according to the pixel value of each pixel point on the skin state map to be identified, and determine that the estimation coefficient group corresponding to the minimum of the optimization function is the Shapley value of each pixel point on the skin state map to be identified;

[0062] A recognition result based feature determination unit, used for determining the pixel value of the pixel point whose Shapley value is not equal to 0 as the feature based on the recognition result;

[0063] in,

[0064] The weighted linear regression model is:

[0065] The optimization function is

[0066] Among them, g(.) is the weighted linear regression model, that is, the explanatory model; z' is the indicative function; z'∈{0,1} M ; M is the size of the largest coalition; φ0 is the average of the Shapley values; φ j is the Shapley value of the jth pixel; z' j is the indicative function value of the jth pixel, z' j 1 means the pixel value of the jth pixel is greater than 0, z' j 0 means that the pixel value of the jth pixel is equal to 0; L(.) is the optimization function, and the estimated coefficient group of the optimization function is the corresponding multiple φ when j in g(.) is 1 to M j f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' to the pixel value of the pixel point x to be explained; π x (.) is the kernel function,

[0067] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0068] In the present invention, a category recognition model is obtained by training a convolutional neural network to identify skin condition categories, and when the category of the skin condition image to be identified is an unhealthy state, the Shapley value method is used to determine the features based on the recognition result, thereby improving the recognition accuracy and interpretability of the skin condition image category. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0070] Figure 1 is a flow chart of a method for class recognition and feature determination of a skin condition image in an embodiment of the present invention;

[0071] Figure 2 It is a schematic diagram of the overall framework of the technical solution in the embodiment of the present invention;

[0072] Figure 3 This is a design diagram of the intelligent diagnosis system for skin diseases in an embodiment of the present invention;

[0073] Figure 4 It is a flow chart of the algorithm in the embodiment of the present invention;

[0074] Figure 5 Schematic diagram of the ROC curve and AUC index of the model in the embodiment of the present invention;

[0075] Figure 6 An example diagram of the SHAP value interpretation of a skin disease image in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] The purpose of the present invention is to provide a method and system for category recognition and feature determination of skin condition images, which can determine the features based on which skin condition categories are identified, thereby improving the recognition accuracy and interpretability of skin condition image categories.

[0078] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] The application of traditional deep learning models in intelligent auxiliary diagnosis of skin diseases still has the following problems: (1) The lack of sufficient clinical data samples limits the widespread application of deep learning in skin disease diagnosis. Although a large amount of unlabeled data can be obtained from public skin disease datasets and websites, the labeling of skin condition types requires professional medical knowledge, which is difficult and costly. (2) The imbalance of sample categories causes the model's prediction results to be biased towards the majority class, resulting in low recognition accuracy of the model. (3) The high-precision prediction of deep learning models depends on complex network models, and the large number of parameters makes the model inefficient during training and use. Figure 1 The present invention provides a method for classifying and determining features of a skin condition image, comprising:

[0080] Step 101: Obtain a skin state image to be identified;

[0081] Step 102: inputting the skin state image to be identified into a category recognition model to identify the category of the skin state image to be identified; the category recognition model is obtained by training a convolutional neural network based on historical skin state images;

[0082] Step 103: When the category of the skin state image to be identified is an unhealthy state, the features based on which the identification result is based are determined using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified.

[0083] Before step 101, the method further includes:

[0084] Get multiple historical skin status pictures;

[0085] Labeling the types of multiple historical skin state images to obtain multiple historical skin state labeled images as training sets;

[0086] With the training set as input and the categories of multiple pictures in the training set as output, the convolutional neural network is trained to obtain a category recognition model.

[0087] Among them, the convolutional neural network is trained with the training set as input and the types of multiple pictures in the training set as output to obtain a category recognition model, which specifically includes:

[0088] Construct a convolutional neural network as a 0th-order category recognition model;

[0089] Let the number of iterations m be equal to 1;

[0090] The training set is input into the m-1th order category recognition model to obtain the mth order category recognition model and multiple historical skin state pseudo-labeled images;

[0091] According to the training set and multiple historical skin status pseudo-annotated images, the formula Determine the loss function for the mth iteration;

[0092] Determine whether the loss function of the mth iteration is less than the loss function threshold and obtain a determination result;

[0093] If the judgment result is no, the training set is updated according to the multiple historical skin state pseudo-labeled images, the value of the iteration number m is increased by 1, and the step of "inputting the training set into the m-1th order category recognition model to obtain the mth order category recognition model and multiple historical skin state pseudo-labeled images" is returned;

[0094] If the judgment result is yes, then the m-th order category recognition model is determined to be the category recognition model;

[0095] Where Loss is the loss function; n is the number of samples; x is the image in the training set at the mth iteration; y is the annotation result of the image in the training set; a is the recognition result of the m-1th order category recognition model.

[0096] Specifically, the training set is updated based on multiple historical skin status pseudo-annotated images, including:

[0097] Determine a plurality of historical skin state pseudo-annotated images with prediction accuracy greater than a prediction accuracy threshold as a backup set;

[0098] Determine the number of images corresponding to each skin condition type in the training set;

[0099] Arrange the various skin condition categories in ascending order according to the number of corresponding pictures;

[0100] Determine the first preset number of skin condition types as the skin condition types to be updated;

[0101] In the spare set, multiple historical skin state pseudo-annotated images corresponding to each type of skin state to be updated are selected and added to the training set.

[0102] In addition, the method for classifying and determining features of skin condition images provided by the present invention, step 103, specifically includes:

[0103] Construct weighted linear regression models and optimization functions;

[0104] According to the pixel value of each pixel point on the skin state map to be identified, the weighted linear regression model is optimized using an optimization function, and the estimation coefficient group corresponding to the minimum of the optimization function is determined to be the Shapley value of each pixel point on the skin state map to be identified;

[0105] Determine the pixel values ​​of pixels whose Shapley values ​​are not equal to 0 as the features based on the recognition results;

[0106] in,

[0107] The weighted linear regression model is:

[0108] The optimization function is

[0109] Among them, g(.) is the weighted linear regression model, that is, the explanatory model; z' is the indicative function; z'∈{0,1} M ; M is the size of the largest coalition; φ0 is the average of the Shapley values; φ j is the Shapley value of the jth pixel; z' j is the indicative function value of the jth pixel, z' j 1 means the pixel value of the jth pixel is greater than 0, z' j 0 means that the pixel value of the jth pixel is equal to 0; L(.) is the optimization function, and the estimated coefficient group of the optimization function is the corresponding multiple φ when j in g(.) is 1 to M j f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' to the pixel value of the pixel point x to be explained; π x (.) is the kernel function,

[0110] like Figure 2-4 ,The present invention is divided into seven parts: data acquisition, data preprocessing, construction and training of Teachermodel, pseudo-labeling of unlabeled samples, enhanced training, model evaluation and final model interpretation.

[0111] 1. Data preprocessing

[0112] The present invention needs to obtain skin disease image sample data through access to medical sites and annotate skin disease images, which specifically includes:

[0113] 1. Record and save the data of skin disease images in Dicom, png or jpeg format. 2. The skin disease images that have been obtained are annotated by dermatologists. Taking the following 9 skin diseases as examples, AK (actinic keratosis) is marked as 0, BCC (basal cell carcinoma) is marked as 1, BKL (benign keratosis) is marked as 2, DF (dermatofibroma) is marked as 3, MEL (melanoma) is marked as 4, NV (melanocytic nevus) is marked as 5, SCC (squamous cell carcinoma) is marked as 6, Unknown (unknown or uncertain) is marked as 7, and VASC (vascular lesion) is marked as 8. The full names, abbreviations, disease names and corresponding annotation symbols of various skin diseases are shown in Table 1:

[0114] Table 1 Skin status image annotation

[0115]

[0116]

[0117] (II) Data preprocessing

[0118] The skin disease images are divided into different skin disease sets according to the annotations, and then the data is divided into training sets and test sets. Specifically, the images are divided into several sets according to the annotations, and 80% of all types of images are used as training sets and 20% as test sets to evaluate the model. The subsequent model evaluation is based on the comprehensive consideration of the test set and the training set. The unlabeled images are separately divided into the unlabeled skin disease image data set, which will be used as the data enhancement data set later.

[0119] (III) Construction and training of Teachermodel

[0120] Model Scaling has always been an important method to improve the effect of convolutional neural networks. This paper attempts to build a Teachermodel based on the efficient EfficientNet neural network framework. Based on the Teacher model with a high accuracy, a large number of unlabeled skin disease images are labeled with pseudo labels, and on this basis, different categories of skin disease samples are added to the training set with a specific probability distribution to continue training, and finally a high-precision skin disease prediction model is obtained. Specifically including:

[0121] 1. Establish a training sample set

[0122] 2. Establish the model. Based on the neural network initial model, the model architecture is based on MBConv (small benchmark convolution), including: 32 convolutional layers with 3×3 convolution kernels, 16 MBConv6, k3×3, 24 MBConv6, k3×3, 40 MBConv6, k5×5, 80 MBConv6, k3×3, 112 MBConv6, k5×5, 192 MBConv6, k5×5, 320 MBConv6, k3×3, 1280 convolutional layers with 1×1 convolution kernels (Conv1×1), an average pooling layer (Pooling) and a fully connected layer (FC) with an output of 9. The EfficientNet neural network structure and the mobile-size baseline structure are shown in Tables 2 and 3 respectively, where the parameter k of each mobile-size baseline refers to the channels value in Table 2.

[0123] Table 2 EfficientNet structure

[0124]

[0125]

[0126] Table 3 mobile-size baseline structure

[0127]

[0128] 3. Construct the loss function of model output a and sample label y:

[0129]

[0130] 4. Based on the training samples, the RMSprop optimizer is used to solve the optimal parameters of the model. At the same time, the validation set is used to evaluate the model effect from the perspective of AUC, accuracy, loss function value and other indicators during the iterative training process;

[0131] 5. Select the model with the best performance during the training process as the Teacher model to annotate pseudo labels for unlabeled skin disease images;

[0132] 6. Pseudo-labeled skin disease image samples are analyzed with probability In the training set: α is a custom parameter, N l is the number of samples of the first disease in the pseudo-label dataset (each disease is sorted in descending order by the number of samples); N1 is the maximum sample size, N L is the sample size of the smallest category, and the subscript here is the index after descending order.

[0133] 7. Continue to train the model on the expanded data set in step 6 until the upward trend of the model accuracy is no longer obvious (i.e., convergence is reached), and save the higher accuracy model weights for alternative selection.

[0134] (IV) Model Verification

[0135] The prediction performance of the test model on the test set is used to evaluate the generalization ability of the final model, which includes the following two aspects:

[0136] 1. Model validation, based on the test set, the candidate model weights were evaluated from the ROC curve, AUC, accuracy, precision, recall, and loss function value. Table 4 lists all the possibilities of the prediction results of a single type of skin disease.

[0137] Table 4 Examples of prediction results for a single type of skin disease

[0138]

[0139] in,

[0140] The accuracy is:

[0141] The precision is:

[0142] The recall rate and true positive rate (TPR) are:

[0143] The false positive rate (FPR) is:

[0144] Traditional accuracy indicators can only serve as a reference in model training. As for specific issues, the main content of medical diagnosis is to diagnose patients with diseases as positive. At this time, the larger the TPR, the better. Misdiagnosing healthy patients as sick means that the lower the FPR, the better. Therefore, in the diagnosis of skin diseases, recall, precision and accuracy are all important criteria for model evaluation. The ROC (Receiver Operating Characteristic) curve measures the relative relationship between the false positive rate (FPR) and the true positive rate (TPR) of the classification model. Based on the alternative model, a TPR and FPR point pair is obtained according to its performance on the test set data, which is mapped to a point on the ROC plane. By adjusting the threshold used by this classifier (from 0 to 1), a ROC curve passing through (0,0) and (1,1) can be obtained. The AUC value is the area under the curve, such as Figure 5 .

[0145] 2. Model selection: Based on the evaluation results, the model with the best prediction performance among the alternative models is selected as the final intelligent diagnosis model for skin diseases.

[0146] Figure 5 The horizontal axis is the false positive rate, the vertical axis is the true positive rate, ROC is the receiver operating characteristic, and Aera is the area under the corresponding curve. The different curves in the figure represent the relative relationship between the "true positive rate" and the "false positive rate" under the model performance of different threshold classifiers. The closer the curve is to the upper left corner, the better the model effect. At the same time, the AUC value represents the area under the corresponding ROC curve. The closer the value is to 1, the stronger the model's recognition ability for this type of sample. In the example, the AUC value of all categories of skin diseases in the test set reached 0.99. At this time, the model performance ability is excellent, so it is used as the final prediction model.

[0147] (V) Model explanation

[0148] The prediction results of the optimal machine learning model are explained by the SHAP (Shapley interpretation) method, and the contribution of each pixel in the skin disease image to the prediction results is calculated.

[0149] SHAP defines explanation as: Among them, g(.) is the explanation model, z'∈{0,1} M is the coalition vector, M is the maximum coalition vector, φ j ∈R is the feature-attributed Shapley value of feature j.

[0150] At the same time, the alliance vector is set. Input 1 indicates that the corresponding feature (i.e., the pixel value of the pixel point) exists, while input 0 indicates that it does not exist. For the instance of interest (i.e., pixel point) x, the alliance vector x' is a vector of all 1s, that is, all feature values ​​exist. The formula is simplified to:

[0151]

[0152] Calculate the Shapley value through SHAP and establish the SHAP kernel:

[0153]

[0154] Where M is the maximum coalition vector and |z′| is the number of current features (input variables) in instance z′.

[0155] Then build a weighted linear regression model:

[0156]

[0157] The linear model g is trained by optimizing the following optimization function L:

[0158]

[0159] Among them, L(.) is the optimization function, f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' (0 or 1) to the corresponding value of the instance x to be explained; Z is the training data, and the estimated coefficient φ of the model is obtained by optimizing the sum of squared errors of the linear model. j This is the Shapley value that needs to be solved, which is also the contribution rate of each pixel point of the skin disease prediction model input image to the final result. When the global importance of each pixel point of the input image is required, according to the additive property of the Shapley value, we can take the average value of the Shapley absolute value of each feature in the data: I j Represents the average contribution.

[0160] Figure 6In the figure, given four skin disease images, from left to right are the prediction results of the top three most likely diseases. Taking the four images in the second row as an example, the first one is the input image, and the second one shows that the image is most likely to be a benign keratosis (BKL), which is also the final prediction result of the model. The red and blue pixels in the image represent the SHAP value of the pixel to this result. Red indicates that the pixel contributes positively to this result, and blue indicates that the pixel contributes negatively to this result. The darker the color, the greater the contribution rate. The same explanation is given in the two images on the right. In contrast, the small block in the middle lesion area contributes negatively to the final prediction value, while this time the contribution to the result is positive, and the other reddish lesion skin around it contributes positively to the final result. This also shows that the model identifies the skin disease image as a benign keratosis rather than a melanocytic nevus (NV) because of the surrounding reddish lesion skin.

[0161] In addition, the present invention also provides a system for classifying and determining features of skin condition images, comprising:

[0162] A module for obtaining a skin state image to be identified, used for obtaining a skin state image to be identified;

[0163] An image category recognition module is used to input the skin state image to be recognized into a category recognition model to recognize the category of the skin state image to be recognized; the category recognition model is obtained by training a convolutional neural network based on historical skin state images;

[0164] The feature determination module is used to determine the features based on the recognition result by using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified when the category of the skin state image to be identified is an unhealthy state.

[0165] In addition, the skin condition image category recognition and feature determination system provided by the present invention further includes:

[0166] A historical skin state image acquisition module is used to acquire multiple historical skin state images;

[0167] A labeling module, used to label the types of multiple historical skin state images, and obtain multiple historical skin state labeling images as training sets;

[0168] The category recognition model building module is used to train the convolutional neural network with the training set as input and the categories of multiple pictures in the training set as output to obtain a category recognition model.

[0169] Specifically, the category recognition model building module includes:

[0170] A 0th order category recognition model building unit, used to build a convolutional neural network as a 0th order category recognition model;

[0171] An iteration number assignment unit, used to set the iteration number m to be equal to 1;

[0172] A training unit, used for inputting the training set into the m-1th order category recognition model to obtain the mth order category recognition model and a plurality of historical skin state pseudo-labeled images;

[0173] The loss function calculation unit is used to calculate the loss function based on the training set and multiple historical skin state pseudo-labeled images using the formula Determine the loss function for the mth iteration;

[0174] A judgment unit is used to judge whether the loss function of the mth iteration is less than the loss function threshold value and obtain a judgment result; if the judgment result is no, the training set update unit is called; if the judgment result is yes, the category recognition model determination unit is called;

[0175] A training set updating unit, used to update the training set according to a plurality of historical skin state pseudo-labeled images, increase the value of the iteration number m by 1, and call the training unit;

[0176] A category recognition model determination unit, used to determine the m-th order category recognition model as the category recognition model;

[0177] Where Loss is the loss function; n is the number of samples; x is the image in the training set at the mth iteration; y is the annotation result of the image in the training set; a is the recognition result of the m-1th order category recognition model.

[0178] Furthermore, the training set updating unit specifically includes:

[0179] A reserve set determination subunit, used for determining a plurality of historical skin state pseudo-annotated images with prediction accuracy greater than a prediction accuracy threshold as a reserve set;

[0180] A quantity determination subunit, used to determine the number of images corresponding to each skin condition type in the training set;

[0181] A sorting subunit, for sorting the multiple skin condition categories in ascending order according to the number of corresponding pictures;

[0182] A skin condition category determination subunit to be updated, used to determine a first preset number of skin condition categories as skin condition categories to be updated;

[0183] The training set updating subunit is used to select multiple historical skin state pseudo-annotated images corresponding to each type of skin state to be updated in the spare set and add them to the training set.

[0184] In addition, the feature determination module specifically includes

[0185] Model building unit, used to build weighted linear regression model and optimization function;

[0186] A Shapley value determination unit is used to optimize the weighted linear regression model using an optimization function according to the pixel value of each pixel point on the skin state map to be identified, and determine that the estimation coefficient group corresponding to the minimum of the optimization function is the Shapley value of each pixel point on the skin state map to be identified;

[0187] A recognition result based feature determination unit, used for determining the pixel value of the pixel point whose Shapley value is not equal to 0 as the feature based on the recognition result;

[0188] in,

[0189] The weighted linear regression model is:

[0190] The optimization function is

[0191] Among them, g(.) is the weighted linear regression model, that is, the explanatory model; z' is the indicative function; z'∈{0,1} M ; M is the size of the largest coalition; φ0 is the average of the Shapley values; φ j is the Shapley value of the jth pixel; z' j is the indicative function value of the jth pixel, z' j 1 means the pixel value of the jth pixel is greater than 0, z' j 0 means that the pixel value of the jth pixel is equal to 0; L(.) is the optimization function, and the estimated coefficient group of the optimization function is the corresponding multiple φ when j in g(.) is 1 to M j f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' to the pixel value of the pixel point x to be explained; π x (.) is the kernel function,

[0192] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0193] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for classifying and determining features of skin condition images, characterized in that: The method comprises: Obtain a skin state image to be identified; Inputting the skin state image to be identified into a category recognition model to identify the category of the skin state image to be identified; the category recognition model is obtained by training a convolutional neural network based on historical skin state images; When the category of the skin state image to be identified is an unhealthy state, the features based on which the identification result is based are determined using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified; Before obtaining the skin state map to be identified, the method further includes: Get multiple historical skin status pictures; Labeling the types of the plurality of historical skin condition images to obtain a plurality of historical skin condition labeled images as a training set; Taking the training set as input and the types of multiple pictures in the training set as output, training the convolutional neural network to obtain a category recognition model; When the category of the skin state image to be identified is an unhealthy state, the features based on the identification result are determined by using the Shapley value method according to the pixel value of each pixel point on the skin state image to be identified, specifically including: Construct weighted linear regression models and optimization functions; According to the pixel value of each pixel point on the skin state map to be identified, the weighted linear regression model is optimized using the optimization function, and the estimation coefficient group corresponding to the minimum of the optimization function is determined to be the Shapley value of each pixel point on the skin state map to be identified; Determine the pixel values ​​of pixels whose Shapley values ​​are not equal to 0 as the features based on the recognition results; in, The weighted linear regression model is: The optimization function is Among them, g(.) is the weighted linear regression model, that is, the explanatory model; z' is the indicative function; z'∈{0,1} M ; M is the size of the largest coalition; φ0 is the average of the Shapley values; φ j is the Shapley value of the jth pixel; z' j is the indicative function value of the jth pixel, z' j 1 means the pixel value of the jth pixel is greater than 0, z' j 0 means that the pixel value of the jth pixel is equal to 0; L(.) is the optimization function, and the estimated coefficient group of the optimization function is the corresponding multiple φ when j in g(.) is 1 to M j f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' to the pixel value of the pixel point x to be explained; π x (.) is the kernel function, 2. The method for classifying and determining features of skin condition images according to claim 1, characterized in that: The method uses the training set as input and the types of multiple pictures in the training set as output to train the convolutional neural network to obtain a category recognition model, specifically including: Construct a convolutional neural network as a 0th-order category recognition model; Let the number of iterations m be equal to 1; Inputting the training set into an m-1th order category recognition model to obtain an mth order category recognition model and a plurality of historical skin state pseudo-labeled images; According to the training set and the plurality of historical skin state pseudo-labeled images, the formula Determine the loss function for the mth iteration; Determine whether the loss function of the mth iteration is less than the loss function threshold and obtain a determination result; If the judgment result is no, the training set is updated according to the multiple historical skin state pseudo-labeled images, the value of the iteration number m is increased by 1, and the step of "inputting the training set into the m-1th order category recognition model to obtain the mth order category recognition model and multiple historical skin state pseudo-labeled images" is returned; If the judgment result is yes, determining the m-th order category recognition model as the category recognition model; Where Loss is the loss function; n is the number of samples; x is the image in the training set at the mth iteration; y is the annotation result of the image in the training set; a is the recognition result of the m-1th order category recognition model.

3. The method for classifying and determining features of skin condition images according to claim 2, characterized in that: The updating of the training set according to the plurality of historical skin state pseudo-labeled images specifically includes: Determine a plurality of historical skin state pseudo-annotated images with prediction accuracy greater than a prediction accuracy threshold as a backup set; Determine the number of images corresponding to each skin condition type in the training set; Arrange the various skin condition categories in ascending order according to the number of corresponding pictures; Determine the first preset number of skin condition types as the skin condition types to be updated; A plurality of historical skin state pseudo-annotated images corresponding to each type of skin state to be updated are selected from the spare set and added to the training set.

4. A system for classifying and determining features of skin condition images, characterized in that: The system comprises: A module for obtaining a skin state image to be identified, used for obtaining a skin state image to be identified; An image category recognition module, used for inputting the skin state image to be recognized into a category recognition model to recognize the category of the skin state image to be recognized; the category recognition model is obtained by training a convolutional neural network based on historical skin state images; A feature determination module, for determining the feature based on which the recognition result is based on the pixel value of each pixel point on the skin state image to be identified by using the Shapley value method when the category of the skin state image to be identified is an unhealthy state; A historical skin state image acquisition module is used to acquire multiple historical skin state images; A labeling module, used for labeling the types of the plurality of historical skin state images to obtain a plurality of historical skin state labeling images as a training set; A category recognition model building module is used to train a convolutional neural network using the training set as input and the types of multiple pictures in the training set as output to obtain a category recognition model; The feature determination module specifically includes Model building unit, used to build weighted linear regression model and optimization function; A Shapley value determination unit is used to optimize the weighted linear regression model using the optimization function according to the pixel value of each pixel point on the skin state map to be identified, and determine that the estimation coefficient group corresponding to the minimum of the optimization function is the Shapley value of each pixel point on the skin state map to be identified; A recognition result based feature determination unit, used for determining the pixel value of the pixel point whose Shapley value is not equal to 0 as the feature based on the recognition result; in, The weighted linear regression model is: The optimization function is Among them, g(.) is the weighted linear regression model, that is, the explanatory model; z' is the indicative function; z'∈{0,1} M ; M is the size of the largest coalition; φ0 is the average of the Shapley values; φ j is the Shapley value of the jth pixel; z' j is the indicative function value of the jth pixel, z' j 1 means the pixel value of the jth pixel is greater than 0, z' j 0 means that the pixel value of the jth pixel is equal to 0; L(.) is the optimization function, and the estimated coefficient group of the optimization function is the corresponding multiple φ when j in g(.) is 1 to M j f(.) represents the predicted value of the explained model, h(.) represents the mapping function, that is, mapping the indicative function z' to the pixel value of the pixel point x to be explained; π x (.) is the kernel function, 5. The system for classifying and determining features of skin condition images according to claim 4, characterized in that: The category recognition model building module specifically includes: A 0th order category recognition model building unit, used to build a convolutional neural network as a 0th order category recognition model; An iteration number assignment unit, used to set the iteration number m to be equal to 1; A training unit, used for inputting the training set into an m-1th order category recognition model to obtain an mth order category recognition model and a plurality of historical skin state pseudo-annotated images; A loss function calculation unit is used to calculate the loss function according to the training set and the plurality of historical skin state pseudo-labeled images using the formula Determine the loss function for the mth iteration; A judgment unit, used to judge whether the loss function of the mth iteration is less than the loss function threshold, and obtain a judgment result; if the judgment result is no, calling the training set updating unit; if the judgment result is yes, calling the category recognition model determination unit; A training set updating unit, used for updating the training set according to the plurality of historical skin state pseudo-annotated images, increasing the value of the number of iterations m by 1, and calling the training unit; A category recognition model determination unit, used to determine the m-th order category recognition model as the category recognition model; Where Loss is the loss function; n is the number of samples; x is the image in the training set at the mth iteration; y is the annotation result of the image in the training set; a is the recognition result of the m-1th order category recognition model.

6. The system for classifying and determining features of skin condition images according to claim 5, characterized in that: The training set updating unit specifically includes: A reserve set determination subunit, used for determining a plurality of historical skin state pseudo-annotated images with prediction accuracy greater than a prediction accuracy threshold as a reserve set; A quantity determination subunit, used to determine the number of pictures corresponding to each skin condition type in the training set; A sorting subunit, for sorting the multiple skin condition categories in ascending order according to the number of corresponding pictures; A skin condition category determination subunit to be updated, used to determine a first preset number of skin condition categories as skin condition categories to be updated; The training set updating subunit is used to select a plurality of historical skin state pseudo-annotated images corresponding to each type of skin state to be updated from the standby set and add them to the training set.

Images