Method for determining the severity of a skin disease based on the proportion of the body surface area occupied by a lesion

Abstract

An image processing method is provided that uses machine learning techniques to automatically calculate a body surface area (BSA) score. The Felzenszwalb image segmentation algorithm is used to define a proposal region in each of a plurality of training set images. The training set images are over-segmented, and then each proposal region in each of the plurality of over-segmented training set images is manually classified as a lesion or a non-lesion. A convolutional neural network (CNN) is then trained using the manually classified proposal regions in each of the plurality of training set images. The trained CNN is then used on test images to calculate a BSA score.

Description

Background Art

[0001] The evaluation of the disease severity of skin diseases such as psoriasis involves calculating the ratio of the body surface area occupied by lesions and inflammation (i.e., the BSA score). Hereinafter, lesions and inflammation are collectively referred to as "lesions". BSA is the measured or calculated surface area of the human body.

[0002] Psoriasis is an autoimmune skin disease that appears as red inflamed areas different from healthy normal skin. An important part of measuring the disease severity of psoriasis is to monitor the ratio of the body surface area occupied by the inflamed area called "lesions". In the case of plaque psoriasis, the two main disease measurements are BSA and PASI (Psoriasis Area and Severity Index), and both of these involve calculating a ratio score that is used to monitor the progression and treatment effect of the disease (A Bozek, A. Reich (2017). Reliability of three psoriasis assessment tools: Psoriasis Area and Severity Index, body surface area, and physician's global assessment. Adv Clin Exp Med. August 2017; 26(5): 851-856. doi:10.17219 / acem / 69804). Currently, these ratios are often evaluated by dermatologists or nurses in a physician's office. The main problem with current psoriasis disease scores is that they are inaccurate and cumbersome evaluations involving human biases. Furthermore, the process of calculating the ratios to obtain the overall PASI score is cumbersome and time-consuming. Another clinical need is that currently, in the case of guttate psoriasis where the affected area is extensive and occupied by a large number of inflammatory lesions ranging in size from 2 to 10 mm and visual measurement is difficult, there is no BSA means.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Therefore, there is a need for a more objective and quantitative method for monitoring skin inflammation using a calculation method. The present invention meets such a need.

Means for Solving the Problems

[0004] An image processing method is provided that automatically calculates a BSA score using machine learning techniques. The Felzenszwalb image segmentation algorithm is used to define proposed regions in each of a plurality of training set images. The training set images are over-segmented (''over-segmented''), and then each proposed region of each of the plurality of over-segmented training set images is manually classified as a lesion or non-lesion. A convolutional neural network (CNN) is then trained using the manually classified proposed regions in each of the plurality of training set images. The trained CNN is then used on a test image to calculate the BSA score. The present invention also includes a device (or computer system) driven by a computer instruction used in connection with a computer-related device or medium for implementing the method, for example, a method known in the relevant technical field.

Brief Description of the Drawings

[0005] This patent or application documents include at least one drawing created in color. A copy of this patent or patent application publication with color drawings will be provided by the Patent Office upon request, upon payment of the necessary fees.

[0006] The above summary and the following detailed description of the preferred embodiments of the present invention will be better understood when read in conjunction with the accompanying drawings. For the purpose of illustrating the present invention, the drawings show currently preferred embodiments. However, the present invention is not limited to the exact configurations and means shown.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7A

Figure 7B

DETAILED DESCRIPTION OF THE INVENTION

[0007] Certain terms are used herein for convenience and should not be construed as limiting the present invention.

[0008] I. Overview FIG. 6 is a flowchart of a computer-implemented method for determining the severity of a skin disease (e.g., psoriasis) based on the percentage of BSA occupied by lesions. FIGS. 7A and 7B are schematic diagrams of the system software and hardware for executing FIG. 6.

[0009] Referring to FIGS. 6, 7A, and 7B, the method operates as follows. Step 600: Perform image segmentation on a plurality of training set images of body surface area using the Felzenszwalb segmentation algorithm (FSA), and output proposed regions in each of the plurality of training set images. Each of the plurality of training set images of body surface area includes skin diseases. This image segmentation step is performed by the processor 700 shown in FIG. 7A. Step 602: Over-segment each of the plurality of training set images. This step is also performed by the processor 700. Step 604: Manually classify each proposed region in each of the plurality of over-segmented training set images as a lesion or non-lesion. This step is performed by one or more human classifications 702 shown in FIG. 7A. Step 606: Use the manually classified proposed regions in each of the plurality of training set images to train a neural network. This step is performed by the neural network 704 shown in FIG. 7A. Step 608: Perform image segmentation on a test image of body surface area including skin diseases using the Felzenszwalb segmentation algorithm, and output regions within the test image of body surface area. This image segmentation step is performed by the processor 700' shown in FIG. 7B. The processor 700' may be the same as the processor 700 or a different processor. Step 610: Over-segment the test image. This step is also performed by the processor 700'. The trained neural network labeled as 704' in FIG. 7B is the same neural network as the neural network 704 in FIG. 7A except that it is being trained here, so input the regions of the over-segmented test image into this trained neural network. Step 614: Using the trained neural network 704’, identify non-lesion regions, filter them from the over-segmented test image, and classify the remaining regions of the over-segmented test image as lesion regions. Step 616: Calculate the percentage of BSA in the test image occupied by the lesion using the area of the classified lesion regions of the over-segmented test image and the area of the identified non-lesion regions of the over-segmented test image. This step is executed by the processor 700”. The processor 700” may be the same as the processor 700 or 700’, or a different processor.

[0010] II. DETAILED DISCLOSURE The following detailed disclosure describes the experimental process that led to the present invention and explains an approach that was more successful than other approaches.

[0011] The present invention addresses the problem of calculating the body surface area, which is an issue in image segmentation. Image segmentation is one of the essential problems in computer vision and is defined as the process of organizing image data into meaningful groups by dividing a digital image into multiple segments. A series of image segmentation methods that can effectively calculate the body surface area of a patient's psoriasis disease were investigated. The preferred embodiment of the present invention uses the Felzenszwalb image segmentation algorithm [3] and the convolutional neural network (A. Krizhevsky, I. Sutskever, G. E. Hinton (2012) “Imagenet classification with deep convolutional neural networks”, “Advances in Neural Information Processing” (NeurIPS), 2012) as a false positive filter to improve the Felzenszwalb segmentation results.

[0012] Several image processing methods were tested, and machine learning techniques applied to 117 images of guttate psoriasis and plaque psoriasis downloaded from the Internet were used to automatically calculate the BSA score. Compared to several other unsupervised segmentation methods, the Felzenszwalb image segmentation algorithm produced the most accurate segmentation with the highest ratio of lesion area to non-lesion area, with 56% of the images having good segmentation while 44% of the images were under-segmented or over-segmented. To improve the segmentation results, a convolutional neural network (CNN) influenced by the Visual Geometry Group (VGG) architecture was run to further filter out false positive lesions from the proposed regions of the Felzenszwalb algorithm. The training data for the CNN consisted of a dataset of human-curated lesion and non-lesion regions output from the Felzenszwalb algorithm. The CNN achieved an accuracy score of 90% in a 5-fold cross-validation test when classifying lesions and non-lesions. This CNN filter was applied together with the Felzenszwalb algorithm and accurately segmented 67 out of 86 guttate psoriasis images, or 77% of the training data. This method is useful for the digitalization of disease severity measurement and remote monitoring of skin diseases such as psoriasis for patients and physicians.

[0013] Psoriasis image data To collect the dataset, approximately 300 images of guttate psoriasis and 100 images of chronic plaque psoriasis were collected from Google Images. Incorrect and / or misleading image data were filtered out, leaving a final dataset of 86 images of guttate psoriasis and 31 images of chronic plaque psoriasis.

[0014] Psoriasis image segmentation method Five different image segmentation algorithms were tested, all implemented in Python using the scikit-learn and OpenCV libraries. The five tested algorithms were Felzenszwalb, Quickshift, SLIC, Compact watershed, and Otsu's thresholding algorithm (see D. Liu, B. Soran, G. Petrie, and L. Shapiro, "A review of computer vision segmentation algorithms," Lecture Notes, 53, 2012). After visually inspecting examples of the combined images, it became clear that the Felzenszwalb method (P.F. Felzenszwalb and D.P. Huttenlocher (2004), "Efficient Graph-Based Image Segmentation," International Journal of Computer Vision 59(2), 167-181) yielded significantly the most promising segmentation.

[0015] Figure 1 shows the output images of four different segmentation algorithms. The yellow boundaries mark the segmentation of the regions proposed by the algorithms. After these tests, it was decided to segment all images with the Felzenszwalb algorithm.

[0016] Psoriasis Segmentation and Binary Classification Filtering Algorithm To enhance the Felzenszwalb segmentation results, a new algorithm was used based on a convolutional neural network filter that removes false positives from the segmentation. This algorithm is currently inspired by and has similarities to the region-based convolutional neural network (R-CNN), which is a state-of-the-art area when dealing with image segmentation problems. The algorithm is as follows: 1. Train a neural network to distinguish between lesions and non-lesions within the proposed regions from the Felzenszwalb segmentation algorithm. 2. Over-segment the image by increasing the k-value parameter to the optimal threshold (k = 250). 3. Use the neural network to filter out the non-lesions proposed by Felzenszwalb segmentation.

[0017] Ensure that most lesions are included within the segmentation by over-segmenting the image in step 2. Then, use the pre-trained neural network from step 1 to distinguish between non-lesion regions and lesion regions in step 3 in order to filter out excessive non-lesions or false positives.

[0018] Training Data Over-segment the image using the proposed regions output from the Felzenszwalb segmentation algorithm (k-value set to 250) to include as many true positive lesion regions as possible. Next, select approximately 30 guttate psoriasis images and generate images of approximately 3000 proposed regions. Then, manually (by humans) classify each of the 3000 proposed regions as a lesion or non-lesion. The process was repeated 3 times to verify the accuracy of this dataset. In large regions occupied by a lot of skin, black background areas, necklaces, noise, etc., some of these non-lesions were easy to identify. Other regions, including areas with shadows, insufficient lighting, scars, etc., were difficult to distinguish between lesions and non-lesions. The main causes of error are thought to be due to the incompatibility of the dataset and the regions where it is difficult to classify these lesions and non-lesions. This could be an explanation for why the binary classification results still do not come close to the state-of-the-art results seen in the Modified National Institute of Standards and Technology (MNIST) or Canadian Institute for Advanced Research (CIFAR) datasets.

[0019] Binary Classification Neural Network Experiment To train the neural network model, the same procedure as described below was followed.

[0020] First, all proposed region inputs were preprocessed by resizing all images to a specific fixed pixel size using cubic interpolation from the opencv library. Next, various parameters of the neural network model were varied, including the model architecture (dense neural network vs convolutional neural network, hidden layer size, batch normalization), parameters (learning rate), and input image interpolation pixel sizes (4×4, 8×8, 16×16, 32×32, 64×64). Finally, all models were tested with an 80:20 training - test data split ratio to determine the accuracy, log loss, and mean squared error scores of each model. This means that 80% of the data was used to build the model and 20% of the remaining data (not seen by some models) was used to evaluate the predictive strength of the model.

[0021] The selected final CNN model was inspired by the VGG architecture, did not use batch normalization, had smaller hidden layers (both the number and width of the hidden layers), had a learning rate of 1e - 4, and used the Adam optimization technique. (Adam is an adaptive learning rate optimization algorithm specifically designed for training deep neural networks). To ensure the accuracy of the final model, this model was tested with 5 - fold cross - validation and obtained an average 5 - fold training accuracy score of 94% and an average 5 - fold test accuracy score of 90%.

[0022] Matters to improve the accuracy results There were three approaches that significantly improved the binary classification accuracy of the neural network.

[0023] The first approach was to interpolate the shape of the input image to a fixed 16×16 pixel size. First, when the image was set to a size of 64×64, the classification accuracy fluctuated around a very low rate of 60%. As the size of the image decreased, it was found that a standard small dense neural network gradually achieved better classification performance until it reached the optimal 16×16 image size. This could be because most of the proposed regions were approximately 16×16 in size. Therefore, by interpolating more information, the region could obtain information that caused misunderstandings, and by interpolating less information, the region could lose valuable information.

[0024] The second approach was to use a convolutional neural network, in contrast to a dense neural network. Convolutional neural networks are well known to function better than dense neural networks in image classification tasks for numerous reasons.

[0025] The third approach was that "the fewer, the more effective" when constructing a convolutional neural network model architecture. For example, when the hidden layer size was reduced, faster training time and more accurate test verification scores were obtained. Specifically, reducing the hidden size of the first dense layer was particularly important as it thereby reduced a significant number of parameters. This could be due to the fact that excessive parameters may prevent the model from being well derived and may cause overfitting. This overfitting problem can be most clearly seen in the nearly 10% loss of accuracy between the training verification score and the test verification score in a large VGG model.

[0026] Matters that do not affect or worsen the segmentation results Some approaches did not improve the accuracy results. More specifically, three approaches had little or a negative impact on the model.

[0027] First, adding batch normalization layers as seen in the VGG model was thought to improve the results. However, as shown in a particular paper (S. Santurkar, D. Tsipras, A. Ilyas, A. Madry (2018) "How Does Batch Normalization Help Optimization?", "Advances in Neural Information Processing" (NeurIPS), 2018), batch normalization does not seem to improve classification accuracy in all cases.

[0028] Second, adjusting the learning rate did not improve the performance. After testing learning rates of 0.01, 0.001, 0.0003, and 0.0001, it was found that the learning rate made a negligible difference in the performance results for the learning rate parameters of 0.001, 0.0003, and 0.0001.

[0029] Third, deeper and wider networks did not improve the model's accuracy. Larger high-density neural network models and convolutional neural network models actually seemed to be up to 10% worse in the cross-validation tests than the simple small VGG model tried. This seems to support the fundamental principle of the agile development principle - simplicity.

[0030] Results and Discussion i. Advantages and Disadvantages of Image Segmentation Algorithms After testing small samples of images with five different algorithms, as shown in Figure 1, it became clear that the Felzenszwalb algorithm produced much better results than the other five unsupervised algorithms. An important feature of this method is that it can retain details in low-variation image regions while ignoring details in high-variation regions. This method is also fast (less than 1 second for a 512×512 image) with a runtime of O(n log n) (where n is the number of pixels). Based on these observations, the Felzenszwalb method was selected as the main method to be applied to psoriasis images.

[0031] ii. Felzenszwalb Image Segmentation Scoring Metric and Results Figure 2 shows an example of good image segmentation. Good image segmentation is defined as image segmentation that does not miss any clearly significant lesions that a normal doctor would naturally classify.

[0032] Figure 3 shows three examples of inaccurate image segmentation. The first image is an example of an under-segmented area. To reduce this problem, it is necessary to decrease the k-value parameter. The second image is an example of an over-segmented area. To reduce this problem, it is necessary to increase the k-value parameter. The last example is an example of a large area classified as a lesion. To reduce this, it is necessary to increase the k-value parameter. Figures 5A and 5B also show examples of under-segmentation and over-segmentation.

[0033] The basic Felzenszwalb algorithm was able to segment 49 out of 88 input images with good segmentation, 30 out of 88 input images were under-segmented, and 9 out of 88 input images were over-segmented. The calculation of the BSA score was generated at the end of each segmentation result.

[0034] iii. Neural Network-Based Filtering Results Twenty-five different neural network models were tested with different parameters, architectures, and input sizes. The test accuracy of each model was calculated. The setting of a smaller image size, the selection of a smaller VGG-based convolutional neural network, and a learning rate of 1e-4 produce the best model with a maximum test accuracy of 0.9.

[0035] iv. Convolutional Neural Network Filter and Felzenszwalb Image Segmentation Results After performing Felzenszwalb image segmentation filtered by a convolutional neural network, 67 out of 86 images provided good segmentation, 16 out of 86 images were under-segmented, and 3 out of 86 images were over-segmented. Due to inaccurate representation of guttate psoriasis, two input images were removed from the original Felzenszwalb image segmentation set. An example of a convolutional neural network for improving segmentation results can be seen in the image of Figure 4.

[0036] In the left image of Figure 4, the convolutional neural network was able to filter a large area on the left side of the arm, all background white areas, and many small noise areas manually classified as non-lesions. With better training data and more extensive training, the neural network model can achieve even better filtering results and can be used as an effective supplement to the Felzenszwalb segmentation algorithm to exclude false positive non-lesions.

[0037] Conclusion Recall that the Felzenszwalb image segmentation algorithm was found to provide a good basic diagnostic algorithm that can effectively calculate the body surface area of a patient's lesion. It was also discovered that a convolutional neural network can accurately classify the proposed lesions and non-lesions from the Felzenszwalb segmentation output when given a good training dataset. Combining these two results, the Felzenszwalb image segmentation algorithm combined with a convolutional neural network filter was proven to perform excellent basic diagnosis by image segmentation and thereby calculating the body surface area score of psoriasis disease.

[0038] Using the BSA calculation method described above, a digitized psoriasis disease score calculation system can be generated. For example, in order to automate the full PASI scoring system, by training a similar convolutional neural network with five input images of the anterior body, back, front legs, hind legs, and head regions, a PASI score of severity index from 0 to 72 can be output. Such a computer system can, as proposed in recent years (C. Fink, L. Uhlmann, C. Klose, et al (2018) “Automated, computer-guided PASI measurements by digital image analysis versus conventional physicians’ PASI calculations: study protocol for a comparative, single-centre, observational study BMJ Open 2018;8:e 018461.doi:10.1136 / bmjopen-2017-018461), assist physicians in making better, quicker, and more informed decisions in the diagnosis and monitoring of skin diseases such as psoriasis.

[0039] Those skilled in the art will understand that modifications can be made to the embodiments described above without departing from the broad inventive concept. Therefore, it is understood that the present invention is not limited to the specific embodiments disclosed, but encompasses modifications within the spirit and scope of the present invention.

Claims

1. A method processed by a computer for determining the severity of a skin disease based on the percentage of the body surface area (BSA) occupied by a lesion, the trained neural network being at least (i) a step of performing image segmentation on a plurality of training set images of BSA using a segmentation algorithm, each of the plurality of training set images of BSA including a skin disease, and the image segmentation performed using the segmentation algorithm outputting a proposed region of the lesion in each of the plurality of training set images, (ii) a step of manually classifying each of the proposed regions as a lesion or non-lesion, (iii) a step of training a neural network using the classified proposed regions in each of the plurality of training set images, and generated by performing The method being (a) a step of performing image segmentation on a test image of BSA including a skin disease using the segmentation algorithm, the image segmentation performed using the segmentation algorithm outputting a plurality of regions in the test image of BSA, (b) a step of inputting the regions of the test image into the trained neural network, (c) using the trained neural network to identify and filter non-lesion regions from the test image, such that the remaining regions of the test image are classified as lesion regions, (d) a step of calculating the percentage of BSA in the test image occupied by the lesion using the area of the classified lesion regions of the test image and the area of the identified non-lesion regions of the test image. A method including.

2. The method according to claim 1, wherein both the training set image and the test image are over-segmented.

3. The method according to claim 1, wherein the neural network is a convolutional neural network.

4. Before the step (iii) of training, a plurality of regions of the plurality of training set images are resized to an image interpolation size of about 16×16 pixels, and before the step (b), the plurality of regions of the test image are resized to an image interpolation size of about 16×16 pixels. The method according to claim 1.

5. The method according to claim 1, wherein the skin disease is psoriasis.

6. A computer system including a non-transitory computer-readable medium storing instructions that, when executed by a processor, perform each step of the method according to claim 1.

7. The method according to claim 1, wherein the segmentation algorithm is the Felzenszwalb segmentation algorithm.

8. Step (ii) is over-segmenting each of the plurality of training set images; manually classifying each of the proposed regions of each of the plurality of over-segmented training set images as a lesion or non-lesion; The method according to claim 1, including.

9. The method according to claim 1, further including a step of over-segmenting the test image before inputting the plurality of regions of the test image into the trained neural network.

10. The method according to claim 2, wherein both the training set image and the test image are over-segmented by increasing a k-value parameter to about 250.

11. The method according to claim 8, wherein the segmentation algorithm is the Felzenszwalb segmentation algorithm.

12. The method according to claim 9, wherein the segmentation algorithm is the Felzenszwalb segmentation algorithm.

13. The method according to claim 8, further comprising the step of oversegmenting the test image before inputting the plurality of regions of the test image into the trained neural network.

Images