Dermatosis Assistant Diagnosis Platform Based on VGG-16 Integrated with Residual Network

Abstract

The present invention relates to the identification and diagnosis of skin diseases, and particularly to a skin disease auxiliary diagnosis platform based on a VGG-16 fusion residual network, which includes a hair removal module and an identification and classification module. The identification and classification module uses a skin disease identification model based on a VGG-16 fusion residual network to identify and classify skin diseases in images. The skin disease identification model based on a VGG-16 fusion residual network includes a preprocessing layer CRBM, a convolutional layer module, an average pooling layer, a fully connected layer, and a classifier that are arranged in sequence. The preprocessing layer CRBM includes a convolutional layer, an activation function, a Batch Normalization layer, and a max pooling layer. A residual network is added to the backend of each convolutional layer module, and a Batch Normalization layer is added to replace the max pooling layer in the last convolutional layer module. A dropout layer is arranged between the fully connected layer and the classifier. The technical solution provided by the present invention can effectively overcome the defect that the prior art cannot accurately identify and classify skin lesion images.

Description

Technical Field

[0001] The present invention relates to the identification and diagnosis of skin diseases, and specifically to a skin disease auxiliary diagnosis platform based on the VGG-16 fusion residual network. Background Art

[0002] Skin diseases are the most common malignant diseases today and have a high carcinogenic rate. The most fatal skin disease is melanoma. Although the mortality rate in the late stage is very high, if it can be detected in time and treated as soon as possible, the survival rate is over 95%. Therefore, achieving early diagnosis of skin diseases is crucial for improving the survival rate of skin disease patients.

[0003] With the development of computer image processing technology, computer-aided diagnosis (CAD) systems have been widely used in medical image processing, which can realize the classification based on dermoscopic images, improve the efficiency and accuracy of diagnosis, and doctors and experts can also prescribe medications according to the special clinical manifestations of different lesions, which has important practical significance and clinical research value.

[0004] In traditional machine learning algorithms, analyzing the contour of lesions has a positive effect on classification and diagnosis. In recent years, deep learning has made breakthroughs in the field of medical image processing and has been successfully applied to the field of medical image classification and diagnosis. The classification of skin lesion images based on deep learning does not rely on prior knowledge, and CNN is often in an "end-to-end" mode, that is, when an image of skin lesions is input, the category of skin lesions can be output, which has higher accuracy and working efficiency than traditional machine learning algorithms.

[0005] Applying deep learning to the classification and diagnosis of skin diseases is a very practical and valuable issue, but it is also very challenging. Therefore, this research not only links the medical and computer directions together to assist doctors in diagnosis, but also closely combines the cutting-edge technology of the era, deep learning, with medical image recognition and classification. Conducting in-depth research on the problem of skin lesion image recognition and classification using neural networks, exploring technical routes with higher recognition and classification accuracy, and proposing some new recognition and classification algorithms and technologies undoubtedly have important theoretical significance and practical application value. Summary of the Invention

[0006] Aiming at the above-mentioned shortcomings of the existing technology, the present invention provides a skin disease auxiliary diagnosis platform based on the VGG-16 fusion residual network, which can effectively overcome the defect that the existing technology cannot accurately identify and classify skin lesion images.

[0007] To achieve the above objectives, the present invention is realized through the following technical solutions:

[0008] A skin disease assisted diagnosis platform based on the VGG-16 fusion residual network, including a hair removal module and an identification and classification module. The hair removal module processes the captured image to remove hair and sends the processed image to the identification and classification module. The identification and classification module uses a skin disease identification model based on the VGG-16 fusion residual network to identify and classify skin diseases in the image;

[0009] The skin disease identification model based on the VGG-16 fusion residual network includes a preprocessing layer CRBM, a convolutional layer module, an average pooling layer, a fully connected layer, and a classifier arranged in sequence. The preprocessing layer CRBM includes a convolutional layer, an activation function, a Batch Normalization layer, and a max pooling layer. A residual network is added to the backend of each convolutional layer module, and a Batch Normalization layer is added to replace the max pooling layer in the last convolutional layer module. A dropout layer is set between the fully connected layer and the classifier;

[0010] The convolutional layer in the preprocessing layer CRBM is used to obtain rich feature maps. The ReLU activation function can change the linear structure of the model, enhance non-linearity, and reduce overfitting. The Batch Normalization layer normalizes the image, and the most discriminative intermediate semantic features of the lesion area are extracted through the max pooling layer;

[0011] The convolutional layer module includes a convolutional layer and a max pooling layer, and there are two fully connected layers.

[0012] Preferably, the training method of the skin disease identification model includes:

[0013] S1. Collect a skin lesion image dataset, adjust the image size, and divide the skin lesion image dataset into a training set, a validation set, and a test set;

[0014] S2. Remove the hair of the skin lesion images in the training set;

[0015] S3. Perform data augmentation on the skin lesion images in the training set to balance the number of training sets corresponding to various skin diseases;

[0016] S4. Input the processed training set into the skin disease identification model for model training;

[0017] S5. Save the model when the loss value of the validation set does not change for a period of time, and complete the training of the skin disease identification model.

[0018] Preferably, removing the hair of the skin lesion images in the training set in S2 includes:

[0019] S21. Convert the RGB source image to a grayscale image;

[0020] S22. Detect the hair contour in the grayscale image using the blackHat operation, create a mask based on the detected hair contour, where the mask recalculates each pixel according to the weighted average of the surrounding pixels, making the hair area more prominent;

[0021] S23. Further highlight the hair area through binarization and erosion operations;

[0022] S24. Apply the image inpainting technique based on the fast marching method (FMM), remove the mask that only contains the hair area from the source image, and replace the removed pixels with the pixels adjacent to the hair area.

[0023] Preferably, in S3, data augmentation is performed on the skin lesion images in the training set to balance the number of training sets corresponding to various skin diseases, including:

[0024] In the range of 0° - 180°, data augmentation methods including channel transformation, flipping, size adjustment, random rotation, horizontal and vertical displacement, and random proportional scaling are used to perform data augmentation on the training set skin lesion images of non-NV diseases.

[0025] Preferably, the hair removal module performs hair removal processing on the captured image, including:

[0026] Convert the RGB source image to a grayscale image;

[0027] Detect the hair contour in the grayscale image using the blackHat operation, create a mask based on the detected hair contour, where the mask recalculates each pixel according to the weighted average of the surrounding pixels, making the hair area more prominent;

[0028] Further highlight the hair area through binarization and erosion operations;

[0029] Apply the image inpainting technique based on the fast marching method (FMM), remove the mask that only contains the hair area from the source image, and replace the removed pixels with the pixels adjacent to the hair area.

[0030] Compared with the prior art, the skin disease auxiliary diagnosis platform based on the VGG-16 fusion residual network provided by the present invention has the following beneficial effects:

[0031] 1) Use morphological operations for hair removal processing, making the separation between the lesion area and the background area clearer. Whether the network model for recognition and classification or the doctor examines the lesion image, only the lesion area is concerned, which can well improve the classification and diagnosis effects and effectively improve the accuracy of skin disease recognition and classification;

[0032] 2) Use VGG-16 as the backbone network, integrate the residual network, reduce the number of convolutional layers in the neural network, and lightweight the preprocessing layer. On this basis, improve the accuracy of skin disease recognition and classification. After the image is sent into the recognition model, the classification result is output for doctors' reference, aiming to directly rely on the recognition model for diagnosis and treatment without the need for doctors to further study and discuss the lesion type, simplifying the medical process and treatment procedures, and realizing the integration of the medical assistance system;

[0033] 3) Improve and enhance the performance of the existing neural network structure, improve the accuracy of recognition and classification, form a complete platform, and the mobile APP can be downloaded and used, enabling the client to scan or take pictures of the lesion area to identify the lesion type, assisting doctors in diagnosing skin diseases, and improving the survival rate of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0035] Figure 1 is a schematic flowchart of the present invention;

[0036] Figure 2 is a schematic diagram of the effect of hair removal treatment in the present invention;

[0037] Figure 3 is a schematic diagram of data augmentation for skin lesion images in the training set in the present invention;

[0038] Figure 4 is a schematic diagram of the structure of the existing VGG-16 network model;

[0039] Figure 5 is a schematic diagram of the structure of the skin disease recognition model based on VGG-16 integrated with the residual network in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0041] Such as Figure 4As shown, the existing VGG-16 network model mainly includes 5 convolutional layer modules. Each convolutional layer module contains 2 or 3 convolutional layers, and a max pooling layer. As is well known, increasing the number of convolutional layers can improve the classification ability of the model. However, adding more convolutional layers requires more training data and higher computing power, which will significantly increase the complexity of training. Based on the above principle, a skin disease recognition model based on VGG-16 fused with a residual network is designed to reduce the model complexity. Taking VGG-16 as the backbone network and fusing the residual network, a recognition and classification model for skin lesions is proposed.

[0042] The skin disease recognition model based on VGG-16 fused with a residual network, as Figure 5 shown, includes a preprocessing layer CRBM, a convolutional layer module, an average pooling layer, a fully connected layer, and a classifier arranged in sequence. The preprocessing layer CRBM includes a convolutional layer, an activation function, a Batch Normalization layer, and a max pooling layer. A residual network is added to the backend of the convolutional layer module, and a Batch Normalization layer is added to replace the max pooling layer in the last convolutional layer module. A dropout layer is set between the fully connected layer and the classifier.

[0043] The convolutional layer in the preprocessing layer CRBM is used to obtain rich feature maps. The ReLU activation function can change the linear structure of the model, enhance non-linearity, reduce overfitting. The Batch Normalization layer normalizes the images, and the most discriminative intermediate semantic features of the lesion area are extracted through the max pooling layer.

[0044] The convolutional layer module includes a convolutional layer and a max pooling layer, and there are two fully connected layers.

[0045] In the technical solution of this application, the skin disease recognition model based on VGG-16 fused with a residual network reduces the number of convolutional layers, adds a lightweight preprocessing layer CRBM, and combines the residual network to reduce the model complexity. Specifically, the improvements of the skin disease recognition model based on VGG-16 fused with a residual network are as follows:

[0046] (1) A preprocessing layer CRBM is introduced, and its structure is "convolutional layer + ReLU + Batch Normalization layer + Maxpooling layer". Among them, the 3×3 convolutional layer is used to obtain rich feature maps (color and texture features), the ReLU activation function can change the linear structure of the model, enhance non-linearity, and reduce overfitting. The Batch Normalization layer normalizes the images, and the Maxpooling layer is used to extract the most discriminative intermediate semantic features of the lesion area. The preprocessing layer CRBM can reduce the interference from non-lesion areas during the initial processing of the model and solve the problem of incomplete feature extraction by the neural network.

[0047] (2) Based on the existing VGG-16 network model, a ResNet residual network is added to the backend of each convolutional layer module. As the network deepens, the gradient will disappear and pathological features will gradually be lost. Adding the ResNet residual network can well prevent problems such as gradient explosion or gradient disappearance.

[0048] For skin lesion images, we must explore fine-grained local features (such as the boundary between the lesion area and the background area) to ensure accurate classification, especially the contour description with lesion trends. Therefore, the feature loss between different convolutional layer modules can be reduced through skip connection operations. Skip connection operations increase the ability of gradient cross-layer propagation and further improve the recognition and classification accuracy of the model.

[0049] (3) It should be noted that a Batch Normalization layer is added to replace the Maxpooling layer in the last convolutional layer module of the existing VGG-16 network model. This operation shortens the feature distance between similar skin lesions and reduces the loss of disease feature information during the downsampling process. After the last convolution, the image features are directly sent to the average pooling layer to reduce the feature dimension.

[0050] (4) In order to reduce the number of parameters of the recognition model, only two fully connected layers are used, and the number of parameters of the neurons is set to 512. After the second fully connected layer, a dropout layer with a dropout rate of 0.6 is added. The softmax function is used as the classifier to classify skin diseases, and the number of neurons in the output layer is set to 7, representing the classification probabilities of 7 skin diseases (the 7 skin diseases are as Figure 1 shown in the lower right corner).

[0051] The training method of the skin disease recognition model includes:

[0052] S1. Collect a skin lesion image dataset, resize the images (resize the images to 224×224×3), and divide the skin lesion image dataset into a training set, a validation set, and a test set according to the ratio of 7:1:2;

[0053] S2. Remove the hair from the skin lesion images in the training set;

[0054] S3. Perform data augmentation on the skin lesion images in the training set to balance the number of training sets corresponding to various skin diseases;

[0055] S4. Input the processed training set into the skin disease recognition model for model training;

[0056] S5. Save the model when the loss value of the validation set (using the focal loss FL as the loss function of the model, suitable for multi-classification tasks) has not changed for 5 consecutive times, and complete the training of the skin disease recognition model.

[0057] Among them, removing the hair from the skin lesion images in the training set includes:

[0058] S21. Convert the RGB source image to a grayscale image;

[0059] S22. Use the blackHat operation to detect the hair contour in the grayscale image, create a mask based on the detected hair contour, and the mask is obtained by recalculating each pixel according to the weighted average of the surrounding pixels, making the hair area more prominent;

[0060] S23. Further highlight the hair area through binarization and erosion operations;

[0061] S24. Apply the image inpainting technique based on the fast marching method FMM to delete the mask containing only the hair area from the source image, and replace the removed pixels with the pixels adjacent to the hair area.

[0062] The effect of hair removal processing is as Figure 2 shown, and we can see that the hair noise on the skin lesion image is almost completely removed.

[0063] Among them, performing data augmentation on the skin lesion images in the training set to balance the number of training sets corresponding to various skin diseases includes:

[0064] In the range of 0° - 180°, for the training set skin lesion images of non-NV diseases, 6 data augmentation methods including channel transformation (randomly translating the channels by a random value), flipping, resizing, random rotation, horizontal and vertical displacement, and random scale scaling are used for data augmentation.

[0065] As Figure 3As shown, examples corresponding to the above six data augmentation methods are displayed. Through the above data augmentation methods, similar but different training samples can be generated, effectively expanding the training set and balancing the number of training sets corresponding to seven skin diseases.

[0066] In the technical solution of this application, an auxiliary diagnosis platform is also disclosed, including a hair removal module and an identification and classification module. The hair removal module processes the captured image to remove hair and sends the processed image to the identification and classification module. The identification and classification module uses a skin disease identification model based on the VGG-16 fusion residual network to identify and classify skin diseases in the image.

[0067] The skin disease identification model based on the VGG-16 fusion residual network includes a preprocessing layer CRBM, a convolutional layer module, an average pooling layer, a fully connected layer, and a classifier arranged in sequence. The preprocessing layer CRBM includes a convolutional layer, an activation function, a BatchNormalization layer, and a max pooling layer. A residual network is added to the backend of the convolutional layer module, and a Batch Normalization layer is added to replace the max pooling layer in the last convolutional layer module. A dropout layer is set between the fully connected layer and the classifier.

[0068] Among them, the hair removal module processes the captured image to remove hair, including:

[0069] Converting the RGB source image to a grayscale image;

[0070] Using the blackHat operation to detect the hair contour in the grayscale image, creating a mask based on the detected hair contour. The mask recalculates each pixel based on the weighted average of surrounding pixels, making the hair area more prominent;

[0071] Further highlighting the hair area through binarization and erosion operations;

[0072] Applying an image inpainting technique based on the fast marching method FMM to delete the mask containing only the hair area from the source image and replace the removed pixels with pixels adjacent to the hair area.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A skin disease assisted diagnosis platform based on the fusion of VGG-16 and residual network, characterized in that: It includes a hair removal module and an identification and classification module. The hair removal module processes the captured image to remove hair and sends the processed image to the identification and classification module. The identification and classification module uses a skin disease identification model based on the VGG-16 fusion residual network to identify and classify skin diseases in the image; The skin disease identification model based on the VGG-16 fusion residual network includes a preprocessing layer CRBM, a convolutional layer module, an average pooling layer, a fully connected layer, and a classifier arranged in sequence. The preprocessing layer CRBM includes a convolutional layer, an activation function, a Batch Normalization layer, and a max pooling layer. A residual network is added to the backend of each convolutional layer module. A Batch Normalization layer is added in the last convolutional layer module to replace the max pooling layer. A dropout layer is set between the fully connected layer and the classifier; The convolutional layer in the preprocessing layer CRBM is used to obtain rich feature maps. The ReLU activation function can change the linear structure of the model, enhance non-linearity, and reduce overfitting. The Batch Normalization layer normalizes the image, and the most discriminative intermediate semantic features of the lesion area are extracted through the max pooling layer; The convolutional layer module includes a convolutional layer and a max pooling layer. There are two fully connected layers.

2. The skin disease assisted diagnosis platform based on the VGG-16 fusion residual network according to claim 1, wherein: The training method of the skin disease identification model includes: S1. Collect a skin lesion image dataset, adjust the image size, and divide the skin lesion image dataset into a training set, a validation set, and a test set; S2. Remove the hair from the skin lesion images in the training set; S3. Perform data augmentation on the skin lesion images in the training set to balance the number of training sets corresponding to various skin diseases; S4. Input the processed training set into the skin disease identification model for model training; S5. Save the model when the loss value of the validation set does not change for a period of time, and complete the training of the skin disease identification model.

3. The skin disease assisted diagnosis platform based on the VGG-16 fusion residual network according to claim 2, wherein: Removing the hair from the skin lesion images in S2 includes: S21. Convert the RGB source image to a grayscale image; S22. Use the blackHat operation to detect the hair contour in the grayscale image, create a mask based on the detected hair contour. The mask recalculates each pixel based on the weighted average of the surrounding pixels, making the hair area more prominent; S23. Further highlight the hair area through binarization and erosion operations; S24. Apply an image inpainting technique based on the fast marching method (FMM) to delete the mask that only contains the hair area from the source image and replace the removed pixels with the pixels adjacent to the hair area.

4. The skin disease assisted diagnosis platform based on the VGG-16 fusion residual network according to claim 2, characterized in that: Performing data augmentation on the skin lesion images in the training set in S3 to balance the number of training sets corresponding to various skin diseases includes: In the range of 0° - 180°, data augmentation methods including channel transformation, flipping, size adjustment, random rotation, horizontal and vertical displacement, and random scale scaling are used to perform data augmentation on the training set skin lesion images of non-NV diseases.

5. The skin disease assisted diagnosis platform based on the VGG-16 fusion residual network according to claim 1, characterized in that: The hair removal module processes the captured image to remove hair, including: Convert the RGB source image to a grayscale image; Detect the hair contour in the grayscale image using blackHat operation, create a mask based on the detected hair contour, where the mask is obtained by recalculating each pixel according to the weighted average of the surrounding pixels, making the hair area more prominent; Further highlight the hair area through binarization and erosion operations; Apply the image inpainting technique based on the Fast Marching Method (FMM), delete the mask that only contains the hair area from the source image, and replace the removed pixels with the pixels adjacent to the hair area.

Images