Skin image classification method, device and product based on hair removal network

Abstract

The invention provides a skin image classification method and device based on a hair removal network and a product, and relates to the technical field of image classification, and the method comprises the steps: carrying out the hair simulation of each first skin image in a first skin image data set, and obtaining a hair simulation image; training a hair removal network by using the hair simulation image to obtain an initial hair removal network; training a skin image classification network by using the first skin image data set to obtain an initial skin image classification network; training the initial hair removal network and the initial skin image classification network by using the second skin image data set to obtain a trained hair removal network and a trained skin image classification network; and inputting a to-be-detected skin image into the trained hair removal network to obtain a processed detection image after hair removal, and inputting the processed detection image into the trained skin image classification network to obtain a skin classification result of the to-be-detected image.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to a method, device and product for skin image classification based on a hair removal network. Background Art

[0002] In the study of skin lesion classification and recognition, classification based on dermoscopic images is an important task. With the development of deep learning technology, the method of using neural networks to train labeled data to extract skin lesion features has become mainstream.

[0003] However, there are often other interference factors (such as hair interference) in skin images, which leads to irrelevant features learned by the neural network model, thereby affecting the accuracy of the final classification detection (i.e., skin lesion type identification). Therefore, it is urgent to propose a skin image classification method, device and product based on hair removal network to improve the classification accuracy of skin images. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a skin image classification method, device and product based on a hair removal network to overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of an embodiment of the present application, a skin image classification method based on a hair removal network is provided, the method comprising: Acquire a first skin image dataset and a second skin image dataset, wherein the skin in each first skin image in the first skin image dataset is hairless, and the skin in each second skin image in the second skin image dataset is hairy; Performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image; Using the hair simulation image and the corresponding first skin image, the hair removal network is trained to obtain an initial hair removal network; Using the first skin image dataset, training a skin image classification network to obtain an initial skin image classification network; Using the second skin image data set, training the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network; The skin image to be detected is input into the trained hair removal network to obtain a processed detection image after hair removal, and the processed detection image is input into the trained skin image classification network to obtain a skin classification result of the image to be detected.

[0006] In a possible implementation, performing hair simulation on each first skin image in the first skin image dataset to obtain a hair simulation image includes: On the blank image, a Bezier curve algorithm is used to generate a random curve to obtain a simulated hair image; The simulated hair image is fused with the first skin image to obtain the hair simulation image.

[0007] In a possible implementation, the network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a feature fusion module based on dynamic snake convolution to replace the convolution module in the UNet network architecture.

[0008] In a possible implementation, the using the hair simulation image and the corresponding first skin image to train the hair removal network to obtain an initial hair removal network includes: Inputting the hair simulation image into the hair removal network to generate a first hair-removed image; Calculating the mean absolute error loss between the first hair-removed image and the corresponding first image; updating parameters of the hair removal network according to the mean absolute error loss; Repeat the above steps until the preset number of training times is reached, and then end the training to obtain the initial hair removal network.

[0009] In a possible implementation, the using the first skin image dataset to train a skin image classification network to obtain an initial skin image classification network includes: Taking the real type of skin in the first skin image as a skin type label, inputting the first skin image into the skin image classification network to obtain a skin classification prediction result for the first skin image; Calculating a first cross entropy loss according to the skin classification prediction result for the first skin image and the skin type label of the first skin image; According to the first cross entropy loss, updating parameters of the skin image classification network; Repeat the above steps until the preset number of training times is reached, and then terminate the training to obtain the initial skin image classification network.

[0010] In a possible implementation, the using the second skin image data set to train the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network includes: Inputting the second skin image in the second skin image dataset into the initial hair removal network to generate a second hair-removed image; Inputting the second hair-removed image into the initial skin image classification network to obtain a skin classification prediction result for the second skin image; Taking the actual type of the skin in the second skin image as the skin type label of the second skin image, calculating a second cross entropy loss between the skin classification prediction result for the second skin image and the skin type label of the second skin image; According to the second cross entropy loss, updating parameters of the initial hair removal network and the initial skin image classification network; Repeat the above steps until the preset number of training times is reached, and then end the training to obtain the trained hair removal network and the trained skin image classification network.

[0011] In a possible implementation, the skin types in the first skin image or the second skin image include at least: actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign pigmented keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0012] The second aspect of the embodiment of the present application further proposes a skin image classification device, the device comprising: An image acquisition module, configured to acquire a first skin image data set and a second skin image data set, wherein the skin in each first skin image in the first skin image data set is hairless, and the skin in each second skin image in the second skin image data set is hairy; a hair simulation module, configured to perform hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image; A first training module, configured to train the hair removal network using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network; A second training module is used to train a skin image classification network using the first skin image dataset to obtain an initial skin image classification network; A third training module is used to train the initial hair removal network and the initial skin image classification network using the second skin image data set to obtain a trained hair removal network and a trained skin image classification network; The skin image classification module is used to input the skin image to be detected into the trained hair removal network to obtain a processed detection image after hair removal, and input the processed detection image into the trained skin image classification network to obtain the skin classification result of the image to be detected.

[0013] The third aspect of the embodiments of the present application also provides an electronic device, including a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps in the skin image classification method based on hair removal network described in the first aspect of the embodiments of the present application.

[0014] The fourth aspect of the embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the skin image classification method based on hair removal network described in the first aspect of the embodiment of the present application are implemented.

[0015] The fifth aspect of the embodiments of the present application also provides a computer program product, which, when running on an electronic device, enables a processor to implement the steps in the skin image classification method based on hair removal network as described in the first aspect of the embodiments of the present application.

[0016] An embodiment of the present application provides a skin image classification method based on a hair removal network, the method comprising: obtaining a first skin image data set and a second skin image data set, wherein the skin in each first skin image in the first skin image data set does not have hair, and the skin in each second skin image in the second skin image data set has hair; performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image; using the hair simulation image and the corresponding first skin image to train the hair removal network to obtain an initial hair removal network; using the first skin image data set to train a skin image classification network to obtain an initial skin image classification network; using the second skin image data set to train the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network; inputting the skin image to be detected into the trained hair removal network to obtain a processed detection image after hair removal, and inputting the processed detection image into the trained skin image classification network to obtain a skin classification result of the image to be detected.

[0017] The specific beneficial effect is that in order to reduce the interference of hair in skin images on skin classification, the embodiment of the present application proposes combining hair removal and skin disease recognition (i.e., combining a hair removal network and a skin image classification network). In the process of constructing the above two networks, a total of two stages of training are performed: first, the hair removal network and the skin image classification network are trained separately using the first skin image data set (the skin in the first skin image does not have hair); then, the hair removal network and the skin image classification network trained in the first stage are trained together using the second skin image data set (the skin in the second skin image has hair), so as to further improve the performance of the two networks. The trained hair removal network can accurately remove the hair in the skin image without significantly changing the characteristics of the original skin lesions, and then the image after hair removal is input into the skin image classification network, so as to obtain a more accurate skin classification result (i.e., the image to be detected is classified to obtain the category to which the skin in the image belongs). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flowchart of the steps of a skin image classification method based on a hair removal network provided in an embodiment of the present application; Figure 2 is a schematic diagram of a curve shape of a Bezier curve provided in an embodiment of the present application; Figure 3 is a schematic diagram of simulated hair generated by a Bezier curve algorithm provided in an embodiment of the present application; Figure 4 is a schematic diagram of a network architecture of a hair removal network provided in an embodiment of the present application; Figure 5 It is a structural diagram of a feature fusion module provided in an embodiment of the present application; Figure 6 is a structural schematic diagram of a skin image classification device provided in an embodiment of the present application; Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0021] In the study of skin lesion classification and recognition, classification based on dermoscopic images is a routine and important task. With the development of deep learning technology, the method of using neural networks to train labeled data to extract skin lesion features has become mainstream. However, hair interference often occurs in dermoscopic images, which is an irrelevant feature that is not desired to be learned by the neural network model during the skin lesion recognition process. Therefore, before classification and recognition, the skin lesion images containing hair can be processed by hair removal, and then further classification and recognition can be performed using deep neural networks. When the skin lesion images after hair removal are classified and recognized again, their accuracy will inevitably be improved.

[0022] Generally speaking, for unsupervised cross-domain image conversion tasks, some unsupervised image-to-image conversion algorithms (such as CycleGAN) are usually used to complete them. However, in actual experiments, it was found that CycleGAN often significantly changes the characteristics of the original skin lesions while removing hair. However, this change is unacceptable in medical image processing, because the obvious change of skin lesions will have a great impact on the diagnosis and identification of skin lesion types, resulting in the final skin image classification accuracy being too low.

[0023] In view of the above problems, the embodiments of the present application propose a skin image classification method, device and product based on a hair removal network to improve the classification accuracy of skin images. The following is a detailed description of a skin image classification method based on a hair removal network provided by the embodiments of the present application through some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0024] The first aspect of the embodiment of the present application provides a skin image classification method based on a hair removal network. The overall technical concept of the method is: in order to reduce the interference of hair in skin images on skin classification, it is proposed to combine hair removal and skin disease recognition (that is, combine hair removal network and skin image classification network). In the process of constructing the above two networks, a total of two stages of training are performed: first, the hair removal network and the skin image classification network are trained separately using the first skin image data set (the skin in the first skin image does not have hair); then, the hair removal network and the skin image classification network trained in the first stage are trained together using the second skin image data set (the skin in the second skin image has hair), so as to further improve the performance of the two networks. Finally, the trained hair removal network and skin image classification network are used to realize the classification and recognition of the input skin image. The specific process is described in the following embodiments.

[0025] Reference Figure 1 , Figure 1 A flowchart of a method for classifying skin images based on a hair removal network is provided in an embodiment of the present application, such as Figure 1 As shown, the method includes: Step S101, obtaining a first skin image dataset and a second skin image dataset, wherein the skin in each first skin image in the first skin image dataset is hairless, and the skin in each second skin image in the second skin image dataset is hairy.

[0026] First, in order to train the hair removal network and the skin image classification network, it is necessary to first collect a large number of skin images (i.e., dermoscopic images) and determine the skin type corresponding to each skin image. In this embodiment, the skin images are divided into two categories, the first skin image and the second skin image. The skin in the first skin image does not have hair (i.e., a skin disease image without hair interference), and the skin in the second skin image has hair (i.e., a skin disease image with hair interference), thereby forming a first skin image data set and a second skin image data set respectively. Among them, there is no one-to-one correspondence between the first skin image and the second skin image.

[0027] For each image (first skin image or second skin image), a corresponding skin type label needs to be pre-set. The skin type label indicates the real type of the skin in the image. Specifically, the skin type (skin type label) includes at least: actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign pigmented keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0028] In a possible implementation, the skin types in the first skin image or the second skin image include at least: actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign pigmented keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0029] After obtaining abundant training data (a first skin image dataset and a second skin image dataset), the embodiment of the present application firstly performs the first stage of training on the hair removal network and the skin image classification network respectively using the first skin image dataset (the skin in the first skin image does not have hair) through steps S102-S104.

[0030] Step S102: performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image.

[0031] Specifically, when the first skin image data set is used to train the hair removal network in the first stage, the first skin image (the skin in the image does not have hair) is used as the training label, and hair simulation is first performed on the first skin image, and the hair on the skin in the first skin image is simulated using the Bezier curve algorithm to obtain a hair simulation image. The hair simulation image is then input into the hair removal network to be trained, so that the hair removal network learns to remove the hair on the skin in the image, so that the image finally output by the hair removal network becomes closer and closer to the corresponding label (the first skin image).

[0032] In a possible implementation manner, the step S102 of performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image includes: Step S1021, using the Bezier curve algorithm on the blank image, generating a random curve to obtain a simulated hair image.

[0033] Step S1022: fusing the simulated hair image with the first skin image to obtain the hair simulation image.

[0034] This embodiment proposes a method for simulating hair generation based on Bezier curves, which simulates curve lines similar to hair through Bezier curves, and then adds the generated hair to a hairless skin disease image (first skin image), thereby simulating a skin image with hair (hair simulation image), so that the hair simulation image and the original hairless first skin image form a pair of training data samples.

[0035] Specifically, the Bezier curve is a smooth curve drawn by using the coordinates of any point at any position proposed by Pierre Bézier, and is widely used in design and graphics generation tasks. The shape of the function curve generated by the quadratic Bezier curve can be used as the simulated hair shape. Among them, the parametric equation of the quadratic Bezier curve can be expressed as: ; Among them, B is the trajectory point for generating the quadratic Bezier curve, and P0, P1, and P2 are the initial coordinate points of three arbitrary positions in the blank image. Figure 2 is a schematic diagram of the curve shape of a Bezier curve, such as Figure 2 As shown, as the parameter t changes, the trajectory of point B forms a quadratic Bezier curve, which is similar to the shape of hair.

[0036] Based on the principle of Bezier curve construction, the embodiment of the present application firstly generates one or more lines of random shapes based on the blank image by using the Bezier curve algorithm through step S1021, and adjusts the thickness of the lines within a certain threshold range as the simulated hair shape, thereby obtaining an image containing multiple simulated hairs, such as Figure 3 As shown, Figure 3 A schematic diagram of simulated hair generated by a Bezier curve algorithm is shown. Then, in step S1022, the simulated hair is merged with the first skin image, and the original first skin image without hair is converted into a hair simulation image with hair on the skin.

[0037] Step S103: using the hair simulation image and the corresponding first skin image, training the hair removal network to obtain an initial hair removal network.

[0038] Specifically, based on the first skin image A, a corresponding hair simulation image A' is generated, and then the hair simulation image A' is input into the hair removal network to be trained with the first skin image A as a label, so that the hair removal network learns to remove the hair on the skin in the image, and the image finally output by the hair removal network becomes closer and closer to the corresponding label (the first skin image A), thereby obtaining the initial hair removal network that has completed the first stage of training.

[0039] In a possible implementation, the network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a feature fusion module based on dynamic snake convolution to replace the convolution module in the UNet network architecture.

[0040] In the embodiment of the present application, the hair removal network can be constructed based on the UNet architecture. Figure 4 , Figure 4A schematic diagram of the network architecture of a hair removal network is shown, Figure 4 As shown, the hair removal network is composed of multiple convolution modules to form a U-shaped network architecture, wherein arrow ① indicates that after the convolution module performs the corresponding convolution operation, a maximum pooling operation is performed, arrow ② indicates that after the convolution module performs the corresponding convolution operation, a bilinear interpolation operation is performed, and the dotted arrow indicates a shortcut connection operation. Furthermore, in the UNet architecture, the embodiment of the present application uses a dynamic snake convolution based feature fusion (DSC-FF) module to replace the ordinary convolution operation. Specifically, refer to Figure 5 , Figure 5 A structural diagram of a feature fusion module is shown, Figure 5 As shown in the figure, in the feature fusion module (i.e., DSC-FF module), three branches of 3×3 ordinary convolution, 9×1 and 1×9 dynamic snake convolution are used. After each branch passes through group normalization (GroupNorm) and ReLU activation function, the features of the three branches are concatenated together and then output after passing through a 3×3 ordinary convolution layer, group normalization (GroupNorm) and ReLU activation function.

[0041] In a possible implementation, the step S103, using the hair simulation image and the corresponding first skin image to train the hair removal network to obtain an initial hair removal network, includes: Step S1031, inputting the hair simulation image into the hair removal network to generate a first hair-removed image.

[0042] Step S1032, calculating the mean absolute error loss between the first hair-removed image and the corresponding first image.

[0043] Step S1033: updating parameters of the hair removal network according to the mean absolute error loss.

[0044] Step S1034, repeat the above steps until the preset training times are reached, and then end the training to obtain the initial hair removal network.

[0045] In the embodiment of the present application, based on the first skin image A, a corresponding hair simulation image A' is generated, and then the first skin image A is used as a label, and the hair simulation image A' is input into the hair removal network to be trained to remove hair and obtain the first hair removal image B. Then, the loss function value is calculated according to the output result (the first hair removal image B) and the label (the first skin image A). In the embodiment of the present application, the output result is closer and closer to the label as the goal, and the mean absolute error loss (MAE) between the two is used as the loss function to update the parameters of the hair removal network according to the calculated loss function value, which is regarded as completing a round of training for the hair removal network. Then, a new hair simulation image and the corresponding first skin image are selected, and the above steps S1031-S1033 are repeated to perform the next round of training for the hair removal network until the preset number of training times is reached, and the first stage of training for the hair removal network is completed to obtain the initial hair removal network.

[0046] Step S104: using the first skin image data set, training a skin image classification network to obtain an initial skin image classification network.

[0047] Specifically, based on the first skin image A, taking the type of skin in the first skin image A as a label, the first skin image A is input into the skin image classification network to be trained, so that the skin image classification network learns to classify and predict the skin in the image, and the skin type finally predicted by the skin image classification network becomes closer and closer to the corresponding label (the actual type of skin in the first skin image A), thereby obtaining an initial skin image classification network that has completed the first stage of training.

[0048] In a possible implementation, the step S104, using the first skin image dataset to train a skin image classification network to obtain an initial skin image classification network, includes: Step S1041, taking the actual type of skin in the first skin image as a skin type label, inputting the first skin image into the skin image classification network to obtain a skin classification prediction result for the first skin image.

[0049] Step S1042: Calculate a first cross entropy loss based on the skin classification prediction result for the first skin image and the skin type label of the first skin image.

[0050] Step S1043: updating parameters of the skin image classification network according to the first cross entropy loss.

[0051] Step S1044, repeat the above steps until the preset number of training times is reached, and then end the training to obtain the initial skin image classification network.

[0052] In the embodiment of the present application, based on the first skin image A, the type of skin in the first skin image A is used as a label (i.e., skin type label a), the first skin image A is input into the skin image classification network to be trained, and the skin in the image is classified and predicted to obtain the skin classification prediction result b. Then, the loss function value is calculated according to the output result (skin classification prediction result b) and the label (skin type label a). In the embodiment of the present application, the output result is the same as the label as the goal, and the first cross entropy loss between the two is used as the loss function to update the parameters of the skin image classification network according to the calculated loss function value, which is regarded as completing a round of training for the skin image classification network. Then, a new first skin image is selected, and the above steps S1041-S1043 are repeated to perform the next round of training on the skin image classification network until the preset number of training times is reached, and the first stage of training for the skin image classification network is completed to obtain the initial skin image classification network.

[0053] Step S105, using the second skin image data set, training the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network.

[0054] In the embodiment of the present application, after the first stage of training of the hair removal network and the skin image classification network is performed using the first skin image data set (the skin in the first skin image does not have hair) through steps S102-S104, the second stage of training of the hair removal network and the skin image classification network is continued through step S105 using the second skin image data set (the skin in the second skin image has hair). Specifically, after the hair of the second skin image is removed through the initial hair removal network, it is input into the initial skin image classification network for skin image classification to obtain the final output classification result. According to the classification result and the classification label of the second skin image, the loss is calculated, so as to update the parameters of the two networks, and by repeating multiple rounds of training, a trained hair removal network and a trained skin image classification network are obtained.

[0055] In a possible implementation, the step S105, using the second skin image data set to train the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network, includes: Step S1051: input the second skin image in the second skin image data set into the initial hair removal network to generate a second hair-removed image.

[0056] Step S1052: input the second hair-removed image into the initial skin image classification network to obtain a skin classification prediction result for the second skin image.

[0057] Step S1053, taking the actual type of the skin in the second skin image as the skin type label of the second skin image, calculating a second cross entropy loss between the skin classification prediction result for the second skin image and the skin type label of the second skin image.

[0058] Step S1054: updating parameters of the initial hair removal network and the initial skin image classification network according to the second cross entropy loss.

[0059] Step S1055, repeat the above steps until the preset number of training times is reached, and then end the training to obtain the trained hair removal network and the trained skin image classification network.

[0060] In the embodiment of the present application, based on the second skin image C, the type of skin in the second skin image C is used as a label (i.e., skin type label c), and the second skin image C is input into the initial hair removal network obtained after the first stage of training is completed, and hair removal is performed to obtain the second hair removed image D. Then, the second hair removed image D is input into the initial skin image classification network obtained after the first stage of training is completed, and classification prediction is performed for the skin in image D to obtain the skin classification prediction result d. Then, the loss function value is calculated based on the output result (skin classification prediction result d) and the label (skin type label c). In the embodiment of the present application, the output result is the same as the label as the goal, and the cross entropy loss between the two is used as the loss function (second cross entropy loss), and the parameters of the initial hair removal network and the initial skin image classification network are updated according to the calculated loss function value, which is regarded as completing one round of training. Then, a new second skin image is selected, and the above steps S1051-S1054 are repeated to perform the next round of training until the preset number of training times is reached, completing the second stage of training the hair removal network and the skin image classification network, and obtaining the trained hair removal network and skin image classification network.

[0061] Step S106, inputting the skin image to be detected into the trained hair removal network to obtain a processed detection image after hair removal, and inputting the processed detection image into the trained skin image classification network to obtain a skin classification result of the image to be detected.

[0062] After completing the two stages of training, the trained hair removal network and skin image classification network are used to classify and predict the skin image. Specifically, for the image E to be detected, it is first input into the hair removal network to remove the hair, and a processed detection image E' after the hair is removed is obtained, and then it is input into the skin image classification network to classify and predict the type of skin in the image, and the skin classification result of the image E to be detected is obtained.

[0063] In order to reduce the interference of hair in skin images on skin classification, the embodiment of the present application proposes combining hair removal and skin disease recognition (i.e., combining a hair removal network and a skin image classification network). In the process of constructing the above two networks, a total of two stages of training are performed: first, the hair removal network and the skin image classification network are trained separately using the first skin image data set (the skin in the first skin image does not have hair); wherein, the present application uses Bezier curves to generate simulated hair, and synthesizes it with the original hairless image data (the first skin image) to produce labeled training samples (hair simulation images), thereby performing the first stage of training on the hair removal network, so that the hair removal network shows a certain hair removal effect on the real hair-bearing image data. Then, the hair removal network and the skin image classification network trained in the first stage are trained together using the second skin image data set (the skin in the second skin image has hair), so as to further improve the performance of the two networks. The trained hair removal network can accurately remove hair from skin images without significantly changing the characteristics of the original skin lesions, and then input the hair-removed images into the skin image classification network to obtain more accurate skin classification results (i.e., classify the image to be detected and obtain the category to which the skin in the image belongs).

[0064] In addition, the embodiment of the present application conducted comparative experiments on multiple backbone networks for the proposed dual-stage multi-task framework (training the hair removal network and the skin image classification network in two stages through steps S101-S105). The experimental results show that the framework proposed in the embodiment of the present application effectively improves the accuracy of the network in identifying skin lesion types on a real hair image dataset.

[0065] Specifically, the skin image classification network initially adopted the classic ResNet18 as the backbone, and on this basis, a partial ablation experiment was conducted, and the experimental results are shown in Table 1. Among them, DSConv Net is a hair removal network proposed in the embodiment of the present application that replaces ordinary convolution operations with dynamic snake convolutions, and UNet represents a UNet network that maintains the original ordinary convolution operations. L1Loss represents absolute error loss, L2Loss represents mean square error loss, and ACC represents the accuracy of hair removal by the network. As can be seen from Table 1, the hair removal network proposed in this application has a better accuracy than the general UNet network.

[0066] Table 1 Statistics of ablation experimental results for hair removal network

[0067] In order to fully verify the effectiveness of the skin image classification network proposed in the embodiment of the present application, experiments were conducted on multiple backbones. The experimental results show that the skin image classification network proposed in the present application (corresponding to DHSLR in Table 2) significantly improves the recognition accuracy of each backbone for the type of skin images with hair. The results are shown in Table 2. Hair in the table indicates that the model only uses the second skin image data set (i.e., skin images with hair) for training, and Clean+Hair indicates that the first skin image data set and the second skin image data set are used for training to ensure that the comparison of the experimental results is fair.

[0068] Table 2 Comparison experimental results of various backbones in skin image classification network

[0069] Finally, this embodiment also obtains the hairy skin disease image and its heat map based on Class Activation Mapping (CAM) (the color of the heat map indicates the contribution of the area to the prediction result) and the skin lesion classification prediction result of the image (dark red indicates the skin disease category corresponding to the current prediction, that is, the maximum probability value of the current prediction), as well as the skin disease image after hair removal and its heat map and classification prediction result. Exemplarily, from the analysis of the heat map, it is found that when the original hairy skin disease image is used for diagnosis and identification of skin disease types, almost all areas of the image contribute greatly to the final prediction and classification results. The final classification prediction result is: the probability that the skin type in the image belongs to melanoma is the highest, the probability of nevus and pigmented benign keratosis is small, and the probability of other categories is close to 0. However, when the skin disease image after hair removal is used for diagnosis and identification of skin disease types, from the analysis of the heat map, only the area where the lesion is located has a significant contribution to the final prediction and classification result, and the contribution of other areas is very small. The final classification prediction result is: the probability of the skin type in the image belonging to benign pigmented keratosis is the highest, the probability of nevus and melanoma is relatively low, and the probability of other categories is close to 0. The actual skin disease type of this sample is benign pigmented keratosis. It can be clearly seen from the heat map that when the original skin disease image with hair is used for diagnosis and identification of skin disease types, the area around the lesion has a greater contribution to the final prediction classification result, while the contribution of the area where the lesion itself is located is relatively small. When the skin disease image after hair removal is used for diagnosis and identification of skin disease types, from the analysis of the heat map, the area where the lesion is located has a significant contribution to the final prediction classification result, and the contribution of other areas is relatively small.

[0070] Exemplarily, in the embodiment of the present application, the first skin image dataset contains a total of 7502 skin disease images without hair (i.e., the first skin images) and their corresponding 10 skin disease categories (actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, pigmented benign keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions), and the number of images in each category is: 285 actinic keratosis images, 1243 basal cell carcinoma images, 90 dermatofibroma images, 2190 melanoma images, 2190 nevus images, 631 pigmented benign keratosis images, 399 seborrheic keratosis images, 95 solar lentigo images, 232 squamous cell carcinoma images, and 147 vascular lesion images. The second skin image dataset contains a total of 3280 skin disease images without hair (i.e., second skin images) and their corresponding 10 skin disease categories. The number of images in each category is: 59 actinic keratosis images, 417 basal cell carcinoma images, 31 skin fibroma images, 556 melanoma images, 1734 nevus images, 103 pigmented benign keratosis images, 275 seborrheic keratosis images, 15 solar lentigo images, 52 squamous cell carcinoma images, and 38 vascular lesion images. Due to the uneven distribution of categories, 20% of the image data of each category is randomly selected as the test set when dividing the dataset, and the remaining 80% of the image data is used as the training set to prevent the problem of insufficient samples caused by the small amount of data in individual categories.

[0071] The second aspect of the embodiment of the present application further provides a skin image classification device, referring to Figure 6 , Figure 6 A schematic diagram of the structure of a skin image classification device is shown. Figure 6 As shown, the device comprises: An image acquisition module, configured to acquire a first skin image data set and a second skin image data set, wherein the skin in each first skin image in the first skin image data set is hairless, and the skin in each second skin image in the second skin image data set is hairy; a hair simulation module, configured to perform hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image; A first training module, configured to train the hair removal network using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network; A second training module is used to train a skin image classification network using the first skin image dataset to obtain an initial skin image classification network; A third training module is used to train the initial hair removal network and the initial skin image classification network using the second skin image data set to obtain a trained hair removal network and a trained skin image classification network; The skin image classification module is used to input the skin image to be detected into the trained hair removal network to obtain a processed detection image after hair removal, and input the processed detection image into the trained skin image classification network to obtain the skin classification result of the image to be detected.

[0072] In a possible implementation, the hair simulation module includes: The first hair simulation submodule is used to generate a random curve on a blank image using a Bezier curve algorithm to obtain a simulated hair image; The second hair simulation submodule is used to fuse the simulated hair image with the first skin image to obtain the hair simulation image.

[0073] In a possible implementation, the network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a feature fusion module based on dynamic snake convolution to replace the convolution module in the UNet network architecture.

[0074] In a possible implementation, the using the hair simulation image and the corresponding first skin image to train the hair removal network to obtain an initial hair removal network includes: Inputting the hair simulation image into the hair removal network to generate a first hair-removed image; Calculating the mean absolute error loss between the first hair-removed image and the corresponding first image; updating parameters of the hair removal network according to the mean absolute error loss; Repeat the above steps until the preset number of training times is reached, and then end the training to obtain the initial hair removal network.

[0075] In a possible implementation, the using the first skin image dataset to train a skin image classification network to obtain an initial skin image classification network includes: Taking the real type of skin in the first skin image as a skin type label, inputting the first skin image into the skin image classification network to obtain a skin classification prediction result for the first skin image; Calculating a first cross entropy loss according to the skin classification prediction result for the first skin image and the skin type label of the first skin image; According to the first cross entropy loss, updating parameters of the skin image classification network; Repeat the above steps until the preset number of training times is reached, and then terminate the training to obtain the initial skin image classification network.

[0076] In a possible implementation, the using the second skin image data set to train the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network includes: Inputting the second skin image in the second skin image dataset into the initial hair removal network to generate a second hair-removed image; Inputting the second hair-removed image into the initial skin image classification network to obtain a skin classification prediction result for the second skin image; Taking the actual type of the skin in the second skin image as the skin type label of the second skin image, calculating a second cross entropy loss between the skin classification prediction result for the second skin image and the skin type label of the second skin image; According to the second cross entropy loss, updating parameters of the initial hair removal network and the initial skin image classification network; Repeat the above steps until the preset number of training times is reached, and then end the training to obtain the trained hair removal network and the trained skin image classification network.

[0077] In a possible implementation, the skin types in the first skin image or the second skin image include at least: actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign pigmented keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0078] The present application also provides an electronic device, referring to Figure 7 , Figure 7 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 7 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the skin image classification method based on hair removal network disclosed in the embodiment of the present application.

[0079] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the skin image classification method based on hair removal network as disclosed in the embodiment of the present application are implemented.

[0080] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor is enabled to implement the steps of the skin image classification method based on the hair removal network as disclosed in the embodiment of the present application.

[0081] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0082] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0086] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0087] The above is a detailed introduction to the skin image classification method, device and product based on hair removal network provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A skin image classification method based on hair removal network, characterized in that: Methods include: Acquire a first skin image dataset and a second skin image dataset, wherein the skin in each first skin image in the first skin image dataset is hairless, and the skin in each second skin image in the second skin image dataset is hairy; Performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image; Using the hair simulation image and the corresponding first skin image, the hair removal network is trained to obtain an initial hair removal network; Using the first skin image dataset, training a skin image classification network to obtain an initial skin image classification network; Using the second skin image data set, training the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network; The skin image to be detected is input into the trained hair removal network to obtain a processed detection image after hair removal, and the processed detection image is input into the trained skin image classification network to obtain a skin classification result of the image to be detected.

2. The skin image classification method based on hair removal network according to claim 1, characterized in that: The performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image includes: On the blank image, a Bezier curve algorithm is used to generate a random curve to obtain a simulated hair image; The simulated hair image is fused with the first skin image to obtain the hair simulation image.

3. The skin image classification method based on hair removal network according to claim 1, characterized in that: The network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a feature fusion module based on dynamic snake convolution to replace the convolution module in the UNet network architecture.

4. The skin image classification method based on hair removal network according to claim 1, characterized in that: The using the hair simulation image and the corresponding first skin image to train the hair removal network to obtain an initial hair removal network includes: Inputting the hair simulation image into the hair removal network to generate a first hair-removed image; Calculating the mean absolute error loss between the first hair-removed image and the corresponding first image; updating parameters of the hair removal network according to the mean absolute error loss; Repeat the above steps until the preset number of training times is reached, and then end the training to obtain the initial hair removal network.

5. The skin image classification method based on hair removal network according to claim 1, characterized in that: The step of training a skin image classification network using the first skin image dataset to obtain an initial skin image classification network includes: Taking the real type of skin in the first skin image as a skin type label, inputting the first skin image into the skin image classification network to obtain a skin classification prediction result for the first skin image; Calculating a first cross entropy loss according to the skin classification prediction result for the first skin image and the skin type label of the first skin image; According to the first cross entropy loss, updating parameters of the skin image classification network; Repeat the above steps until the preset number of training times is reached, and then terminate the training to obtain the initial skin image classification network.

6. The skin image classification method based on hair removal network according to claim 1, characterized in that: The using the second skin image data set to train the initial hair removal network and the initial skin image classification network to obtain a trained hair removal network and a trained skin image classification network includes: Inputting the second skin image in the second skin image dataset into the initial hair removal network to generate a second hair-removed image; Inputting the second hair-removed image into the initial skin image classification network to obtain a skin classification prediction result for the second skin image; Taking the actual type of the skin in the second skin image as the skin type label of the second skin image, calculating a second cross entropy loss between the skin classification prediction result for the second skin image and the skin type label of the second skin image; According to the second cross entropy loss, updating parameters of the initial hair removal network and the initial skin image classification network; Repeat the above steps until the preset number of training times is reached, and then end the training to obtain the trained hair removal network and the trained skin image classification network.

7. The skin image classification method based on hair removal network according to claim 1, characterized in that: The types of skin in the first skin image or the second skin image include at least: actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign pigmented keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

8. A skin image classification device, characterized in that: The device comprises: An image acquisition module, configured to acquire a first skin image data set and a second skin image data set, wherein the skin in each first skin image in the first skin image data set is hairless, and the skin in each second skin image in the second skin image data set is hairy; a hair simulation module, configured to perform hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image; A first training module, configured to train the hair removal network using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network; A second training module is used to train a skin image classification network using the first skin image dataset to obtain an initial skin image classification network; A third training module is used to train the initial hair removal network and the initial skin image classification network using the second skin image data set to obtain a trained hair removal network and a trained skin image classification network; The skin image classification module is used to input the skin image to be detected into the trained hair removal network to obtain a processed detection image after hair removal, and input the processed detection image into the trained skin image classification network to obtain the skin classification result of the image to be detected.

9. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the skin image classification method based on hair removal network according to any one of claims 1-7.

10. A computer program product, characterized in that When the computer program product is run on an electronic device, the processor is enabled to implement the skin image classification method based on hair removal network as described in any one of claims 1 to 7.

Images