A skin image classification method, device and product based on a hair removal network

Abstract

The application provides a skin image classification method, device and product based on a hair removal network, relates to the technical field of image classification, and the method comprises the following steps: performing hair simulation on each first skin image in a first skin image data set to obtain a hair simulation image; training the hair removal network by using the hair simulation image to obtain an initial hair removal network; training the skin image classification network by using the first skin image data set to obtain an initial skin image classification network; training the initial hair removal network and the initial skin image classification network by using a second skin image data set to obtain a trained hair removal network and a trained skin image classification network; inputting a to-be-detected skin image into the trained hair removal network to obtain a processed detection image after hair removal; and inputting the processed detection image into the trained skin image classification network to obtain a skin classification result of the to-be-detected image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, and in particular to a skin image classification method, device and product based on a hair removal network. BACKGROUND

[0002] In the research of skin lesion classification and identification, classification based on dermoscope images is an important task. With the development of deep learning technology, the method of using neural networks to train labeled data to extract skin lesion features has become the mainstream.

[0003] However, there are often other interference factors (such as hair interference) in skin images, which cause the neural network model to learn irrelevant features, thereby affecting the accuracy of the final classification detection (i.e., skin lesion type identification). Therefore, it is urgent to propose a skin image classification method, device and product based on a hair removal network to improve the classification accuracy of skin images. SUMMARY

[0004] In view of the above problems, the embodiments of the present application provide a skin image classification method, device and product based on a hair removal network in order to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of the embodiments of the present application provides a skin image classification method based on a hair removal network, which comprises:

[0006] obtaining a first skin image data set and a second skin image data set, each first skin image in the first skin image data set not having hair on the skin, each second skin image in the second skin image data set having hair on the skin;

[0007] performing hair simulation on each first skin image in the first skin image data set to obtain a hair simulation image;

[0008] training the hair removal network using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network;

[0009] training a skin image classification network using the first skin image data set to obtain an initial skin image classification network;

[0010] training the initial hair removal network and the initial skin image classification network using the second skin image data set to obtain a trained hair removal network and a trained skin image classification network;

[0011] The skin image to be detected is input into the trained hair removal network to obtain a processed detection image after hair removal, and the processed detection image is input into the trained skin image classification network to obtain a skin classification result of the image to be detected.

[0012] In a possible implementation, the hair simulation on each first skin image in the first skin image dataset to obtain a hair simulation image includes:

[0013] A random curve is generated on a blank image by using a Bezier curve algorithm to obtain a simulated hair image;

[0014] The simulated hair image is fused with the first skin image to obtain the hair simulation image.

[0015] In a possible implementation, the network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a feature fusion module based on dynamic snake convolution to replace a convolution module in the UNet network architecture.

[0016] In a possible implementation, the training of the hair removal network by using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network includes:

[0017] The hair simulation image is input into the hair removal network to generate a first hair-removed image;

[0018] An average absolute error loss between the first hair-removed image and the corresponding first image is calculated;

[0019] Parameters of the hair removal network are updated according to the average absolute error loss;

[0020] The above steps are repeated until a preset training number is reached, and the training is ended to obtain the initial hair removal network.

[0021] In a possible implementation, the training of the skin image classification network by using the first skin image dataset to obtain an initial skin image classification network includes:

[0022] The first skin image is input into the skin image classification network with a skin type label of a real type of skin in the first skin image to obtain a skin classification prediction result for the first skin image;

[0023] A first cross-entropy loss is calculated according to the skin classification prediction result for the first skin image and the skin type label of the first skin image.

[0024] perform parameter updating on the skin image classification network according to the first cross-entropy loss;

[0025] The above steps are repeated until a preset number of training times is reached, and the training is ended to obtain the initial skin image classification network.

[0026] In a possible implementation, the training of the initial hair removal network and the initial skin image classification network using the second skin image dataset to obtain the trained hair removal network and the trained skin image classification network includes:

[0027] The second skin image in the second skin image dataset is input into the initial hair removal network to generate a second hair-removed image;

[0028] The second hair-removed image is input into the initial skin image classification network to obtain a skin classification prediction result for the second skin image;

[0029] A second cross-entropy loss between the skin classification prediction result for the second skin image and a skin type label of the second skin image is calculated, where the skin type label of the second skin image is a true type of skin in the second skin image;

[0030] Parameter updating is performed on the initial hair removal network and the initial skin image classification network according to the second cross-entropy loss;

[0031] The above steps are repeated until a preset number of training times is reached, and the training is ended to obtain the trained hair removal network and the trained skin image classification network.

[0032] In a possible implementation, the type of skin in the first skin image or the second skin image includes at least actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0033] The second aspect of the embodiment of the present application further provides a skin image classification device, and the device includes:

[0034] An image acquisition module is configured to acquire a first skin image dataset and a second skin image dataset, where the skin in each first skin image in the first skin image dataset is free of hair, and the skin in each second skin image in the second skin image dataset is provided with hair;

[0035] A hair simulation module is configured to perform hair simulation on each first skin image in the first skin image dataset to obtain a hair-simulated image.

[0036] The first training module is configured to train the hair removal network by using the hair simulation image and the corresponding first skin image, to obtain an initial hair removal network.

[0037] The second training module is configured to train a skin image classification network by using the first skin image dataset, to obtain an initial skin image classification network.

[0038] The third training module is configured to train the initial hair removal network and the initial skin image classification network by using the second skin image dataset, to obtain a trained hair removal network and a trained skin image classification network.

[0039] The skin image classification module is configured to input a to-be-detected skin image into the trained hair removal network, to obtain a processed detection image after hair removal, and input the processed detection image into the trained skin image classification network, to obtain a skin classification result of the to-be-detected image.

[0040] The third aspect of the embodiments of the present application further provides an electronic device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps in the skin image classification method based on the hair removal network in the first aspect of the embodiments of the present application.

[0041] The fourth aspect of the embodiments of the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps in the skin image classification method based on the hair removal network in the first aspect of the embodiments of the present application.

[0042] The fifth aspect of the embodiments of the present application further provides a computer program product, which is run on an electronic device, and when the processor executes the computer program product, the steps in the skin image classification method based on the hair removal network in the first aspect of the embodiments of the present application are implemented.

[0043] The embodiment of the present application provides a skin image classification method based on a hair removal network, which comprises the following steps: acquiring a first skin image dataset and a second skin image dataset, wherein the skin in each first skin image in the first skin image dataset is free of hair, and the skin in each second skin image in the second skin image dataset is provided with hair; performing hair simulation on each first skin image in the first skin image dataset to obtain a hair simulation image; training the hair removal network by using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network; training a skin image classification network by using the first skin image dataset to obtain an initial skin image classification network; training the initial hair removal network and the initial skin image classification network by using the second skin image dataset to obtain a trained hair removal network and a trained skin image classification network; inputting a to-be-detected skin image into the trained hair removal network to obtain a processed detection image after hair removal, and inputting the processed detection image into the trained skin image classification network to obtain a skin classification result of the to-be-detected image.

[0044] Specific beneficial effects are that, in order to reduce the interference of hair in a skin image on skin classification, the embodiment of the present application proposes to combine hair removal and skin disease identification (i.e. combine a hair removal network and a skin image classification network), and in the process of constructing the above two networks, two-stage training is performed: first, the first skin image dataset (the skin in the first skin image is free of hair) is used to train the hair removal network and the skin image classification network respectively; then, the second skin image dataset (the skin in the second skin image is provided with hair) is used to train the hair removal network and the skin image classification network trained in the first stage together, so as to further improve the performance of the two networks. The trained hair removal network accurately removes the hair in the skin image without significantly changing the characteristics of the original skin lesion, and then the image after hair removal is input into the skin image classification network, so that a more accurate skin classification result (i.e. classification of the to-be-detected image to obtain the category to which the skin in the image belongs) is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1is a step flow chart of a skin image classification method based on a hair removal network provided by an embodiment of the present application.

[0047] Figure 2 is a curve shape schematic diagram of a Bezier curve provided by an embodiment of the present application.

[0048] Figure 3 is a simulated hair schematic diagram generated by a Bezier curve algorithm provided by an embodiment of the present application.

[0049] Figure 4 is a network architecture schematic diagram of a hair removal network provided by an embodiment of the present application.

[0050] Figure 5 is a structure schematic diagram of a feature fusion module provided by an embodiment of the present application.

[0051] Figure 6 is a structure schematic diagram of a skin image classification device provided by an embodiment of the present application.

[0052] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present application will be described in detail hereinafter with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood, and the scope of the present application can be accurately conveyed to those skilled in the art.

[0054] In the research of skin lesion classification and recognition, classification based on dermoscope images is a routine and important task. With the development of deep learning technology, the method of using neural networks to train labeled data to extract skin lesion features has become the mainstream. However, hair interference often occurs in dermoscope images, which is an irrelevant feature that is not desired to be learned by the neural network model in the process of skin lesion recognition. Therefore, before classification and recognition, the skin lesion images containing hair can be processed for hair removal, and then further classification and recognition can be performed using deep neural networks. When the skin lesion images processed by hair removal are classified and recognized, the accuracy will certainly be improved.

[0055] Generally speaking, for the unsupervised cross-domain image conversion task, some unsupervised image-to-image conversion algorithms (for example, CycleGAN) are usually used to complete. However, it is found in actual experiments that CycleGAN often changes the characteristics of the original skin lesions while removing the hair. However, such changes are unacceptable in medical image processing, because the significant change of the skin lesions will have a greater impact on the diagnosis and identification of skin lesion types, resulting in too low skin image classification accuracy.

[0056] In view of the above problems, the embodiment of the present application proposes a skin image classification method, device and product based on a hair removal network to improve the classification accuracy of skin images. In the following, the skin image classification method based on the hair removal network provided by the embodiment of the present application will be described in detail in combination with the drawings and some embodiments and their application scenarios.

[0057] The first aspect of the embodiment of the present application provides a skin image classification method based on a hair removal network. The overall technical concept of the method is that in order to reduce the interference of hair in the skin image on skin classification, a combination of hair removal and skin disease identification (i.e. a combination of a hair removal network and a skin image classification network) is proposed. In the process of constructing the above two networks, two stages of training are performed: first, the first skin image dataset (the skin in the first skin image does not have hair) is used to train the hair removal network and the skin image classification network respectively; then, the second skin image dataset (the skin in the second skin image has hair) is used to train the hair removal network and the skin image classification network trained in the first stage together, to further improve the performance of the two networks. Finally, the trained hair removal network and skin image classification network are used to realize the classification and identification of the input skin image. The specific process is described in the following embodiments.

[0058] Reference Figure 1 , Figure 1 The step flow chart of the skin image classification method based on the hair removal network provided by the embodiment of the present application is shown in Figure 1 , which comprises the following steps.

[0059] In step S101, a first skin image dataset and a second skin image dataset are obtained. The skin in each first skin image in the first skin image dataset does not have hair, and the skin in each second skin image in the second skin image dataset has hair.

[0060] First, in order to train the hair removal network and the skin image classification network, a large number of skin images (i.e. dermoscopy images) need to be collected first, and the skin type corresponding to each skin image needs to be determined. In the present embodiment, the skin images are divided into two categories, first skin images and second skin images. The skin in the first skin images does not have hair (i.e. skin disease pictures without hair interference), and the skin in the second skin images has hair (i.e. skin disease pictures with hair interference), thereby forming a first skin image dataset and a second skin image dataset, respectively. There is no one-to-one correspondence between the first skin images and the second skin images.

[0061] For each image (first skin image or second skin image), a corresponding skin type label needs to be set in advance, which represents the real type of the skin in the image. Specifically, the type of the skin (skin type label) includes at least actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, pigmented benign keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0062] In a possible implementation, the type of the skin in the first skin image or the second skin image includes at least actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, pigmented benign keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0063] After obtaining rich training data (the first skin image dataset and the second skin image dataset), the present embodiment first trains the hair removal network and the skin image classification network in the first stage by steps S102-S104 using the first skin image dataset (the skin in the first skin image does not have hair).

[0064] In step S102, for each first skin image in the first skin image dataset, hair simulation is performed to obtain a hair simulation image.

[0065] Specifically, when the first skin image dataset is used to train the hair removal network in the first stage, the first skin image (the skin in the image does not have hair) is taken as a label for training. First, hair simulation is performed on the first skin image, and hair simulated on the skin in the first skin image by using a Bezier curve algorithm is obtained to obtain a hair simulation image. Then, the hair simulation image is input into the hair removal network to be trained, so that the hair removal network learns to remove the hair on the skin in the image. The image output by the hair removal network finally becomes more and more close to the corresponding label (the first skin image).

[0066] In a possible implementation, the step S102, for each first skin image in the first skin image dataset, simulating hair to obtain a hair simulation image, includes:

[0067] The step S1021 generates a random curve on the blank image by using a Bezier curve algorithm to obtain a simulated hair image.

[0068] The step S1022 fuses the simulated hair image with the first skin image to obtain the hair simulation image.

[0069] The embodiment proposes a simulated hair generation method based on a Bezier curve. A curve similar to hair is simulated by using a Bezier curve, and then the generated hair is added to a skin disease image without hair (a first skin image) to simulate a skin image with hair (a hair simulation image), so that the hair simulation image and the original first skin image without hair form a pair of training data samples.

[0070] Specifically, the Bezier curve is a smooth curve drawn by using coordinates of points at any positions, which is proposed by Pierre Bézier and is widely used in design and graphics generation tasks. A function curve generated by using a quadratic Bezier curve can be used as a simulated hair shape. A parameter equation of the quadratic Bezier curve can be expressed as:

[0071] ;

[0072] wherein B is a trajectory point of the quadratic Bezier curve, P0, P1, and P2 are initial coordinate points of three arbitrary positions in the blank image. Figure 2 FIG. 1 is a schematic diagram of a curve shape of a Bezier curve. Figure 2 As shown in FIG. 1, with the change of the parameter t, the trajectory of the point B forms a quadratic Bezier curve, which is similar to the shape of the hair.

[0073] Based on the principle of the Bezier curve, in the embodiment, first, one or more lines with random shapes are generated on the blank image by using the Bezier curve algorithm, and the thickness of the line is adjusted within a certain threshold range as a simulated hair shape, so as to obtain an image containing multiple simulated hairs, as shown in FIG. 2. Figure 3 Figure 3 FIG. 2 is a schematic diagram of simulated hair generated by using the Bezier curve algorithm. Then, the simulated hair is fused with the first skin image by using the step S1022, and the original first skin image without hair is converted into the hair simulation image with hair on the skin.

[0074] ​Step S103, training the hair removal network by using the hair simulation image and the corresponding first skin image, to obtain an initial hair removal network.

[0075] Specifically, based on the first skin image A, a corresponding hair simulation image A' is generated, and then the hair simulation image A' is input into the hair removal network to be trained, taking the first skin image A as a label, so that the hair removal network learns to remove the hair on the skin in the image. The image output by the hair removal network finally becomes more and more close to the corresponding label (the first skin image A), and an initial hair removal network trained in the first stage is obtained.

[0076] In a possible implementation, the network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a dynamic snake convolution based feature fusion module to replace a convolution module in the UNet network architecture.

[0077] In the embodiments of the present application, the hair removal network can be constructed based on the UNet architecture as the main body. Referring to Figure 4 , Figure 4 A network architecture schematic diagram of a hair removal network is shown, as shown in Figure 4 The hair removal network is composed of a U-shaped network architecture with multiple convolution modules, wherein arrow ① represents that after the convolution module performs the corresponding convolution operation, a maximum pooling (Max Pooling) operation is performed, arrow ② represents that after the convolution module performs the corresponding convolution operation, a bilinear interpolation (Bilinear Interpolation) operation is performed, and the dashed arrow represents a shortcut connection (Shortcut Connection) operation. Further, in the UNet architecture, the dynamic snake convolution based feature fusion (DSC-FF) module is used to replace the ordinary convolution operation,

[0078] Specifically, referring to Figure 5 , Figure 5 A structure schematic diagram of a feature fusion module is shown, as shown in Figure 5 In the feature fusion module (i.e., the DSC-FF module), three branches of 3x3 ordinary convolution, 9x1 and 1x9 dynamic snake convolution are used, each branch is subjected to group normalization (Group Normalization, GroupNorm) and ReLU activation function, then the features of the three branches are spliced together, and then output through a 3x3 ordinary convolution layer, group normalization (Group Normalization, GroupNorm) and ReLU activation function.

[0079] In a possible implementation, the step S103 of training the hair removal network by using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network comprises the following steps.

[0080] A step S1031 of inputting the hair simulation image into the hair removal network to generate a first hair-removed image.

[0081] A step S1032 of calculating a mean absolute error loss between the first hair-removed image and the corresponding first image.

[0082] A step S1033 of performing parameter updating on the hair removal network according to the mean absolute error loss.

[0083] A step S1034 of repeating the above steps until a preset training number is reached, and ending the training to obtain the initial hair removal network.

[0084] In the embodiments of the present application, the corresponding hair simulation image A' is generated based on the first skin image A, and then the hair simulation image A' is input into the hair removal network to be trained with the first skin image A as a label to perform hair removal and obtain a first hair-removed image B. Then, a loss function value is calculated according to the output result (the first hair-removed image B) and the label (the first skin image A). In the embodiments of the present application, the output result is closer and closer to the label as the goal, and the mean absolute error loss (Mean Absolute Error, MAE) between the two is used as the loss function to perform parameter updating on the hair removal network according to the calculated loss function value, which is regarded as completing one round of training of the hair removal network. Then, a new hair simulation image and the corresponding first skin image are selected to repeat the steps S1031-S1033 to perform the next round of training on the hair removal network until a preset training number is reached, the first-stage training of the hair removal network is completed, and an initial hair removal network is obtained.

[0085] A step S104 of training a skin image classification network by using the first skin image dataset to obtain an initial skin image classification network.

[0086] Specifically, based on the first skin image A, the first skin image A is input into the skin image classification network to be trained with the type of skin in the first skin image A as a label, so that the skin image classification network learns to classify and predict the skin in the image. The skin type predicted by the skin image classification network finally becomes closer and closer to the corresponding label (the real type of skin in the first skin image A), and an initial skin image classification network trained in the first stage is obtained.

[0087] In a possible implementation, the step S104 of training the skin image classification network by using the first skin image dataset to obtain an initial skin image classification network comprises:

[0088] The step S1041 comprises: inputting the first skin image into the skin image classification network by taking the real type to which the skin in the first skin image belongs as a skin type label, to obtain a skin classification prediction result for the first skin image.

[0089] The step S1042 comprises: calculating a first cross-entropy loss according to the skin classification prediction result for the first skin image and the skin type label of the first skin image.

[0090] The step S1043 comprises: performing parameter updating on the skin image classification network according to the first cross-entropy loss.

[0091] The step S1044 comprises: repeating the above steps until a preset training number is reached, ending the training, and obtaining the initial skin image classification network.

[0092] In the embodiment of the application, based on the first skin image A, the first skin image A is input into the skin image classification network to be trained by taking the type to which the skin in the first skin image A belongs as a label (i.e., a skin type label a), and a skin classification prediction result b is obtained by performing classification prediction on the skin in the image. Then, a loss function value is calculated according to the output result (the skin classification prediction result b) and the label (the skin type label a). In the embodiment of the application, the same output result and label are taken as a target, the first cross-entropy loss between the two is taken as a loss function, and the skin image classification network is updated according to the calculated loss function value, which is regarded as completing one round of training of the skin image classification network. Then, a new first skin image is selected, and the steps S1041-S1043 are repeated to perform the next round of training on the skin image classification network, until a preset training number is reached, the first-stage training of the skin image classification network is completed, and an initial skin image classification network is obtained.

[0093] The step S105 comprises: training the initial hair removal network and the initial skin image classification network by using the second skin image dataset to obtain a trained hair removal network and a trained skin image classification network.

[0094] After the first-stage training of the hair removal network and the skin image classification network by using the first skin image dataset (the skin in the first skin image does not have hair) through steps S102-S104, the second-stage training of the hair removal network and the skin image classification network is continued by using the second skin image dataset (the skin in the second skin image has hair) through step S105. Specifically, after the second skin image is input into the initial hair removal network for hair removal, the initial skin image classification network is input for skin image classification, and a final output classification result is obtained. According to the classification result and the classification label of the second skin image, loss calculation is performed, so that the parameters of the two networks are updated, and through repeated training for multiple rounds, the trained hair removal network and the trained skin image classification network are obtained.

[0095] In a possible implementation, the step S105, training the initial hair removal network and the initial skin image classification network by using the second skin image dataset to obtain the trained hair removal network and the trained skin image classification network, includes:

[0096] Step S1051, inputting the second skin image in the second skin image dataset into the initial hair removal network to generate a second hair-removed image.

[0097] Step S1052, inputting the second hair-removed image into the initial skin image classification network to obtain a skin classification prediction result for the second skin image.

[0098] Step S1053, taking the real type of the skin in the second skin image as the skin type label of the second skin image, calculating a second cross-entropy loss between the skin classification prediction result for the second skin image and the skin type label of the second skin image.

[0099] Step S1054, updating the parameters of the initial hair removal network and the initial skin image classification network according to the second cross-entropy loss.

[0100] Step S1055, repeating the above steps until a preset training number is reached, ending the training, and obtaining the trained hair removal network and the trained skin image classification network.

[0101] In the embodiment of the present application, based on the second skin image C, the second skin image C is input into the initial hair removal network obtained after the first stage training, with the type to which the skin in the second skin image C belongs as the label (i.e., the skin type label c), to perform hair removal, and obtain a second hair-removed image D. Then, the second hair-removed image D is input into the initial skin image classification network obtained after the first stage training, to perform classification prediction on the skin in the image D, and obtain a skin classification prediction result d. Then, a loss function value is calculated according to the output result (the skin classification prediction result d) and the label (the skin type label c). In the embodiment of the present application, the same output result and label are taken as the target, and the cross-entropy loss between the two is taken as the loss function (the second cross-entropy loss), to perform parameter updating on the initial hair removal network and the initial skin image classification network according to the calculated loss function value, which is regarded as completing one round of training. Then, a new second skin image is selected, and the above steps S1051-S1054 are repeated to perform the next round of training, until a preset number of training times is reached, the training of the hair removal network and the skin image classification network in the second stage is completed, and the trained hair removal network and skin image classification network are obtained.

[0102] In step S106, the to-be-detected skin image is input into the trained hair removal network to obtain a processed detection image after hair removal, and the processed detection image is input into the trained skin image classification network to obtain a skin classification result of the to-be-detected image.

[0103] After the two-stage training is completed, the trained hair removal network and the skin image classification network are used to perform classification prediction on a skin image. Specifically, for a to-be-detected image E, the to-be-detected image E is first input into the hair removal network to perform hair removal, and a processed detection image E' after hair removal is obtained, and then the processed detection image E' is input into the skin image classification network to perform classification prediction on the type to which the skin in the image belongs, and obtain a skin classification result of the to-be-detected image E.

[0104] In order to reduce the interference of hair in the skin image on skin classification, the embodiment of the present application proposes to combine hair removal and skin disease recognition (i.e. combine hair removal network and skin image classification network), and in the process of constructing the above two networks, two stages of training are performed in total: first, the first skin image dataset (the skin in the first skin image does not have hair) is used to train the hair removal network and the skin image classification network respectively; wherein, the present application uses the Bezier curve to generate simulated hair, and synthesizes it with the original hairless image data (the first skin image) to make labeled training samples (hair simulation images), thereby performing the first stage of training on the hair removal network, so that the hair removal network shows a certain hair removal effect on the real hair image data. Then, the second skin image dataset (the skin in the second skin image has hair) is used to train the hair removal network and the skin image classification network trained in the first stage together, to further improve the performance of the two networks. So that the trained hair removal network accurately removes the hair in the skin image without significantly changing the characteristics of the original skin lesions, and then inputs the image after removing the hair into the skin image classification network, so as to obtain a more accurate skin classification result (i.e. classifying the image to be detected to obtain the category to which the skin in the image belongs).

[0105] In addition, the embodiment of the present application conducts comparative experiments on multiple backbone networks for the proposed two-stage multi-task framework (two-stage training of the hair removal network and the skin image classification network through steps S101-S105). The experimental results show that the framework proposed in the embodiment of the present application effectively improves the skin lesion category recognition accuracy of the network on the real hair image dataset.

[0106] Specifically, the skin image classification network initially uses the classic ResNet18 as Backbone, and conducts part of the ablation experiment on this basis, and the experimental results are shown in Table 1. Among them, DSConv Net is the hair removal network proposed in the embodiment of the present application which replaces the ordinary convolution operation with dynamic snake convolution, and UNet represents the UNet network which keeps the original ordinary convolution operation. L1Loss represents the absolute error loss, L2Loss represents the mean square error loss, and ACC represents the accuracy of the network for hair removal. From Table 1, it can be seen that the accuracy of the hair removal network proposed in the present application is better than that of the general UNet network.

[0107] Table 1 Ablation experiment result statistics table for hair removal network

[0108]

[0109] To fully verify the effectiveness of the skin image classification network proposed in the embodiments of the present application, experiments were conducted on multiple backbones. The experimental results prove that the skin image classification network proposed in the present application (corresponding to DHSLR in Table 2) significantly improves the recognition accuracy of each Backbone for the hair-bearing skin image category. The results are shown in Table 2, where Hair indicates that the model is trained only using the second skin image dataset (i.e., hair-bearing skin images), and Clean+Hair indicates that the first skin image dataset and the second skin image dataset are used for training to ensure that the comparison of the experimental results is fair.

[0110] Table 2 Comparison of experimental results of various Backbones of skin image classification network

[0111]

[0112] Finally, the embodiments also obtain the hair-bearing skin disease image, the heat map (the color height of the heat map represents the contribution of the region to the prediction result) based on the Class Activation Mapping (CAM), and the skin lesion classification prediction result (dark red color represents the current predicted skin disease category, i.e., the maximum probability value of the current prediction) of the image, as well as the skin disease image after removing the hair and its heat map and classification prediction result. For example, from the heat map, it is analyzed that when the original hair-bearing skin disease image is used for skin disease category diagnosis and recognition, almost all regions of the image have a greater contribution to the final prediction classification result. The final classification prediction result is that the skin type in the image belongs to the melanoma with the maximum probability, the nevus and the benign keratosis with a smaller probability, and the probabilities of other categories are close to 0. However, when the skin disease image after removing the hair is used for skin disease category diagnosis and recognition, from the heat map, only the lesion area has a significant contribution to the final prediction classification result, and the contribution of the remaining regions is small. The final classification prediction result is that the skin type in the image belongs to the benign keratosis with the maximum probability, the nevus and the melanoma with a smaller probability, and the probabilities of other categories are close to 0. The actual skin disease category of this sample is benign keratosis. From the heat map, it can be clearly seen that when the original hair-bearing skin disease image is used for skin disease category diagnosis and recognition, the region around the lesion has a greater contribution to the final prediction classification result, while the contribution of the region where the lesion itself is located is smaller. When the skin disease image after removing the hair is used for skin disease category diagnosis and recognition, from the heat map, the lesion area has a significant contribution to the final prediction classification result, and the contribution of the remaining regions is small.

[0113] Exemplarily, in the embodiments of the present application, there are totally 7502 skin disease images (i.e., first skin images) without hair and corresponding to 10 skin disease categories (actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesion) in the first skin image dataset, and the number of images of each category is respectively: 285 images of actinic keratosis, 1243 images of basal cell carcinoma, 90 images of dermatofibroma, 2190 images of melanoma, 2190 images of nevus, 631 images of benign keratosis, 399 images of seborrheic keratosis, 95 images of solar lentigo, 232 images of squamous cell carcinoma, and 147 images of vascular lesion. There are totally 3280 skin disease images (i.e., second skin images) without hair and corresponding to 10 skin disease categories in the second skin image dataset, and the number of images of each category is respectively: 59 images of actinic keratosis, 417 images of basal cell carcinoma, 31 images of dermatofibroma, 556 images of melanoma, 1734 images of nevus, 103 images of benign keratosis, 275 images of seborrheic keratosis, 15 images of solar lentigo, 52 images of squamous cell carcinoma, and 38 images of vascular lesion. Due to the uneven distribution of categories, when the dataset is divided, 20% of the image data of each category is randomly extracted as a test set, and the remaining 80% of the image data is used as a training set to prevent the problem of insufficient samples caused by the small amount of data of individual categories.

[0114] The second aspect of the embodiments of the present application further provides a skin image classification device, which refers to Figure 6 , Figure 6 The structure of the skin image classification device is shown in the structure diagram, as Figure 6 shown, the device comprises:

[0115] An image acquisition module is configured to acquire a first skin image dataset and a second skin image dataset, the skin in each first skin image in the first skin image dataset is without hair, and the skin in each second skin image in the second skin image dataset is with hair;

[0116] A hair simulation module is configured to simulate hair for each first skin image in the first skin image dataset to obtain a hair simulation image;

[0117] A first training module is configured to train the hair removal network by using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network;

[0118] A second training module is configured to train a skin image classification network by using the first skin image dataset to obtain an initial skin image classification network.

[0119] a third training module configured to train the initial hair removal network and the initial skin image classification network by using the second skin image dataset, to obtain a trained hair removal network and a trained skin image classification network;

[0120] a skin image classification module configured to input a to-be-detected skin image into the trained hair removal network to obtain a processed detection image after hair removal, and input the processed detection image into the trained skin image classification network to obtain a skin classification result of the to-be-detected image.

[0121] In a possible implementation, the hair simulation module comprises:

[0122] a first hair simulation submodule configured to generate a random curve on a blank image by using a Bezier curve algorithm to obtain a simulated hair image;

[0123] a second hair simulation submodule configured to fuse the simulated hair image with the first skin image to obtain the hair simulation image.

[0124] In a possible implementation, the network architecture of the hair removal network is a UNet network architecture, and the hair removal network uses a feature fusion module based on dynamic snake convolution to replace a convolution module in the UNet network architecture.

[0125] In a possible implementation, the training of the hair removal network by using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network comprises:

[0126] inputting the hair simulation image into the hair removal network to generate a first hair-removed image;

[0127] calculating an average absolute error loss between the first hair-removed image and the corresponding first image;

[0128] updating parameters of the hair removal network according to the average absolute error loss;

[0129] repeating the above steps until a preset training number is reached, and ending the training to obtain the initial hair removal network.

[0130] In a possible implementation, the training of the skin image classification network by using the first skin image dataset to obtain an initial skin image classification network comprises:

[0131] inputting the first skin image into the skin image classification network according to a skin type label of the first skin image, the skin type label being a real type of skin in the first skin image, to obtain a skin classification prediction result for the first skin image;

[0132] calculating a first cross-entropy loss according to the skin classification prediction result for the first skin image and the skin type label of the first skin image;

[0133] updating parameters of the skin image classification network according to the first cross-entropy loss;

[0134] repeating the above steps until a preset training number is reached, ending the training, and obtaining the initial skin image classification network.

[0135] In a possible implementation, the training of the initial hair removal network and the initial skin image classification network using the second skin image dataset to obtain a trained hair removal network and a trained skin image classification network includes:

[0136] inputting a second skin image in the second skin image dataset into the initial hair removal network to generate a second hair-removed image;

[0137] inputting the second hair-removed image into the initial skin image classification network to obtain a skin classification prediction result for the second skin image;

[0138] calculating a second cross-entropy loss between the skin classification prediction result for the second skin image and a skin type label of the second skin image, the skin type label being a real type of skin in the second skin image;

[0139] updating parameters of the initial hair removal network and the initial skin image classification network according to the second cross-entropy loss;

[0140] repeating the above steps until a preset training number is reached, ending the training, and obtaining the trained hair removal network and the trained skin image classification network.

[0141] In a possible implementation, the type of skin in the first skin image or the second skin image includes at least actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, benign keratosis, seborrheic keratosis, solar lentigo, squamous cell carcinoma, and vascular lesions.

[0142] The embodiments of the present application also provide an electronic device, which refers to Figure 7 , Figure 7is a schematic diagram of an electronic device proposed by an embodiment of the present application. As shown in Figure 7 The electronic device 100 includes a memory 110 and a processor 120, the memory 110 and the processor 120 are in communication connection through a bus, the memory 110 stores a computer program, the computer program can run on the processor 120, and then the steps in the skin image classification method based on the hair removal network disclosed in the embodiment of the present application are implemented.

[0143] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps in the skin image classification method based on the hair removal network disclosed in the embodiment of the present application.

[0144] The embodiment of the present application also provides a computer program product, which, when running on an electronic device, causes a processor to implement the steps in the skin image classification method based on the hair removal network disclosed in the embodiment of the present application.

[0145] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0146] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal equipment produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0147] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0148] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, thus the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the flowchart Figure 1 one block or multiple blocks.

[0149] Although the preferred embodiments of the present application have been described, those skilled in the art who understand the basic inventive concept after getting to know the present application can make additional changes and modifications to the embodiments. Therefore, the appended claims are intended to cover the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0150] Finally, it should also be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or terminal device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or terminal device including the element.

[0151] The above describes in detail the method, device and product provided by the present application for classifying skin images based on a hair removal network. The principles and implementation modes of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A skin image classification method based on a hair removal network, characterized in that, The methods include: Obtain a first skin image dataset and a second skin image dataset, wherein the skin in each first skin image in the first skin image dataset is hairless, and the skin in each second skin image in the second skin image dataset is hairy; For each first skin image in the first skin image dataset, perform hair simulation to obtain a hair simulation image; Using the hair simulation image and the corresponding first skin image, the hair removal network is trained to obtain an initial hair removal network; Using the first skin image dataset, the skin image classification network is trained to obtain the initial skin image classification network; Using the second skin image dataset, the initial hair removal network and the initial skin image classification network are trained to obtain the trained hair removal network and the trained skin image classification network. The skin image to be detected is input into the trained hair removal network to obtain a processed detection image after hair removal. The processed detection image is then input into the trained skin image classification network to obtain the skin classification result of the skin image to be detected. The step of performing hair simulation on each first skin image in the first skin image dataset to obtain a hair simulation image includes: On a blank image, a random curve is generated using the Bézier curve algorithm to obtain a simulated hair image; The simulated hair image is fused with the first skin image to obtain the hair simulation image; The hair removal network uses a UNet network architecture, and the hair removal network replaces the convolutional modules in the UNet network architecture with a feature fusion module based on dynamic snake convolution.

2. The skin image classification method based on a hair removal network according to claim 1, characterized in that, The step of training the hair removal network using the simulated hair image and the corresponding first skin image to obtain an initial hair removal network includes: The hair simulation image is input into the hair removal network to generate the first image after hair removal. Calculate the mean absolute error loss between the first hair-removed image and the corresponding first skin image; The parameters of the hair removal network are updated based on the mean absolute error loss. Repeat the training steps until the preset number of training iterations is reached, then end the training to obtain the initial hair removal network.

3. The skin image classification method based on a hair removal network according to claim 1, characterized in that, The step of training the skin image classification network using the first skin image dataset to obtain the initial skin image classification network includes: Using the true skin type in the first skin image as the skin type label, the first skin image is input into the skin image classification network to obtain the skin classification prediction result for the first skin image; Based on the skin classification prediction result for the first skin image and the skin type label of the first skin image, calculate the first cross-entropy loss; The parameters of the skin image classification network are updated based on the first cross-entropy loss; Repeat the training steps until the preset number of training iterations is reached, then end the training to obtain the initial skin image classification network.

4. The skin image classification method based on a hair removal network according to claim 1, characterized in that, The step of training the initial hair removal network and the initial skin image classification network using the second skin image dataset to obtain the trained hair removal network and the trained skin image classification network includes: The second skin image from the second skin image dataset is input into the initial hair removal network to generate the second hair-removed image; The image after removing the second hair is input into the initial skin image classification network to obtain the skin classification prediction result for the second skin image. Using the true type of the skin in the second skin image as the skin type label of the second skin image, calculate the second cross-entropy loss between the skin classification prediction result for the second skin image and the skin type label of the second skin image; Based on the second cross-entropy loss, the parameters of the initial hair removal network and the initial skin image classification network are updated; Repeat the training steps until the preset number of training iterations is reached, then end the training to obtain the trained hair removal network and the trained skin image classification network.

5. The skin image classification method based on a hair removal network according to claim 1, characterized in that, The skin type in the first skin image or the second skin image includes at least: actinic keratosis, basal cell carcinoma, dermatofibroma, melanoma, nevus, pigmented benign keratosis, seborrheic keratosis, solar freckles, squamous cell carcinoma, and vascular lesions.

6. A skin image classification device based on a hair removal network, characterized in that, The device includes: The image acquisition module is used to acquire a first skin image dataset and a second skin image dataset, wherein the skin in each first skin image in the first skin image dataset does not contain hair, and the skin in each second skin image in the second skin image dataset contains hair; The hair simulation module is used to perform hair simulation on each first skin image in the first skin image dataset to obtain a hair simulation image. The first training module is used to train the hair removal network using the hair simulation image and the corresponding first skin image to obtain an initial hair removal network. The second training module is used to train the skin image classification network using the first skin image dataset to obtain an initial skin image classification network. The third training module is used to train the initial hair removal network and the initial skin image classification network using the second skin image dataset to obtain the trained hair removal network and the trained skin image classification network. The skin image classification module is used to input the skin image to be detected into the trained hair removal network to obtain the processed detection image after hair removal, and input the processed detection image into the trained skin image classification network to obtain the skin classification result of the skin image to be detected. The hair simulation module includes: The first hair simulation submodule is used to generate random curves on a blank image using the Bézier curve algorithm to obtain a simulated hair image. The second hair simulation submodule is used to fuse the simulated hair image with the first skin image to obtain the hair simulation image; The hair removal network uses a UNet network architecture, and the hair removal network replaces the convolutional modules in the UNet network architecture with a feature fusion module based on dynamic snake convolution.

7. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the skin image classification method based on a hair removal network as described in any one of claims 1-5.

8. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the processor to implement the skin image classification method based on hair removal network as described in any one of claims 1-5.

Images