An intelligent classification device for skin lesion images based on self-supervised learning

Abstract

This invention relates to an intelligent classification device for skin lesion images based on self-supervised learning, belonging to the field of image processing technology. The device includes a basic module, an image acquisition module, an image processing module, a neural network training module, a neural network classification module, and an output display module. The neural network training module applies a multi-channel self-supervised learning training method to obtain a neural network suitable for skin feature extraction. This addresses the overfitting problem of existing neural networks caused by insufficient numbers of skin lesion images. Ultimately, this invention provides a skin lesion image classification method and device suitable for rapid detection in clinical settings, reducing the burden on dermatologists and improving service efficiency.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to an intelligent classification device for skin lesion images based on self-supervised learning. Background Technology

[0002] Skin diseases, as non-fatal diseases worldwide, affect people of different ages and regions. Furthermore, there are many types of skin diseases, ranging from common squamous skin conditions to more serious ones like melanoma. Different skin diseases share similar visual characteristics, while the lesions of the same disease can vary greatly in different locations, making diagnosis difficult for general practitioners. Patients are increasingly younger and live in economically underdeveloped areas, while the limited number and uneven distribution of dermatologists hinders timely treatment.

[0003] In recent years, the rise and development of theories and technologies such as big data and artificial intelligence have provided new research ideas for the auxiliary diagnosis and treatment of skin diseases. Developing auxiliary diagnostic and treatment systems for skin diseases using artificial intelligence technologies such as image processing, pattern recognition, and machine learning has become a current research hotspot. Skin diseases, as globally recognized as the most suitable diseases for visual diagnosis through photographs, should benefit from the use of next-generation information technology to reduce the burden on dermatologists, improve the level of medical modernization, optimize resource allocation, and increase service efficiency.

[0004] The classification of skin lesion images using intelligent information technology mainly falls into two categories: traditional techniques and machine learning methods. Traditional information technology requires manually designing feature extractors for each skin lesion feature in each type of skin image before classification, resulting in low efficiency and accuracy. Machine learning methods face challenges related to both the input image and the classification categories. Existing artificial intelligence-based skin lesion image classification systems use dermoscopic images as input, which is not portable enough for assisted diagnosis. Furthermore, existing skin lesion image classification networks focus primarily on skin cancers with low incidence rates, with limited research on neural networks for classifying common skin lesions.

[0005] Neural networks excel at uncovering hidden patterns in training samples for rapid classification. However, neural network models require a large number of training samples, and overfitting easily occurs when the sample size is small. Currently, there is a severe shortage of open-environment skin lesion image data. Furthermore, existing open-environment skin lesion image categories do not accurately reflect actual incidence rates, focusing primarily on low-incidence skin cancers; while images of common skin lesions such as acne and eczema (around 200 types) are scarce. These factors make it difficult to apply neural networks to skin lesion image classification. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide an intelligent classification device for skin lesion images based on self-supervised learning. By constructing a neural network and using self-supervised learning to assist in tasks, the feature feature layer of the neural network can extract multiple features from skin images. Then, the output of the feature extraction layer of the neural network is classified to achieve neural network classification of skin lesion images, thereby reducing the burden on dermatologists and improving service efficiency.

[0007] To achieve the above objectives, the specific solution adopted by the present invention is as follows:

[0008] A skin lesion image intelligent classification device based on self-supervised learning includes a basic module, an image acquisition module, an image processing module, a neural network training module, a neural network classification module, and an output display module;

[0009] The basic module includes a power supply, a processor, a memory, and an interactive bus;

[0010] The image acquisition module is used to acquire images of skin lesions in an open environment;

[0011] The image processing module processes the acquired skin lesion images and inputs the processed images into the neural network training module and the neural network classification module, respectively.

[0012] The neural network classification module uses a skin image classification neural network to perform classification operations on the image to obtain classification prediction results, and outputs and displays the classification prediction results through the output display module;

[0013] The neural network training module updates the skin image classification neural network and synchronizes it to the neural network classification module after the output is displayed.

[0014] Preferably, the image processing module includes an image processing program. , , , ,in , , The output is used by the neural network training module and the processing program. The results are used by the neural network classification module;

[0015] Image processing program The image in the storage is randomly flipped horizontally, randomly flipped vertically, randomly cropped to a different size, and then resized to 224*224 pixels to obtain the final image. Image processing program Enhance the training images in the memory to increase the amount of data and prevent the neural network from overfitting.

[0016] Image processing program For images Random regions in the image are discarded, and the discarded regions are filled with 255. The image after discarding is denoted as [image name missing]. The image of the discarded region is denoted as and will and Perform normalization processing;

[0017] Image processing program For images The image is obtained by performing two random color dithering, grayscale transformation, and Gaussian blur operations. , Since both operations are performed randomly, , They are not necessarily completely equal, and the final result is... and Perform normalization processing;

[0018] Image processing program For images Perform a random flip, with the angle limited to (0°, 90°, 180°, 270°), and retain the flipped image. and and their corresponding tags (0,1,2,3), finally for the image and Perform normalization processing;

[0019] Image processing program The newly captured image in the memory is resized to maintain a size of 224*224 pixels. Then, the image is normalized and standardized to ensure that the captured image is not affected by light.

[0020] Preferably, the neural network training module includes a self-supervised learning pre-trained neural network method and a fine-tuning pre-trained neural network method; a self-supervised learning neural network is constructed using the self-supervised learning pre-trained neural network method, a pre-trained neural network is obtained by training the self-supervised learning neural network, and then a skin image classification neural network is finally obtained by fine-tuning the pre-trained neural network method, and the skin image classification neural network is applied to the neural network classification module.

[0021] Furthermore, the neural network training module, the training method for the skin image classification neural network, specifically includes the following steps:

[0022] A. Construct a dataset consisting of unlabeled skin images and labeled skin lesion images in the device's memory;

[0023] B. Divide the dataset into training, test, and validation sets using K-fold cross-validation.

[0024] C. Establish the backbone network of the self-supervised learning pre-trained neural network method, discard the pooling layers and classifier of ResNet50, retain the remaining convolutional layers as feature extraction layers, and set the modified network as the backbone network of the self-supervised learning pre-trained neural network method.

[0025] D. Establish a self-supervised learning image matting auxiliary task branch network, set up a branch network with five upconvolution modules, the output of each upconvolution module is twice the input, and perform interpolation operation between the final output and the original image to calculate the loss;

[0026] E. Establish a self-supervised learning contrastive learning auxiliary task branch network. Set up a contrastive learning branch network with a prediction head. The prediction head consists of two multilayer perceptrons and a ReLU activation function between the multilayer perceptrons. Each multilayer perceptron consists of a fully connected layer and a BN layer. The network calculates the similarity between input images, so that the greater the similarity between the enhanced images, the smaller the similarity with other images.

[0027] F. Establish a self-supervised learning discrimination auxiliary task branch network, setting up a stitching layer, a pooling layer and a prediction layer. The prediction layer consists of a ReLU activation function and a fully connected layer. This branch network determines the rotation relationship between two images and calculates the loss through the cross-entropy loss function.

[0028] G. Combining the backbone network, the matting auxiliary task branch network, the contrast learning auxiliary task branch network, and the discrimination auxiliary task branch network to form a self-supervised learning neural network;

[0029] H. The training set from step A is processed by the image processing module. , , The program then obtains the image. , ;

[0030] I. Image Input the backbone network and the matting auxiliary task branch network and calculate the loss. , Input the backbone network and the contrastive learning auxiliary task branch network and calculate the loss. Input the backbone network and the auxiliary task branch network, and calculate the loss;

[0031] J. Combine the losses of the three auxiliary tasks proportionally and update the self-supervised learning neural network using gradient descent.

[0032] K. Modify the trained self-supervised learning neural network by connecting pooling layers and fully connected layers to the backbone network to obtain a pre-trained neural network;

[0033] L. Train the pre-trained neural network using the training set portion of the dataset containing labeled images, modifying the learning rate of the pre-trained neural network, and setting the learning rate of the feature extraction layer to [value missing]. The learning rate for the fully connected layer is set to ;

[0034] M. Set the classification loss function of the pre-trained neural network to the balanced cross-entropy function to solve the sample imbalance, and use gradient descent to update the network;

[0035] N. Select the pre-trained neural network with the highest accuracy on the validation set of labeled skin lesion images as the final trained neural network, and use it in the neural network classification module.

[0036] Beneficial Effects: This invention proposes an intelligent classification method and device for skin lesion images based on self-supervised learning. The device includes an image acquisition module, an image processing module, a neural network training module, a neural network classification module, and an output display module. In the neural network training module, a multi-channel self-supervised learning training method is applied to obtain a neural network suitable for skin feature extraction. This addresses the overfitting problem in existing technologies caused by insufficient numbers of skin lesion images, ultimately resulting in a skin lesion image classification method and device suitable for rapid detection in clinical settings. Attached Figure Description

[0037] Figure 1 This is a structural diagram of a skin image classification device;

[0038] Figure 2 This is a structural diagram of the image processing module;

[0039] Figure 3 This is a diagram of the neural network training module structure;

[0040] Figure 4 This is a structural diagram of a neural network classification module;

[0041] Figure 5 This is a flowchart of a skin image classification device;

[0042] Figure 6 This is a flowchart of the neural network training module. Detailed Implementation

[0043] This invention proposes a method and apparatus for classifying skin lesion images in an open environment. The apparatus acquires skin lesion images in an open environment using an image acquisition module, then processes the acquired images using an image processing module. The processed images are then input into a neural network training module and a neural network classification module. The neural network classification module performs classification operations on the images and outputs the results through an output display module. The neural network training module updates the neural network and synchronizes it to the neural network classification module after the output is displayed. The basic modules include a power supply, a processor, a memory, and an interactive bus. The device structure diagram is shown below. Figure 1 As shown.

[0044] The main processes of each module and method of this patent are explained in detail below:

[0045] (I) Basic Module

[0046] This module includes a power supply, a processor, a memory, and an interactive bus. The power supply provides power to the skin classification device to ensure normal operation, and the processor interacts with the memory via the bus to ensure normal storage and retrieval of data by each module of the classification device.

[0047] (ii) Image acquisition module

[0048] This module includes a lens and a control device; the user controls the lens to capture skin images in an open environment and saves the images to the memory of the basic module.

[0049] (III) Image Processing Module

[0050] The image processing module contains image processing programs. , , , ,in , , The output is used by the neural network training module and the processing program. The results are used by the neural network classification module. Its module structure diagram is as follows: Figure 2 As shown.

[0051] Image processing program The image in the storage is randomly flipped horizontally, randomly flipped vertically, and randomly cropped (the cropped area is 50%~100% of the original image area). The image size is then adjusted to 224*224 pixels to obtain the final image. Image processing program The training images in the memory are enhanced to increase the amount of data and prevent the neural network from overfitting.

[0052] Image processing program For images Random regions in the image are discarded (the area of ​​the discarded region does not exceed the area of ​​the image). (1 / 4 of the area), fill the discarded area of ​​the image with 255, and denote the image after discarding as The image of the discarded region is denoted as and will and Normalization is performed.

[0053] Image processing program For images The image is obtained by performing two random color dithering, grayscale transformation, and Gaussian blur operations. , Since both operations are performed randomly, , They are not necessarily completely equal, and the final result is... and Normalization is performed.

[0054] Image processing program For images Perform a random flip, with the angle limited to (0°, 90°, 180°, 270°), and retain the flipped image. and and their corresponding tags (0,1,2,3), finally for the image and Normalization is performed.

[0055] Image processing program The newly captured image in the memory is resized to maintain a size of 224*224 pixels. Then, the image is normalized and standardized to ensure that the captured image is not affected by light.

[0056] (iv) Neural Network Training Module

[0057] The neural network training module mainly includes a self-supervised learning pre-trained neural network method and a fine-tuning pre-trained neural network method. A self-supervised learning neural network is constructed using the self-supervised learning pre-trained neural network method. The pre-trained neural network is obtained by training the self-supervised learning neural network. Then, the skin image classification neural network is finally obtained through the fine-tuning method, and this skin image classification neural network is applied to the neural network classification module. The flowchart of the neural network training module is as follows: Figure 3 As shown.

[0058] 1. Self-supervised learning pre-trained neural network method

[0059] The self-supervised learning neural network method establishes a self-supervised learning neural network by adding three auxiliary tasks—image matting, contrast learning, and discrimination—to the backbone network. The image is input through the image processing module to train the self-supervised learning neural network. With the help of the loss function of the joint auxiliary tasks, a pre-trained neural network after self-supervised learning is finally obtained.

[0060] (1) Establish a backbone network

[0061] ResNet50 was selected as the base network for the backbone network. The ResNet50 was modified by discarding the pooling layers and classifiers, and retaining the convolutional layers at the beginning as feature extraction layers. The modified network was then set as the backbone network for the self-supervised learning pre-trained neural network method.

[0062] (2) Establish a branch network for image matting auxiliary tasks

[0063] A branch network with five convolutional modules is established, where the output sample of each convolutional module is twice that of the input, so that the final output feature map has the same dimensions as the image after the image processing module. The number of convolutional kernels in each convolutional module is half that of the previous convolutional module. The initial number of convolutional kernels in the convolutional module is 1024, and the number of convolutional kernels in the last convolutional module is 3, so that the feature map output by the matting auxiliary task branch network has a dimension of 3.

[0064] The image processing module has been processed Image after processing The main network and the matting auxiliary task branch network are input to obtain the final output feature vector. .Will And the original image and image processing module The mask image is used to perform interpolation calculations on the matting regions according to the loss function, thereby determining the specific loss of the matting auxiliary task branch. The loss function is as follows:

[0065] (1)

[0066] in Represents the product of individual pixels in an image; The masked area is 1, and the non-masked area is 0.

[0067] (3) Establish a contrastive learning auxiliary task branch network

[0068] A contrastive learning branch network with a prediction head is constructed. The prediction head consists of two multilayer perceptrons and a ReLU activation function between them. Each multilayer perceptron consists of a fully connected layer and a batch normalization (BN) layer. The image processing module then... Image after processing , The final output feature vectors are obtained by inputting the main network and the contrastive learning auxiliary task branch network respectively. and Then calculate the cosine similarity between the two feature vectors. The formula is as follows:

[0069] (2)

[0070] in represent Transpose of eigenvectors Representing vectors norm, The range of cosine similarity can be adjusted. This not only requires calculating the enhanced image... , To determine the similarity between images, we also need to calculate the similarity between the enhanced images and other input images, ensuring that the greater the similarity between the enhanced images, the smaller the similarity with other images. By exponentializing the cosine similarity and combining it with the cross-entropy loss function, we obtain the loss function for the contrastive learning auxiliary task:

[0071] (3)

[0072] (4) Establish a discrimination auxiliary task branch network

[0073] The discrimination auxiliary task branch network consists of a splicing layer, a pooling layer, and a prediction layer. The prediction layer consists of a ReLU activation function and a fully connected layer. The splicing layer connects the backbone network... and The feature vectors of the images are concatenated, and a pooling layer reduces the dimensionality of the concatenated feature vectors. Finally, a prediction layer determines the rotation relationship between the two images and uses the labels as a basis. The true values ​​are used to calculate the loss using a loss function. The discrimination auxiliary task allows the feature extraction layer of the self-supervised network to learn high-level semantic information about the image. Its loss function is shown below:

[0074] (4)

[0075] in This represents the value predicted by the fully connected layer. The transformation prediction probability of the network is expressed by the following formula:

[0076] (5)

[0077] (5) Joint loss function and parameter updates of the backbone network and three branch networks

[0078] By combining the losses from image matting assistance, contrast learning assistance, and discrimination assistance, and updating the backbone network and the branch networks for each assistance task according to a certain ratio, the backbone network of the self-supervised learning neural network acquires the overall perceptual ability of the image, the surface feature details of the image, the contextual consistency of the image, and the high-level semantic information of the image, ultimately resulting in a pre-trained neural network. The joint feature loss function of the self-supervised learning pre-trained neural network is shown below:

[0079] (6)

[0080] in The loss function value is updated on the self-supervised neural network using gradient descent (SGD) to achieve better feature extraction. In gradient descent... This indicates the parameters to be extracted. The formula for the learning rate is as follows:

[0081] (7)

[0082] 2. Fine-tuning pre-trained neural networks

[0083] By using the fine-tuning pre-trained neural network method, the pre-trained network obtained from self-supervised learning is subjected to supervised learning, and the parameters of the pre-trained network are fine-tuned to finally obtain a skin image classification neural network.

[0084] 1) Modify the pre-trained neural network and learning rate

[0085] Supervised parameter tuning of the self-supervised pre-trained network is performed using labeled skin images stored in the device's memory. First, the self-supervised pre-trained neural network is modified by adding a... The classifier is configured to classify different types of skin lesion images based on the types of labeled images stored in memory. Then, the modified network parameters are adjusted using the labeled images stored in memory, with the learning rate of the self-supervised pre-trained neural network set to [value missing]. The learning rate for the fully connected layer is set to This ensures the combination of rapid convergence of the fully connected layers and fine-tuning of the feature extraction layers. Finally, K-fold cross-validation is used to adjust the pre-trained network.

[0086] 2) Fine-tuning the classification loss function of the neural network

[0087] Because the incidence rates of diseases vary, the set of labeled skin lesion images collected in the device's memory becomes imbalanced. This causes the neural network to prefer classifying images as diseases with a higher sample count in the dataset, thus improving precision. However, this results in a lower overall recall rate for the neural network, leading to frequent misdiagnosis of skin diseases with lower incidence rates. Therefore, a balanced cross-entropy function is used as the overall loss function for the labeled skin images in memory to balance the sample imbalance. The balanced cross-entropy function is:

[0088] (8)

[0089] in The value is the inverse ratio of the number of samples in each category in the memory, thus ensuring that the loss function can also be updated well for skin diseases with a small number of samples.

[0090] (v) Neural Network Classification Module

[0091] The neural network classification module primarily uses the skin image classification neural network obtained from the neural network training module to classify the images processed by the image processing module. The processing program performs classification prediction on the input image, and then processes and stores the prediction results. The flowchart of the neural network classification module is as follows: Figure 4 As shown.

[0092] The processed skin images stored in memory are input into a skin image classification neural network. This network predicts the diseases associated with the input images. The neural network classification module processes the predicted disease values, selects the highest predicted value and its corresponding disease, and saves them to memory.

[0093] (vi) Output display module

[0094] This module mainly includes a display device and a playback device; the highest predicted value of the input image stored in the memory and its disease information are displayed to the photographer through the display device and the playback device.

[0095] Implementation Description: This invention presents a neural network classification method for skin images in an open environment. It includes flowcharts of the skin image classification device and the neural network training module.

[0096] The detailed steps are as follows:

[0097] Step 1: In an open environment, the subject uses the image acquisition module program of the device to acquire skin images and store them in the device's memory.

[0098] Step 2: Process the images taken in Step 1 through the program in the device's image processing module. The images are then resized, normalized, and standardized to ensure they are not affected by light conditions before being stored in the device's memory.

[0099] Step 3: The processed images in the memory are sent to the neural network classification module. The skin image classification neural network is used to predict diseases in the images. The prediction results are then processed, and the highest prediction value and its corresponding disease are selected and saved to the memory.

[0100] Step 4: Display and play the highest predicted value of the input image stored in the memory along with its disease information through the output display module.

[0101] In addition, the described steps are preceded by training the neural network, which is mainly accomplished by the neural network training module and includes the following steps:

[0102] A. Construct a dataset consisting of unlabeled skin images and labeled skin lesion images in the device's memory;

[0103] B. Divide the dataset into training, test, and validation sets using K-fold cross-validation.

[0104] C. Establish the backbone network of the self-supervised learning pre-trained neural network method, discard the pooling layers and classifier of ResNet50, retain the remaining convolutional layers as feature extraction layers, and set the modified network as the backbone network of the self-supervised learning pre-trained neural network method.

[0105] D. Establish a self-supervised learning image matting auxiliary task branch network, set up a branch network with five upconvolution modules, the output of each upconvolution module is twice the input, and perform interpolation operation between the final output and the original image to calculate the loss;

[0106] E. Establish a self-supervised learning contrastive learning auxiliary task branch network. Set up a contrastive learning branch network with a prediction head. The prediction head consists of two multilayer perceptrons and a ReLU activation function between the multilayer perceptrons. Each multilayer perceptron consists of a fully connected layer and a BN layer. The network calculates the similarity between input images, so that the greater the similarity between the enhanced images, the smaller the similarity with other images.

[0107] F. Establish a self-supervised learning discrimination auxiliary task branch network, setting up a stitching layer, a pooling layer and a prediction layer. The prediction layer consists of a ReLU activation function and a fully connected layer. This branch network determines the rotation relationship between two images and calculates the loss through the cross-entropy loss function.

[0108] G. Combine the backbone network, the matting auxiliary task branch network, the contrast learning auxiliary task branch network, and the discrimination auxiliary task branch network to form a self-supervised learning network;

[0109] H. The training set from step A is processed by the image processing module. , , The program then obtains the image. , ;

[0110] I. Image Input the backbone network and the matting auxiliary task branch network and calculate the loss. , Input the backbone network and the contrastive learning auxiliary task branch network and calculate the loss. Input the backbone network and the auxiliary task branch network, and calculate the loss;

[0111] J. Combine the losses of the three auxiliary tasks proportionally and update the self-supervised learning network using gradient descent.

[0112] K. Modify the trained self-supervised network by connecting pooling layers and fully connected layers to the backbone network to obtain a pre-trained neural network;

[0113] L. Train the pre-trained neural network using the training set portion of the dataset containing labeled images, modifying the learning rate of the pre-trained neural network, and setting the learning rate of the feature extraction layer to [value missing]. The learning rate for the fully connected layer is set to ;

[0114] M. Set the classification loss function of the pre-trained neural network to the balanced cross-entropy function to solve the sample imbalance, and use gradient descent to update the network;

[0115] N. Select the pre-trained neural network with the highest accuracy on the validation set of labeled skin lesion images as the final trained neural network, and use it in the neural network classification module.

[0116] It should be noted that the above-described embodiments should be understood as illustrative, not as limiting the scope of protection of this invention. The scope of protection of this invention is defined by the claims. For those skilled in the art, some non-essential improvements and adjustments made to this invention without departing from the essence and scope of this invention still fall within the scope of protection of this invention.

Claims

1. A smart classification device for skin lesion images based on self-supervised learning, characterized in that: It includes a basic module, an image acquisition module, an image processing module, a neural network training module, a neural network classification module, and an output display module; The basic module includes a power supply, a processor, a memory, and an interactive bus; The image acquisition module is used to acquire images of skin lesions in an open environment; The image processing module processes the acquired skin lesion images and inputs the processed images into the neural network training module and the neural network classification module, respectively. The neural network classification module uses a skin image classification neural network to perform classification operations on the image to obtain classification prediction results, and outputs and displays the classification prediction results through the output display module; The neural network training module updates the skin image classification neural network and synchronizes it to the neural network classification module after the output is displayed. The image processing module includes image processing programs G0, G1, G2, G3, and G4, wherein the output results of G0, G1, G2, and G3 are used by the neural network training module, and the result of processing program G4 is used by the neural network classification module. Image processing program G0 randomly flips the images in memory horizontally, randomly flips them vertically, randomly crops them to a size of 224*224 pixels, and then adjusts the image size to obtain image I0. Image processing program G0 also enhances the training images in memory to increase the amount of data and prevent overfitting during neural network training. Image processing program G1 discards random regions in image I0, fills the discarded regions with 255, records the discarded image as I1, records the image of the discarded regions as I`1, and normalizes I1 and I`1. Image processing program G2 performs two random color dithering, grayscale transformation, and Gaussian blur operations on image I0 to obtain image I. 21 I 22 Since both operations are performed randomly, therefore I 21 I 22 They are not necessarily completely equal, and the final result for I... 21 and I 22 Normalization is performed; the image processing program G3 randomly flips the image I0 once, with the angle limited to (0°, 90°, 180°, 270°) and retains the flipped images I3 and I`3 and their corresponding labels y3 (0, 1, 2, 3). Finally, the images I3 and I`3 are normalized. The image processing program G4 resizes the newly captured image in the memory to keep the image size at 224*224 pixels, and then performs normalization and standardization processing on the image to ensure that the captured image is not affected by light. The neural network training module includes a self-supervised learning pre-trained neural network method and a fine-tuning pre-trained neural network method. A self-supervised learning neural network is constructed using the self-supervised learning pre-trained neural network method, and a pre-trained neural network is obtained by training the self-supervised learning neural network. Then, the skin image classification neural network is finally obtained by fine-tuning the pre-trained neural network method, and the skin image classification neural network is applied to the neural network classification module. The neural network training module, specifically the training method for the skin image classification neural network, includes the following steps: A. Construct a dataset consisting of unlabeled skin images and labeled skin lesion images in the device's memory; B. Divide the dataset into training, test, and validation sets using K-fold cross-validation. C. Establish the backbone network of the self-supervised learning pre-trained neural network method, discard the pooling layers and classifier of ResNet50, retain the remaining convolutional layers as feature extraction layers, and set the modified network as the backbone network of the self-supervised learning pre-trained neural network method. D. Establish a self-supervised learning image matting auxiliary task branch network, set up a branch network with five upconvolution modules, the output of each upconvolution module is twice the input, and interpolate the final output with the original image to calculate the loss; E. Establish a self-supervised learning contrastive learning auxiliary task branch network. Set up a contrastive learning branch network with a prediction head. The prediction head consists of two multilayer perceptrons and a ReLU activation function between the multilayer perceptrons. Each multilayer perceptron consists of a fully connected layer and a BN layer. The network calculates the similarity between input images, so that the greater the similarity between the enhanced images, the smaller the similarity with other images. F. Establish a self-supervised learning discrimination auxiliary task branch network, setting up a stitching layer, a pooling layer and a prediction layer. The prediction layer consists of a ReLU activation function and a fully connected layer. This branch network determines the rotation relationship between two images and calculates the loss through the cross-entropy loss function. G. Combining the backbone network, the matting auxiliary task branch network, the contrast learning auxiliary task branch network, and the discrimination auxiliary task branch network to form a self-supervised learning neural network; H. The training set from step A is processed through the G0, G1, G2, and G3 procedures of the image processing module to obtain images I1, I`1, and I... 21 I 22 、I3、I`3; I. Input images I1 and I`1 into the backbone network and the matting auxiliary task branch network, and calculate the loss, L. 21 I 22 Input the backbone network and the contrastive learning auxiliary task branch network and calculate the loss. Input I3 and I`3 into the backbone network and the discriminative auxiliary task branch network and calculate the loss. J. Combine the losses of the three auxiliary tasks proportionally and update the self-supervised learning neural network using gradient descent. K. Modify the trained self-supervised learning neural network by connecting pooling layers and fully connected layers to the backbone network to obtain a pre-trained neural network; L. Train the pre-trained neural network using the training set portion of the labeled images in the dataset, and modify the learning rate of the pre-trained neural network, setting the learning rate of the feature extraction layer to 0.001~0.0001 and the learning rate of the fully connected layer to 0.1~0.01; M. Set the classification loss function of the pre-trained neural network to the balanced cross-entropy function to solve the sample imbalance, and use gradient descent to update the network; N. Select the pre-trained neural network with the highest accuracy on the validation set of labeled skin lesion images as the final trained neural network, and use it in the neural network classification module.

Images