A melanoma auxiliary diagnosis method based on deep learning

Abstract

The application discloses a kind of melanoma auxiliary diagnosis methods based on deep learning, using optimized Transform classification network, for melanoma image, ideal auxiliary classification effect is obtained.The application generates malignant melanoma using the way of adversarial generation, makes up the problem of imbalance between classes in data set.The application can judge the benign or malignant of melanoma, the result shows that the accuracy of the application can reach 98.4%, higher than 75% of human eye recognition, can effectively help patients early self-screening, or assist doctors in clinical diagnosis.The application can realize model light weight, reduce the amount of model parameters, reduce the resource and time cost of actual development program, facilitate the application of algorithm to actual development program.The application can apply the related knowledge of computer vision to real life, effectively simplify the process of detecting skin melanoma, so that doctors and patients can obtain detection reference results in real time by simply operating the application program.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a melanoma auxiliary diagnosis method based on deep learning. Background Art

[0002] Malignant melanoma is a malignant tumor that originates in melanocytes, most often evolving from benign melanocytic moles. It is associated with both genetic and physical factors. Early and accurate diagnosis of malignant melanoma is crucial for reducing mortality. Currently, commonly used clinical methods are susceptible to the influence of physician skill and experience, resulting in high subjectivity, low diagnostic efficiency, and long examination cycles. These methods are no longer suitable for modern healthcare needs. Therefore, the use of artificial intelligence (AI) to assist in the diagnosis of melanoma using medical images is becoming a future direction.

[0003] Melanoma closely resembles melanocytic nevi, making diagnosis significantly more challenging. Currently, the primary diagnostic method for melanoma is based on visual observation using a dermatoscope. This method is susceptible to variations in physician skill and experience, resulting in an accuracy rate of 75% to 80% and low diagnostic efficiency. In clinical diagnosis, early malignant transformation of melanocytic nevi is monitored and assessed using methods such as pattern analysis, the "ABCD" rule, and the seven-point method. While simple, these methods have a high misdiagnosis rate and are not guaranteed to be accurate. Physicians often rely on biopsy histological examinations to draw conclusions. Biopsy of stained pathological specimens using a conventional microscope only allows for the observation of two-dimensional spatial features, making the morphological characteristics of the specimen difficult to discern. This process is time-consuming and expensive. Variations in the use of stains and the resulting procedures can also lead to variations in the resulting pathological specimens, and there are currently no tools available to provide more detailed quantitative analysis of stained specimens. The extent of the biopsy procedure is also difficult to determine before a tumor is confirmed to be benign or malignant. The cost of medical diagnosis is rapidly increasing. Skin biopsy is an invasive and expensive procedure, which not only causes physical pain to patients but also places a huge financial burden on them. Therefore, the medical community urgently needs to develop non-invasive and low-cost melanoma diagnosis technologies.

[0004] According to statistics from the World Health Organization (WHO), approximately 132,000 new cases of melanoma are diagnosed worldwide each year. According to the 2022 American Cancer Society statistics, skin cancer is one of the most common types of cancer, and melanoma accounts for 64% of skin cancer mortality. In 2022, 97,920 new cases of melanoma were diagnosed, resulting in 7,650 deaths. Malignant melanoma lacks effective treatment other than early surgical excision. Therefore, early diagnosis and treatment of malignant melanoma are extremely important. The earlier the diagnosis and treatment of melanoma, the better, as it can prolong patient survival and effectively reduce mortality.

[0005] To sum up, the number of melanoma patients is extremely large, but the doctor-patient ratio is seriously unbalanced, medical resources are scarce, and traditional diagnostic methods are expensive, all of which need to be urgently addressed. Summary of the Invention

[0006] Currently, melanoma diagnosis suffers from low efficiency, low examination accuracy, and long diagnostic cycles. The present invention provides a melanoma auxiliary diagnosis method based on deep learning, comprising the following specific steps:

[0007] 1) Data preprocessing: Segmentation and hair removal are performed on the original images of the training, validation, and test sets to accurately identify the skin lesion area and remove factors that interfere with diagnosis, such as hair;

[0008] 2) The number of different categories in the dataset is balanced. To address the imbalance in the training set samples, a generative adversarial network is used to generate malignant melanoma images to balance positive and negative samples. To avoid overfitting and ensure generalization, an adaptively enhanced StyleGAN2 is used to generate images. This means that data augmentation is used in both the generator and the discriminator.

[0009] 3) Network classification model optimization: Based on the Vision Transformer, the number of model heads was changed, and the modified model backbone was combined with the BatchFormer module. At the same time, two shared classifiers were added to enable mutual learning between sample features, further addressing the data imbalance problem and obtaining reliable classification results.

[0010] 4) Model lightweighting: Based on the network in step 3), a model lightweighting operation is performed. A distillation flag is added to the model, and then the knowledge distillation method is used to compress the model. The original features after normalization and sliding time window processing are input into the teacher network for model training. The trained teacher model guides the training of the student model. That is, the teacher model's predictions help train the student model. The two are connected through a loss function. By setting hyperparameter values ​​to control the proportion of soft loss and hard loss, and then performing a sum operation, the loss value of the final distillation model is used to help the student model train better. Finally, after training, a lightweight model with excellent classification performance is obtained.

[0011] 5) Model migration: Using computational model migration technology to integrate model weights with mobile and web applications, this allows models that originally ran on GPUs to run normally and quickly on mobile phones or websites, while also visualizing model results.

[0012] 6) Clinical Validation and Optimization: After transplanting the deep learning model for melanoma-assisted diagnosis, we entered the algorithm testing phase. The mobile and web models were provided to doctors and patients for use. We continuously summarized the test results and optimized the algorithm model to ultimately obtain a reliable algorithm model.

[0013] Preferably, the training method is supervised learning with labeled data.

[0014] Preferably, in order to improve the quality of network-generated images, a multi-scale fusion module is added to the original network to reduce the semantic gap between different feature channel layers.

[0015] Preferably, during training, a portion of the image is generated by two latent codes to prevent correlation between adjacent skin cancer lesion types.

[0016] Preferably, the generator network adds noise to each pixel in the melanoma image after each convolution to achieve diversity and randomness.

[0017] Preferably, in the generative network, the original lesion image is modulated and demodulated, the custom convolutional layer and the demodulated style vector are fused together, the fused feature map is added with bias and noise, and the final accumulated RGB image is output as the generated melanoma image.

[0018] Preferably, the classification and recognition network model does not rely on a fixed-size convolution kernel and can well capture melanoma pathological characteristics including asymmetry, boundary irregularity, color, size, concave-convex shape, differential structure, etc.

[0019] The advantages and beneficial effects of the present invention are:

[0020] The present invention provides a melanoma auxiliary diagnosis method based on deep learning, which uses deep learning methods to assist in the diagnosis of benign and malignant melanoma, and applies it to mobile terminals to help patients with early screening, assist doctors in diagnosis, alleviate patients' difficulty in seeing a doctor, and reduce high mortality rates.

[0021] The present invention has the following characteristics:

[0022] 1) This paper uses an optimized Transformer classification network to achieve ideal results for skin melanoma images. The network is also applicable to other medical image detection.

[0023] 2) The present invention's deep learning-based melanoma-assisted diagnosis method can detect seven types of skin tumors. Results show that this method has a detection accuracy of 98.4%, higher than the 75% achieved by the human eye. This method can facilitate early self-screening for patients and effectively assist doctors in diagnosis.

[0024] 3) This invention can achieve model lightweighting, use knowledge distillation to reduce the number of model parameters, reduce the resource and time costs of actual program development, and facilitate the application of the algorithm to actual program development. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a basic flow chart of the deep learning-based melanoma auxiliary diagnosis method of the present invention;

[0026] Figure 2 A flowchart of image adversarial generation in the present invention;

[0027] Figure 3 This is a diagram of the network model structure in the present invention;

[0028] Figure 4 This is a system flow chart of the lightweight knowledge distillation method of the model of the present invention;

[0029] Figure 5 This is a flow chart of the algorithm design, verification and optimization model method of the present invention. Implementation Method

[0030] The following embodiments are further described in conjunction with the accompanying drawings and examples. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0031] The hardware environment for implementing the solution of the present invention is as follows: the CPU is Intel(R) Xeon(R) CPU E5-2623 v4 @2.60GHz, the GPU is NVIDIA-SMI 470.86, and the operating environment is Python 3.8 and PyTorch.

[0032] A Transformer model is used to process global information in melanoma medical images based on an encoding and decoding network structure. The internal structure of the model and related algorithms are optimized to address the problem that existing auxiliary diagnosis technologies require massive data samples to improve classification and recognition accuracy. This ensures that the classification and recognition network model does not rely on fixed-size convolution kernels and can well capture melanoma pathological characteristics including asymmetry, boundary irregularities, color, size, concave and convex shapes, and differential structures, thereby improving the generalization ability of the melanoma classification and recognition model and the accuracy of auxiliary diagnosis.

[0033] Compared to other mainstream networks, the network presented in this paper overcomes the limitation of RNN models, which cannot perform parallel computations. Compared to CNNs, the number of operations required to calculate the association between two locations does not increase with distance. Furthermore, self-attention can produce a more interpretable model. This paper examines the distribution of attention within the model; individual attention heads can learn to perform different tasks. Furthermore, for melanoma images, this paper incorporates a BatchFormer module to enhance sample features and address the severe dataset imbalance.

[0034] like Figure 1 、 Figure 5As shown, the melanoma auxiliary diagnosis method based on deep learning of the present invention includes melanoma image data acquisition, image preprocessing, image adversarial generation, Transformer network learning classifier, model knowledge distillation lightweight and clinical verification and optimization of the algorithm.

[0035] The melanoma auxiliary diagnosis method based on deep learning of the present invention includes the following specific steps:

[0036] 1) We obtained over 30,000 dermoscopic images from the ISIC database, of which only 584 were malignant melanomas. We divided the dataset into a training set and a validation set. We used supervised learning with labeled data.

[0037] 2) Image segmentation was performed to address image quality issues, identifying the lesion area and performing preprocessing such as hair removal. To unify the image size and accelerate model training, the image size was compressed to 224*224 and pixel value ranges were normalized using numerical normalization techniques.

[0038] 3) To address the imbalance of positive and negative samples in the training set, the present invention uses a generative adversarial network to generate a certain number of malignant melanoma images;

[0039] 4) The generated melanoma images and real samples are fed into the network for training, ultimately obtaining classification results for auxiliary diagnosis;

[0040] 5) Later, the training samples are used to fine-tune the trained classifier model to improve the classification accuracy;

[0041] 6) After the model is trained, the test set is fed into the model for prediction, and the image-level classification evaluation metrics are calculated based on the prediction results. The optimal model weights obtained by the algorithm are transplanted into the application, and then tested with doctors and patients. Based on the results and user feedback, the algorithm and program are continuously adjusted and optimized.

[0042] like Figure 2As shown, the present application is balanced between different categories of data, and an adversarial generative model is introduced. The generator part of the model is mainly divided into three parts, initial latent code, nonlinear mapping network and affine learning. In order to improve the quality of network generated pictures, a multi-scale fusion module is added on the basis of the original network to reduce the semantic gap between different feature channel layers. During training, part of the image is generated by two latent codes, which can prevent the correlation between adjacent skin cancer lesion types. The generation network adds noise to each pixel in the melanoma image after each convolution to achieve diversity and randomness. In the generation network, the original lesion image is modulated and demodulated, and the custom convolution layer and the demodulated style vector are fused together. The fused feature map is added with bias and noise, and the finally accumulated RGB picture is output as the generated melanoma image. At the same time, in order to avoid overfitting during training, adaptive image enhancement method is adopted in the generator and discriminator of the adversarial generation.

[0043] The present application adopts the Transformer model as the classification model backbone, which is composed of an encoder and a decoder, and each module is composed of multiple Transformer blocks. Figure 3 As shown, in order to enhance the features of sample data, BatchFormer is inserted into the ViT network feature extractor to obtain an optimized network structure. BatchFormer module can promote representation learning by exploring sample relationships. A Transformer structure is inserted after the feature extractor, which is along the batch dimension. In order to reduce the gap between training and testing, a pair of shared classifiers is added before and after it, forming the Batch-MVit network in Figure 3 The generated melanoma image and the real sample are input into the network for training, and finally the classification result of auxiliary diagnosis is obtained.

[0044] As shown in Figure 4As shown, the present invention uses knowledge distillation to obtain a lightweight model. Knowledge distillation includes two network structures, a teacher and a student. The teacher network is larger in scale and has a deeper network layer, while the student network is a shallow network with fewer parameters. The predicted value of the teacher network helps train the student network, realizing knowledge transfer between the teacher and student networks. The present invention uses the knowledge distillation method to compress the model, and the overall process can be roughly divided into three stages. In the first stage, the original features after normalization and sliding time window processing are input into the teacher network for model training. The trained teacher model will participate in the training of the student model in the second stage, that is, the predicted value of the teacher model helps train the student model. The two are connected through the loss function. By setting the hyperparameter value to control the proportion of soft loss and hard loss, and then perform the sum operation as the loss value of the final distillation model, it helps the student model to train better results. Since knowledge distillation can transfer knowledge between homogeneous networks and between heterogeneous networks, the first distillation of the method proposed in the present invention is the knowledge transfer of different architectures, and the second distillation selects the same architecture for knowledge transfer. In the third stage, the optimal student model obtained from the first distillation is used as the teacher model for the second distillation, a new student model is trained, and the final melanoma medical image classification and recognition is performed based on the student model obtained after two distillations.

[0045] In summary, the present invention provides a melanoma auxiliary diagnosis method based on deep learning, which adopts the optimized Transformer classification network to achieve ideal auxiliary classification effect for melanoma images. The present invention adopts an adversarial generation method to generate malignant melanoma to make up for the imbalance problem between classes of the data set. The present invention can judge whether melanoma is benign or malignant. The results show that the accuracy of the present invention can reach 98.4%, which is higher than the 75% of human eye recognition. It can effectively help patients with early self-screening or assist doctors in clinical diagnosis. The present invention can achieve model lightweighting, reduce the number of model parameters, reduce the resource and time cost of actual development programs, and facilitate the application of algorithms to actual development programs. The present invention can apply the relevant knowledge of computer vision to real life, effectively simplify the process of detecting skin melanoma, so that doctors and patients can obtain detection reference results in real time by simply operating the application.

[0046] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A melanoma auxiliary diagnosis method based on deep learning, characterized in that: The specific steps include: 1) Data preprocessing: Segmentation and hair removal are performed on the original images for the training, validation, and test sets, respectively, to accurately identify the skin lesion area and remove hair interference factors that interfere with diagnosis; 2) The number of different categories in the dataset is balanced. To address the imbalance in the training set samples, a generative adversarial network is used to generate malignant melanoma images to balance positive and negative samples. To avoid overfitting and ensure generalization, an adaptively enhanced StyleGAN2 is used to generate images, which uses data augmentation in both the generator and the discriminator. 3) Network classification model optimization: Based on the Vision Transformer, the number of model heads was changed; the modified model backbone was combined with the BatchFormer module; and two shared classifiers were added to enable mutual learning between sample features, further addressing the data imbalance problem and obtaining reliable classification results. 4) Model lightweighting: Based on the network in step 3), a model lightweighting operation is performed. A distillation flag is added to the model, and then the knowledge distillation method is used to compress the model. The original features after normalization and sliding time window processing are input into the teacher network for model training; 5) Model migration: Using computational model migration technology to integrate model weights with mobile and web applications, this allows models that originally ran on GPUs to run normally and quickly on mobile phones or websites, while also visualizing model results. 6) Clinical Validation and Optimization: After transplanting the deep learning model for melanoma-assisted diagnosis, we entered the algorithm testing phase. The mobile and web models were provided to doctors and patients for use. We continuously summarized the test results and optimized the algorithm model to ultimately obtain a reliable algorithm model.

2. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: In step 4), the trained teacher model guides the training of the student model. That is, the teacher model's predictions help train the student model, and the two are connected through the loss function. By setting hyperparameter values ​​to control the proportion of soft loss and hard loss, and then performing sum calculations, which serve as the loss value of the final distillation model, the student model can be trained to achieve better results. Finally, after training, a model with excellent classification performance and lightweightness is obtained.

3. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: The training method is supervised learning with labeled data.

4. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: In order to improve the quality of network-generated images, a multi-scale fusion module is added to the original network to reduce the semantic gap between different feature channel layers.

5. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: During training, a portion of the image is generated by two latent codes to prevent correlation between adjacent skin cancer lesion types.

6. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: The generator network adds noise to each pixel in the melanoma image after each convolution to achieve diversity and randomness.

7. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: In the generative network, the original lesion image is modulated and demodulated, the custom convolutional layer and the demodulated style vector are fused together, the fused feature map is added with bias and noise, and the final accumulated RGB image is used as the generated melanoma image output.

8. The melanoma auxiliary diagnosis method based on deep learning according to claim 1, characterized in that: The classification and recognition network model does not rely on fixed-size convolution kernels and can capture melanoma pathological characteristics including asymmetry, boundary irregularity, color, size, concave-convex shape, and differential structure.

Images