Skin state classification method based on progressive transfer learning and modal domain adaptation

Abstract

The invention provides a skin state classification method based on progressive transfer learning and modal domain adaptation, relates to the technical field of skin state recognition based on a neural network, and aims to realize a skin health state classification method which is accurate and efficient and is not limited by labeled data volume and not influenced by clinical modals. Comprising the following steps: constructing a plurality of intermediate data sets through unlabeled samples in different types of skin state fields; adjusting color space and feature distribution of the plurality of intermediate data sets based on HSV color space conversion and a CycleGAN model; on the basis of modal domain adaptation, a progressive transfer learning strategy is adopted, parameters of an identification model are finely adjusted step by step through a plurality of intermediate data sets, and data sources of unlabeled and label-free skin images of different sources and different types are adopted as labels of the skin images; and adjusting the identification model according to the marked target data set through a self-adaptive unfreezing method. The method has the advantages of improving the classification accuracy and reducing the dependence on the marked data.

Description

Technical Field

[0001] The present invention relates to the technical field of skin health recognition based on neural network, and in particular to a skin condition classification method based on progressive transfer learning and modal domain adaptation. Background Art

[0002] The skin is the largest organ in the human body and has important functions such as protection, regulating body temperature, maintaining water balance, and sensing external stimuli.

[0003] Although some specific skin conditions, such as vitiligo, eczema, acne, and pityriasis alba, are not directly life-threatening, the changes in appearance they cause may have a significant impact on an individual's quality of life and bring psychological and economic pressures. In these cases, it is crucial to accurately classify an individual's skin health status. Accurate classification helps determine appropriate treatment strategies and provides guidance for the selection of treatment timing. Currently, doctors usually use information from multiple modalities to comprehensively judge the health status of the skin. For example, vitiligo can be divided into two states, progressive and stable, by combining clinical examination and Wood's lamp examination. Clinical examination shows the edge pigmentation and contour characteristics of white spots, while Wood's lamp examination emphasizes the hypopigmentation under white spots and lesions. However, the accuracy of these classification methods depends largely on the doctor's expertise and experience, as well as the number of cases they have encountered. When dermatologists are underqualified or there is a shortage of professional doctors, the accuracy of their diagnosis is difficult to guarantee. The rapid development of deep neural networks has provided a new solution for the classification of skin health status. Several studies have successfully developed artificial intelligence models for intelligent diagnosis of skin status. However, these methods generally have a limitation: model performance is highly dependent on the number of training samples. In the busy clinical work of dermatology, collecting multimodal data for skin health status classification is complex and expensive, and the limited amount of data limits the performance of deep learning models. Transfer learning can be used as an effective strategy to alleviate the shortage of training data for deep learning models. It pre-trains deep neural networks in sample-rich source domains and then transfers the learned knowledge to the target domain, so that it can learn quickly and efficiently in the target task with limited data. However, several factors affect the effectiveness of transfer learning in practical applications. Generally, most skin state classification methods based on transfer learning use natural image datasets (such as ImageNet) as the source domain for pre-training models, but the domain difference between natural images and medical images may limit the effectiveness of transfer learning. At the same time, in clinical practice, multiple imaging techniques are often used to capture the state characteristics of the same medical sample under multiple modalities. When collecting data from different modalities of skin state learning (such as clinical images and Wood's lamp images), due to the differences between imaging equipment and clinical settings, domain shift and changes in sample distribution may occur, which in turn affect the performance of the model.

[0004] Therefore, a skin health status classification method that is accurate, efficient, and not limited by data volume and clinical modality is needed. Summary of the invention

[0005] The purpose of the present invention is to provide a skin condition classification method based on progressive transfer learning and modal domain adaptation, which can achieve accurate and efficient skin health status classification that is not limited by data volume and is not affected by clinical modalities.

[0006] The present invention is achieved through the following technical solutions:

[0007] The skin state classification method based on progressive transfer learning and modality domain adaptation includes the following steps:

[0008] Construct multiple intermediate datasets through unlabeled samples in different skin condition fields;

[0009] The color space and feature distribution of multiple intermediate data sets are adjusted based on HSV color space conversion and CycleGAN model respectively;

[0010] Based on modal domain adaptation, a progressive transfer learning strategy is adopted to gradually fine-tune the parameters of the recognition model through multiple intermediate data sets. For unlabeled skin images of different types and sources, their data sources are used as their labels in progressive transfer learning.

[0011] The recognition model is adjusted on the labeled target dataset by an adaptive unfreezing method.

[0012] Preferably, the number of the intermediate data sets is two; the first intermediate data set includes dermoscopic images of melanoma, the second intermediate data set includes unlabeled samples from the field of skin images of the target state type to be identified, and the second intermediate data set includes dermoscopic images of multiple hypopigmentation states.

[0013] Preferably, the target category to be identified is vitiligo.

[0014] Preferably, the first intermediate data set is adjusted by the HSV color space conversion, and the method is:

[0015] To perform HSV conversion:

[0016]

[0017] Among them, H represents hue, S represents saturation, V represents brightness, R represents red of RGB value, G represents green of RGB value, B represents blue of RGB value, max(.) and min(.) are functions for finding maximum and minimum respectively;

[0018] Adjust saturation and hue;

[0019] Convert the HSV image back to RGB space based on the adjusted saturation and adjusted hue.

[0020] Preferably, the method for adjusting saturation and hue is:

[0021] Adjust the saturation by the saturation adjustment factor α:

[0022] S′=α×S,0<α<1;

[0023] Among them, S′ is the adjusted saturation;

[0024] If the hue of the pixel does not exceed the preset threshold, the hue of the red area of ​​the dermoscopic image is adjusted by the hue offset ΔH:

[0025] H′=H+ΔH;

[0026] ΔH=(H′ max -H′ min )×(H max -H min ) / (H max -H min );

[0027] Among them, H′ is the adjusted hue, H min and H max are the lower and upper limits of the hue range before adjustment, H′ min and H′ max Respectively, they are the lower and upper limits of the adjusted hue range.

[0028] Preferably, the second intermediate data set is adjusted based on the CycleGAN model, and the method is:

[0029] Based on the second intermediate data set, performing modality conversion using the original modality image to generate a second modality image;

[0030] The adversarial loss function is used to ensure that the generated images match the transformed image distribution.

[0031] The cycle consistency loss function is used to avoid conflicts between the generator and the discriminator.

[0032] Preferably, the adversarial loss function for:

[0033]

[0034] in, The clinical images X of the second intermediate dataset clinTransformed to image Y in target domain wood The generator of D wood is the discriminator, x is X clin The data in , y is Y wood The data in It means to calculate the expected value of the probability distribution of sample y. Indicates the expected value calculation of the probability distribution of sample x;

[0035] The cycle consistency loss function for:

[0036]

[0037] in, To transform the image Y in the target domain wood Convert clinical images X into the second intermediate dataset clin The generator.

[0038] Preferably, the method of adjusting the recognition model on the labeled target data set by the adaptive unfreezing method is:

[0039] Freeze the parameters of layers other than the fully connected layer;

[0040] Set the gradient threshold γ and the performance improvement threshold τ;

[0041] Unfreeze each layer one by one and train it;

[0042] After each unfreezing, the performance of the recognition model on the validation set is obtained, and the absolute average of the gradients of the unfrozen layers is calculated to measure the activity of the layer on the task;

[0043] If the absolute average value of the gradient is greater than a preset threshold, the next layer is unfrozen; otherwise, the weight of the layer is kept unchanged and the unfreezing of subsequent layers is skipped.

[0044] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0045] The present invention effectively classifies skin health status through progressive transfer learning and modal domain adaptation, assists doctors in improving the efficiency and accuracy of diagnosis, and reduces missed diagnoses and misdiagnoses caused by insufficient doctor qualifications or shortage of professional doctors.

[0046] The present invention can realize efficient skin health status recognition under limited labeled data, reducing the dependence on a large amount of target field training data; for unlabeled skin images of different types and sources, their data sources are used as their labels in progressive transfer learning;

[0047] The present invention uses a modal domain adaptation method to effectively reduce the impact of visual differences between different imaging modalities and changes in sample distribution on model performance, thereby improving the generalization ability of the model in multimodal scenarios.

[0048] The present invention dynamically adjusts model parameters through an adaptive parameter unfreezing strategy, thereby avoiding overfitting, saving computing resources, and improving model performance;

[0049] The invention is reasonably designed and can be easily applied to various skin health conditions, especially the identification of different states of vitiligo. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a process flow of a skin state classification method based on progressive transfer learning and modal domain adaptation provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a progressive transfer learning process provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the visual difference between a dermoscopic image, a clinical image and a Wood's lamp image provided by an embodiment of the present invention;

[0053] Figure 4 A flowchart of a domain adaptation method based on HSV color space conversion provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a modal domain adaptation process based on CycleGAN provided in an embodiment of the present invention;

[0055] Figure 6 A schematic diagram of the process of the adaptive thawing method provided by an embodiment of the present invention;

[0056] Figure 7 Schematic diagram of visualization results of two modal domain adaptation methods provided in embodiments of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0058] Example 1

[0059] This embodiment provides a skin condition classification method based on progressive transfer learning and modality domain adaptation, comprising the following steps:

[0060] Construct multiple intermediate datasets through unlabeled samples in different skin condition fields;

[0061] The color space and feature distribution of multiple intermediate data sets are adjusted based on HSV color space conversion and CycleGAN model respectively;

[0062] Based on modal domain adaptation, a progressive transfer learning strategy is adopted to gradually fine-tune the parameters of the recognition model through multiple intermediate data sets. For unlabeled skin images of different types and sources, their data sources are used as their labels in progressive transfer learning.

[0063] The recognition model is adjusted on the labeled target dataset by an adaptive unfreezing method.

[0064] Finally, the original scores can be converted into probability distributions through the training and optimization of the recognition model, and the category with the highest probability is taken as the final classification result of the skin health status.

[0065] As a preferred solution of this embodiment, the number of the intermediate data sets is two; the first intermediate data set includes dermoscopic images of melanoma, the second intermediate data set includes unlabeled samples from the field of skin images of the target state type to be identified, and the second intermediate data set includes dermoscopic images of various hypopigmentation states. Based on this, it can be specifically taken to refer to Figure 1 The following scheme performs the operation of this embodiment:

[0066] Step S1: constructing a first intermediate data set and a second intermediate data set through unlabeled samples in the field of different types of skin conditions, wherein the second intermediate data set includes unlabeled samples in the field of skin conditions of the target state type to be identified; here, a pyramid pooling Transformer (P2T-tiny) pre-trained on ImageNet can be selected as a model to verify this method; for unlabeled skin images of different types from different sources, their data sources are used as their labels in progressive transfer learning;

[0067] Step S2: adjusting the color space and feature distribution of the first intermediate data set and the second intermediate data set respectively by HSV color space conversion and CycleGAN model; the HSV color space conversion technology is used here to convert the first intermediate data set (D1) image from RGB color space to HSV color space, and adjust the hue, saturation and brightness of the dermoscopic image to make it closer to the clinical modality and Wood light modality of the target data set. Secondly, CycleGAN is used for the second intermediate data set (D2), and a new image of the target morphology is synthesized through an adversarial learning mechanism to adapt to the distribution of data sets with different morphologies;

[0068] Step S3: Based on the modal domain adaptation, a progressive transfer learning strategy is adopted to gradually fine-tune the parameters of the recognition model through the first intermediate data set and the second intermediate data set; in this step, the progressive transfer learning strategy is applied to the ImageNet pre-trained model, and the knowledge learned by the model is gradually transferred from ImageNet to the dermatology field using the two intermediate data sets adjusted in step S2, and then a two-step transfer is performed in the dermatology data set, so that the knowledge learned by the model gradually adapts to the target task such as vitiligo status classification;

[0069] Step S4: The recognition model is adjusted on the labeled target data set by an adaptive thawing method; on the accurately labeled target data set, the model is fine-tuned by an adaptive thawing method. In this embodiment, the strategy freezes all layers except the fully connected layer at the beginning of training, and sets thresholds for the gradient and performance improvement of the model. After each thawing, the performance of the model on the validation set is evaluated. If the gradient and performance improvement thresholds are met, the next layer is thawing; otherwise, the weight of the current layer is retained and the subsequent thawing is skipped.

[0070] On this basis, the first intermediate data set includes dermoscopic images of melanoma, and the second intermediate data set includes dermoscopic images of various hypopigmentation states.

[0071] Furthermore, the target state type to be identified is vitiligo.

[0072] As a preferred solution of this embodiment, the method for adjusting the first intermediate data set through the HSV color space conversion is:

[0073] To perform HSV conversion:

[0074]

[0075] Among them, H represents hue, S represents saturation, V represents brightness, R represents red of RGB value, G represents green of RGB value, B represents blue of RGB value, max(.) and min(.) are functions for finding maximum and minimum respectively;

[0076] Adjust saturation and hue;

[0077] Converting the HSV image back to RGB space based on the adjusted saturation and adjusted hue can be done in the following way:

[0078] R=V×I(H′)

[0079] G = V × I (H′ + 120) mod 360

[0080] B = V × I (H′ + 240) mod 360;

[0081] Here, I is a helper function that calculates the RGB components based on the hue value.

[0082] Furthermore, the method for adjusting saturation and hue is:

[0083] Adjust the saturation by the saturation adjustment factor α:

[0084] S′=α×S,0<α<1;

[0085] Among them, S′ is the adjusted saturation;

[0086] To adjust the hue:

[0087] If the hue of the pixel does not exceed the preset threshold, the hue of the red area of ​​the dermoscopic image is adjusted by the hue offset ΔH:

[0088] H′=H+ΔH;

[0089] ΔH=(H′ max -H′ min )×(H max -H min ) / (H max -H min );

[0090] Among them, H′ is the adjusted hue, H min and H max are the lower and upper limits of the hue range before adjustment, H′ min and H′ max Respectively, they are the lower and upper limits of the adjusted hue range.

[0091] In addition, the method for adjusting the second intermediate data set based on the CycleGAN model is:

[0092] Based on the second intermediate data set, performing modality conversion using the original modality image to generate a second modality image;

[0093] The adversarial loss function is used to ensure that the generated images match the transformed image distribution.

[0094] The cycle consistency loss function is used to avoid conflicts between the generator and the discriminator.

[0095] Specifically, the adversarial loss function for:

[0096]

[0097] in, The clinical images X of the second intermediate dataset clin Transformed to image Y in target domain wood The generator of D wood is the discriminator, x is X clin The data in , y is Y wood The data in It means to calculate the expected value of the probability distribution of sample y. Indicates the expected value calculation of the probability distribution of sample x;

[0098] The cycle consistency loss function for:

[0099]

[0100] in, To transform the image Y in the target domain wood Convert clinical images X into the second intermediate dataset clin The generator.

[0101] Finally, the method of adjusting the recognition model by the labeled target data set through the adaptive unfreezing method is preferably:

[0102] Freeze the parameters of layers other than the fully connected layer;

[0103] Set the gradient threshold γ and the performance improvement threshold τ;

[0104] Unfreeze each layer one by one and train it;

[0105] After each unfreezing, the performance of the recognition model on the validation set is obtained, and the absolute average of the gradients of the unfrozen layers is calculated to measure the activity of the layer on the task;

[0106] If the absolute average value of the gradient is greater than a preset threshold, the next layer is unfrozen; otherwise, the weight of the layer is kept unchanged and the unfreezing of subsequent layers is skipped.

[0107] The technical core of this embodiment is to apply progressive transfer learning and modality domain adaptation strategies to optimize the performance of deep learning models in the target field. The basic idea of ​​the present invention is: using a large number of unlabeled medical images in the field of skin conditions as a bridge, the unlabeled medical images use their data sources as labels, and reduce the differences between natural images and medical images through multi-step transfer learning. In addition, in view of the visual differences and distribution changes brought about by different modality data, the present invention proposes a modality domain adaptation method based on HSV color space conversion and CycleGAN. By adjusting the color space and feature distribution of the intermediate data, the intermediate data set is made closer to the target modality. Finally, for the precisely labeled target data set, the present invention applies an adaptive thawing strategy to optimize the model training process and reduce the risk of overfitting.

[0108] The following provides specific implementation cases and experimental results based on the technical solution of this embodiment:

[0109] The hardware environment and software environment used in this embodiment are shown in Table 1:

[0110] Table 1 Hardware environment and software environment

[0111]

[0112] This embodiment involves three data sets: a target data set and two intermediate data sets, all of which belong to the field of skin health status. The target data set is a multimodal vitiligo data set, which collects clinical images and Wood lamp images of 292 patients, including 311 pairs of stable vitiligo samples and 614 pairs of progressive vitiligo samples. Each skin lesion area has been accurately annotated, including the corresponding clinical image (TL) and the image taken under the Wood lamp (TW). The first intermediate data set (D1) integrates two public melanoma data sets (ISIC Challenges 2020 and HAM10000), containing a total of 44141 dermoscopic images. The second intermediate data set (D2) combines its own hypopigmentation state data set and the vitiligo public data set provided by 7 public skin state atlas websites, containing a total of 6341 hypopigmentation state images. Since the main goal of progressive transfer learning is to improve the learning efficiency of the model and promote stable weight convergence, there is no strict requirement for image labeling accuracy. For the two intermediate data sets, this embodiment adopts a strategy of randomly assigning labels, that is, using the name of the data set, that is, its data source, as the label of the image.

[0113] Then, the solution based on this embodiment performs training optimization of the model on the above data. Figure 2It is a progressive transfer learning process. The transferable knowledge is first transferred from ImageNet to the dataset in the field of skin states, and then transferred in two steps in the skin state dataset. Finally, it is transferred to the target multimodal vitiligo dataset. The present invention selects two intermediate skin state datasets to evaluate their effectiveness in the progressive transfer learning strategy. Among them, the D1 dataset contains dermoscopic images of melanoma. Thanks to the widespread attention of skin cancer in skin state research, the D1 dataset has a richer sample size, but its state categories are different from the target task. The D2 dataset focuses on a variety of hypopigmentation states including vitiligo, and has a high consistency in state categories with the target dataset, but the sample size is relatively small. The selection of these two datasets is intended to simulate the common category differences and data scarcity problems in real vitiligo state classification tasks.

[0114] Figure 3 The visual difference between dermoscopic images and clinical images and Wood's lamp images. Compared with the clinical images in the target dataset, dermoscopic images show more obvious red tones and higher saturation. Under the Wood's lamp, normal skin appears bluish purple under a specific wavelength, which contrasts sharply with the gray-white fluorescence of abnormal areas. This proves that in the classification of skin condition images, different imaging modes can lead to considerable visual differences between images.

[0115] Figure 4 This is a flow chart of the domain adaptation method based on HSV color space conversion. This technology converts the image from RGB color space to HSV color space and adjusts the hue, saturation, and brightness of the dermoscopic image to make it closer to the clinical modality and Wood's lamp modality images.

[0116] Figure 5 It is a modal domain adaptation process based on CycleGAN. In this embodiment, m samples are randomly selected from the source domain and the target domain to train the CycleGAN model, and a small part of the source domain images are used to synthesize the target domain modal images. The number of synthesized target images depends on the size of the target data set. In view of the fact that the sample size of the target data set is usually smaller than the source domain data set, this embodiment uses data enhancement to amplify the number of synthesized Wood's lamp images to expand the training set of the classification model. TW represents the Wood's lamp image of the target data set, D2L represents the clinical image of the D2 data set, D2W represents the corresponding wood's lamp image generated by CycleGAN, and D2W' represents the data set that has been processed with data enhancement and used for training. During the training process, CycleGAN learns the structural features and color space features of the two modal images respectively, and generates images of another modality corresponding to the input images of the original modality.

[0117] at last, Figure 6This is the final adaptive thawing method process.

[0118] This embodiment compares the effects of three training strategies - training from scratch, traditional transfer learning, and progressive transfer learning. The experimental results are shown in Tables 2 and 3. Progressive transfer learning has achieved improvements in key indicators such as accuracy (Acc), sensitivity (Sen), specificity (Spe), precision (Pre) and F1 score (F1-score). Compared with the traditional transfer learning method, the progressive strategy not only integrates the underlying common features of ImageNet, but also integrates the specific domain knowledge of the intermediate data set, thereby showing higher accuracy in the recognition and classification of target domain images. This result fully demonstrates the effectiveness of the progressive transfer learning strategy when dealing with limited target task data.

[0119] Table 2 Experimental results

[0120]

[0121] Table 3 Experimental results

[0122]

[0123] Figure 7 The two modality domain adaptation methods are visualized. (a) shows the dermoscopic image obtained after the conversion based on the HSV color space in this embodiment. The clinical modality domain adaptation process (marked as D1L) significantly changes the hue of the original dermoscopic image, making the color of the image closer to the true color of the skin in the clinical image. At the same time, the image processed by the Wood's lamp modality adaptation (marked as D1W) also shows good results. The normal skin area in the image presents a blue-purple tone consistent with the actual Wood's lamp irradiation effect, which can be used for scenarios where large data sets are converted quickly and in large quantities. (b) shows the Wood's lamp modality image in the D2 data set. The cycle-consistent loss restricts the basic structure of the clinical modality and the Wood's lamp modality, retaining the original part and position correspondence information. At the same time, the adversarial loss effectively simulates the skin characteristics under the illumination of the Wood's lamp, that is, the blue-purple result of normal skin and the fluorescence effect of the abnormal area with hypopigmentation.

[0124] Table 4 shows the experimental results of applying progressive transfer learning on the clinical modality dataset (TL) and the Wood lamp modality dataset (TW) after using the modality domain adaptation method on the D1 and D2 datasets. Among them, D1L refers to the simulation of clinical modality using the HSV domain adaptation method on the D1 dataset, D1W refers to the simulation of wood lamp modality using the HSV domain adaptation method on the D1 dataset, and D2W refers to the simulation of Wood lamp modality using the CycleGAN method on the D2 dataset. After using the modality domain adaptation method, the model showed significant performance improvement. On the clinical modality (TL) dataset, the accuracy was improved to 0.921 and the sensitivity reached 0.898; on the Wood lamp modality (TW) dataset, the accuracy reached 0.928 and the F1 score increased to 0.889. The experimental results show that using the modality domain adaptation technology on the intermediate dataset can make the model more easily adapt to different imaging modes, lighting conditions and imaging environments, and improve the recognition accuracy and generalization ability of the model.

[0125] Table 4 Experimental results of progressive transfer learning

[0126]

[0127] Table 5 shows the effectiveness of the adaptive parameter unfreezing strategy. In the clinical modality I→D1L→D2→TL experiment, the accuracy was improved to 92.4%, the sensitivity was greatly improved to 95.5%, and the F1 score was improved to 88.9%. In the Wood lamp modality I→D1W→D2W→TW experiment, the accuracy was improved to 93.5%, and the specificity and F1 score were also improved, proving that the overall performance of the model was optimized in this modality. The experimental results show that the adaptive parameter unfreezing strategy can dynamically adjust the parameters so that the model can better adapt to the changes in data and the needs of the task.

[0128] Table 5 Effectiveness of adaptive parameter unfreezing strategy

[0129]

[0130] The method of this embodiment is compared in detail with six mainstream backbone networks on clinical image datasets (TL) and Wood lamp image (TW) datasets. These six backbone networks include MobileNet V2, ResNet18, ShuffleNetV2, ViT, Swin Transformer and P2T. All six networks adopt the traditional transfer learning method, using the model pre-trained on ImageNet, and Table 6 lists the specific comparison results of each network in detail. On the TL dataset, the sensitivity of the method of this embodiment is as high as 95.5%, and the F1 score is 88.9%, which is significantly higher than other backbone networks. This advantage shows that the method of this embodiment has stronger performance in vitiligo state recognition and classification accuracy. On the TW dataset, the specificity of the method of this embodiment is 96.3%, and the F1 score also reaches 89.5%, which further confirms the efficiency and accuracy of the method of this embodiment in the vitiligo state recognition task.

[0131] Table 6 Comparison of the method of this embodiment with six mainstream backbone networks

[0132]

[0133]

[0134] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A skin state classification method based on progressive transfer learning and modality domain adaptation, characterized in that: The following steps are involved: Construct multiple intermediate datasets through unlabeled samples in different skin condition fields; The color space and feature distribution of multiple intermediate data sets are adjusted based on HSV color space conversion and CycleGAN model respectively; Based on modal domain adaptation, a progressive transfer learning strategy is adopted to gradually fine-tune the parameters of the recognition model through multiple intermediate data sets. For unlabeled skin images of different types and sources, their data sources are used as their labels in progressive transfer learning. The recognition model is adjusted on the labeled target dataset by an adaptive unfreezing method.

2. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 1, characterized in that: The number of the intermediate data sets is two; wherein the first intermediate data set includes dermoscopic images of melanoma, the second intermediate data set includes unlabeled samples from the field of skin images of the target state type to be identified, and the second intermediate data set includes dermoscopic images of multiple hypopigmentation states.

3. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 2 is characterized in that: The target category to be identified is vitiligo.

4. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 2, characterized in that: The first intermediate data set is adjusted by the HSV color space conversion, and the method is: To perform HSV conversion: Among them, H represents hue, S represents saturation, V represents brightness, R represents the red color of RGB value, G represents the green color of RGB value, B represents the blue color of RGB value, max(.) and mmin(.) are functions for finding maximum and minimum values ​​respectively; Adjust saturation and hue; Convert the HSV image back to RGB space based on the adjusted saturation and adjusted hue.

5. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 4 is characterized in that: The method for adjusting saturation and hue is: Adjust the saturation by the saturation adjustment factor α: S′=α×S,0<α<1; Among them, S′ is the adjusted saturation; If the hue of the pixel does not exceed the preset threshold, the hue of the red area of ​​the dermoscopic image is adjusted by the hue offset ΔH: H′=H+ΔH; ΔH=(H′ max -H′ min )×(H max -H min ) / (H max -H min ); Among them, H′ is the adjusted hue, H min and H max are the lower and upper limits of the hue range before adjustment, H′ min and H′ max Respectively, they are the lower and upper limits of the adjusted hue range.

6. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 2, characterized in that: The second intermediate data set is adjusted by the CycleGAN model, and the method is: Based on the second intermediate data set, performing modality conversion using the original modality image to generate a second modality image; The adversarial loss function is used to ensure that the generated images match the transformed image distribution. The cycle consistency loss function is used to avoid conflicts between the generator and the discriminator.

7. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 6, characterized in that: The adversarial loss function for: in, The clinical images X of the second intermediate dataset clin Transformed to the target domain image Y wood The generator of D wood is the discriminator, x is X clin The data in , y is Y wood The data in It means to calculate the expected value of the probability distribution of sample y. Indicates the expected value calculation of the probability distribution of sample x; The cycle consistency loss function for: in, To transform the image Y in the target domain wood Convert clinical images X into the second intermediate dataset clin The generator.

8. The skin state classification method based on progressive transfer learning and modal domain adaptation according to claim 1, characterized in that: The method for adjusting the recognition model by the target data set labeled by the adaptive unfreezing method is: Freeze the parameters of layers other than the fully connected layer; Set the gradient threshold γ and the performance improvement threshold τ; Unfreeze each layer one by one and train it; After each unfreezing, the performance of the recognition model on the validation set is obtained, and the absolute average of the gradients of the unfrozen layers is calculated to measure the activity of the layer on the task; If the absolute average value of the gradient is greater than a preset threshold, the next layer is unfrozen; otherwise, the weight of the layer is kept unchanged and the unfreezing of subsequent layers is skipped.

Images