A large-scale model-based acne grading model migration method

Abstract

The present invention discloses a large-scale model-based acne grading model migration method, which is applied to the intersection of computer science and medicine. Existing deep learning-based acne grading models are usually trained on data collected in a specific hospital. Therefore, when the model is migrated to a new hospital or other region, the offset of the data distribution usually leads to a decrease in effectiveness. The present invention first collects acne grading data from multiple different hospitals; then designs an acne grading model and selects one hospital data as source data for training and testing; secondly, the datasets of other hospitals are used as the target domain (i.e., the target hospital), and these data are all unlabeled. The large-scale model is adjusted on the target hospital dataset to make it suitable for the acne grading task; then, the acne grading model is adjusted on the target hospital dataset with the help of the output of the large-scale model and the designed loss function to complete the model migration process; finally, the model is tested.

Description

Technical Field

[0001] The present invention belongs to the intersection of computer and medicine, and particularly relates to an acne grading model migration technology. Background Art

[0002] Acne is one of the most common skin diseases. Aside from adolescents being susceptible to acne due to hormonal fluctuations during puberty, unhealthy lifestyles such as staying up late and consuming spicy and greasy foods can also contribute to facial acne. Consequently, there is a significant number of patients in urgent need of treatment. Dermatologists typically grade acne severity based on the number of acne lesions and the patient's overall facial condition, combining medical standards with their own experience. Common grading standards, such as the Hayashi scale, classify facial images into four levels: "mild," "moderate," "severe," and "very severe." Doctors use the number of lesions and their own experience to determine severity and design specific treatment plans. To reduce the burden on doctors, many studies have begun using deep learning models to assess acne severity as an aid to diagnosis.

[0003] However, due to the differences in shooting equipment used in different hospitals and for different patients, there are large differences between the data obtained and the data used to train the original model. Therefore, the trained model often performs poorly when transferred to other hospitals for use. Figure 1 This section shows datasets used for acne grading in different studies. Note that the angle, position, and style of the patient's face in datasets a, b, and c vary. Therefore, a model trained on one dataset often yields poorer accuracy when used directly on another dataset.

[0004] Existing acne grading model migration technologies include the following: 1. Based on the robustness of the trained model itself, it is directly used in other hospitals or regions without migration. 2. After migrating the model to another hospital, the data of the new hospital is collected again and labeled and trained. 3. Using the transfer learning method, the feature extractor part of the model is adjusted based on adversarial learning. By using both the original labeled data and the collected unlabeled new data, the positive model is adjusted to adapt it to the conditions of the new region and hospital. 4. Using the passive domain adaptation method, the original model is used to predict labels for the collected new data, and then the model is migrated using the new data and predicted labels.

[0005] Method 1 relies on the model's ability to adapt to data changes, and while this method has some effectiveness in natural imagery, it is not ideal for medical images and acne grading. Acne lesions are typically small, and variations in image style can severely impact grading. Therefore, directly applying a trained model to other hospitals is ineffective.

[0006] Method 2 involves re-collecting and labeling data at a new hospital, then training the model using this supplementary data based on the trained acne grading model. However, re-collecting and labeling data requires significant manpower and resources, and the amount of data collected may not be sufficient for training, which presents limitations.

[0007] Method 3 utilizes contrastive learning. By comparing the original data with the new data, the acne grading model is adjusted to adapt to the new hospital's data distribution, achieving better transfer results. However, in many cases in the medical field, the data used to train the model involves user privacy, so the new hospital cannot access this data. This method lacks practical application.

[0008] Method 4, based on passive domain adaptation, migrates the acne grading model, requiring only unlabeled data from the new hospital. However, the pseudo-labeling strategy introduces noise and fails to effectively distinguish the target domain (new hospital) data, resulting in poor migration results.

[0009] The shortcomings of the four aforementioned acne grading model transfer methods are as follows: Methods without transfer generally perform poorly in the acne grading field. Methods that collect and train data from scratch fail to leverage existing models, and data collection and labeling are labor-intensive and inefficient, lacking versatility. On the other hand, transfer learning methods either require access to data from the original hospital or are based on pseudo-labels, which introduces noise. These methods fail to effectively transfer models and also fail to adequately address the high inter-class similarity and large intra-class variability in acne grading, resulting in limited transfer effectiveness.

[0010] Acne image grading models can automatically complete acne diagnosis, assist doctors in diagnosis, and provide patients with a way to understand their condition. However, deep learning-based acne grading models are usually trained on data collected in a specific hospital. Therefore, when the model is migrated to a new hospital or other region, the shift in data distribution usually leads to a decrease in effectiveness, so measures need to be taken to ensure that the model can be reasonably migrated to other environments. In addition, existing methods either require re-collecting data for training in the target hospital during migration, which consumes a lot of manpower and material resources, or require obtaining the original training data during migration, which often involves patient privacy and ethics in the field of medical imaging. In addition, some migration methods that do not require original data ignore the characteristics of the acne grading task, and therefore the migration effect is poor.

[0011] In summary, the shortcomings of existing acne grading models and the technical problems to be solved by this method are summarized as follows:

[0012] 1) The acne grading model relies on the training data, and its performance deteriorates in new regions or hospitals due to the shift in data distribution.

[0013] 2) Some methods improve the robustness of the model by smoothing labels during model training to adapt to new data. However, this is not effective in the field of acne grading.

[0014] 3) Some methods collect and label data from new hospitals, perform fine-tuning and supplementary training to achieve model migration, which consumes a lot of manpower and material resources, and under some conditions, it is not possible to collect sufficient data for training.

[0015] 4) When acquiring new unlabeled data, existing migration methods also require the acquisition of labeled data from the original hospital. In the field of medical imaging, this usually violates patient privacy and ethics and lacks practical utility.

[0016] 5) Some migration methods use pseudo-labeling for new data to migrate the acne grading model, but the noise caused by pseudo-labeling will reduce the model effect.

[0017] 6) The model transfer ignores the characteristics of the acne grading task, the similarities between different acne severities, and the changes within the same severity. Summary of the Invention

[0018] In order to solve the above technical problems, the present invention proposes a large-model-based acne grading model migration method.

[0019] The technical solution adopted by the present invention is: a method for migrating an acne grading model based on a large model, wherein the large model includes a visual encoder and a text encoder; and further includes establishing an acne grading model, wherein the acne grading model includes a feature extractor and a classifier. The migration method specifically includes the following steps:

[0020] S1. The data with acne severity level labels is recorded as the source domain dataset, and the data without acne severity level labels is recorded as the target domain dataset; the initial acne grading model is the acne grading model that has been trained on the source domain dataset;

[0021] S2. Divide the target domain dataset into multiple batches, each batch including several target domain data;

[0022] S3. During the iteration of the current batch, the current target domain data in the current batch is input into the large model and the acne grading model respectively. The visual encoder in the large model outputs a visual vector based on the input target domain data; the text encoder in the large model outputs a text vector based on the input target domain data; the large model outputs the large model prediction result corresponding to the current target domain data based on the similarity between the visual vector and the text vector; the feature extractor in the acne grading model outputs a feature vector of the current target domain data, and the classifier in the acne grading model outputs a prediction result corresponding to the current target domain data based on the feature vector;

[0023] S4. The large model calculates a mutual trust loss function corresponding to the current target domain data based on the prediction result output by the large model and the prediction result output by the current acne grading model; and adjusts the text encoder in the large model based on the mutual trust loss function;

[0024] S5. The acne grading model calculates a diversity loss function based on the prediction result output by the current large model and the prediction result output by the current acne grading model;

[0025] Calculate the neighbor consistency loss based on the feature vector output by the feature extractor in the current acne grading model and the prediction result output by the current acne grading model;

[0026] Calculate the category incentive loss based on the prediction results output by the current acne grading model and the corresponding weights;

[0027] The total loss function corresponding to the current target domain data in the current batch is calculated based on the sum of the diversity loss function, the neighbor consistency loss, and the class incentive loss; and the current acne grading model is updated using the gradient descent method based on the total loss function;

[0028] S6. Calculate the classification accuracy of the acne grading model corresponding to each target domain data during the iteration of the current batch, retain the acne grading model with the highest classification accuracy, and if all batches of iterative training have been completed, use the acne grading model with the highest current classification accuracy as the migrated acne grading model; otherwise, return to step S3.

[0029] Beneficial effects of the present invention: Therefore, this method is based on the research on domain adaptation in deep learning, and proposes a large-model-based acne grading model migration method. This method can solve the problem of performance degradation when the acne grading model is migrated to a new hospital. At the same time, the proposed method does not need to obtain the data of the original hospital during migration training, and also avoids ethical and privacy issues when obtaining the original data. In the new hospital, only an appropriate amount of unlabeled data needs to be collected to complete the migration, and a good effect is achieved, which greatly reduces the data difficulty in the migration process of the acne grading model. At the same time, based on the characteristics of the acne grading model, the proposed neighbor prediction consistency loss and inter-class discrimination loss also improve the diagnostic performance of the migration model for acne. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Acne images from different datasets;

[0031] Figure 2 for Figure 2 CLIP overall structure;

[0032] Figure 3 is a flow chart of the method of the present invention;

[0033] Figure 4 This is the structural diagram of the acne classification model;

[0034] Figure 5 This is the overall structure diagram of the model migration method;

[0035] Figure 6 Schematic diagram of consistency loss for neighbor prediction;

[0036] Figure 7 Schematic diagram of category incentive loss;

[0037] Figure 8 This is the structure diagram of the CLIP submodule of the large model;

[0038] Figure 9 Focus visualization for acne characteristics. DETAILED DESCRIPTION

[0039] To facilitate those skilled in the art to understand the technical content of the present invention, the following technical terms are first explained:

[0040] 1. Deep Neural Network (DNN)

[0041] Deep neural networks are a type of feedforward neural network. Convolutional neural network (CNN) models are often used in computer vision. For acne severity grading, the convolutional neural network typically extracts image features using its convolution kernels. Then, a classifier layer is trained based on these features. By training with a certain amount of data, the parameters of the feature extraction and classification layers are continuously adjusted to achieve the desired grading results.

[0042] 2. Large Model

[0043] The large model is usually a deep neural network model with hundreds of millions to tens of billions of parameters. It is trained in an unsupervised and self-supervised manner using a large amount of unlabeled data to learn rich representations from the massive amount of data. It has stronger versatility and is often able to perform well on multiple tasks (such as image recognition, natural language processing, etc.), and can classify images without any labels without any training. The method of the present invention is to utilize the general classification ability of the CLIP (Contrastive Language-Image Pre-training, a pre-training model based on contrasting text-image pairs) large model to achieve the migration of the acne grading model on the unlabeled target dataset, so that the acne grading model can also be accurately graded on the target dataset. The structure of the CLIP model is as follows: Figure 2 shown.

[0044] 3. Domain Adaptation

[0045] Domain adaptation is a key research area in transfer learning. Its goal is to transfer a model trained on one dataset to another while maintaining its original performance. This is because models trained on a specific dataset often perform poorly when deployed or transferred to similar tasks due to changes in data distribution. Collecting and annotating data in the target domain often requires additional resources, and in some cases, it may not be possible to collect sufficient data to retrain the model. Transfer learning aims to leverage a trained model to achieve better results with a small amount of training on the target domain dataset.

[0046] The proposed method first uses a trained small model to adjust the large model, adapting it to the acne grading domain. The large model's zero-shot grading capabilities are then leveraged to perform grading on unlabeled data collected at the target hospital, serving as a guide for model transfer. Furthermore, a designed neighbor prediction consistency loss and inter-class discrimination loss are employed to further improve the model's ability to distinguish between different acne severity levels. Ultimately, the trained acne grading model can be transferred to a new hospital without requiring extensive labeling of new data, while maintaining improved acne grading capabilities.

[0047] This invention proposes a complete set of acne grading model migration methods, which can be divided into four parts: (1) Collecting acne grading data from multiple different hospitals; (2) Designing an acne grading model and selecting a hospital data as the source data for training and testing; (3) Using the data sets of other hospitals as the target domain (i.e., the target hospital), and these data are all unlabeled, adjust the large model on the target hospital data set to make it suitable for the acne grading task. (4) Using the output of the large model and the designed loss function on the target hospital data set, adjust the acne grading model to complete the model migration process. (5) Model testing. The overall flow chart is as follows: Figure 3 As shown:

[0048] (1) Prepare acne image data from multiple hospitals: Collect acne image data from multiple hospitals or using different shooting methods, and use each category as an acne grading dataset. Professional dermatologists then annotate the acne image data, and then divide these datasets into source and target domains. The source domain is used to train the acne grading model, simulating existing research. The target domain represents the data from a new hospital when the model is used.

[0049] 11. Collect datasets from different hospitals or different styles

[0050] There are three datasets used in this embodiment: the first dataset a is the public acne severity grading dataset ACNE04, which has a picture style with varying angles. The shooting equipment includes mobile phones and cameras, and focuses on shooting the acne locations on the patient's face. It contains a total of 1,457 patient pictures. The second dataset b is professional diagnostic pictures taken by VISIA equipment collected in cooperation with dermatologists. The picture contains the complete part of the patient's face and retains the high-definition details of the face. The dataset is named HXSKIN04 and has a total of 1,540 pictures. The third dataset c is the data uploaded by patients collected through the mini program. The patients took pictures and uploaded them with their own mobile phones. In addition to the patient's facial area, there are also environmental background and clothing background interference in the pictures. It includes a total of 656 pictures, and the dataset is named WXSKIN04. Some examples of these three datasets can be found in Figure 1Finally, with the help of dermatologists, the data in each dataset were classified into four different levels according to the severity of acne; the severity level served as the label for the corresponding image.

[0051] 12. Data Processing

[0052] Data processing is a prerequisite for model training and model migration. The acne images used in this embodiment have three dimensions: length, width, and channel. During training and migration, they are uniformly scaled to 520×520×3 using bilinear interpolation, and then cropped to 512×512×3 by center. Then, channel normalization, random flipping, random rotation and other enhancements are performed. The acne severity corresponding to the image, that is, the label part, is first encoded into a corresponding number and then processed into a one-hot form, as shown in Table 1. It should be noted that labels are only used when the dataset is used as the source domain, and there are no labels when it is used as the target domain.

[0053] Table 1 Coding results of acne severity

[0054] Severity level Corresponding code One-hot Tag Mild 0 [1,0,0,0] Moderate 1 [0,1,0,0] serious 2 [0,0,1,0] Urgent need for medical treatment 3 [0,0,0,1]

[0055] 13. Divide the source domain and target domain datasets

[0056] The model migration task is different from the usual deep learning process. It is necessary to evaluate the effect of migrating a model trained on a certain dataset to other datasets for similar tasks. Therefore, the method of the present invention uses three acne grading datasets with different data distributions: ACNE04, HXSKIN04, and WXSKIN04, which are divided into source domain and target domain datasets respectively. The source domain dataset refers to the acne grading model trained on this dataset using labeled data, representing a general grading method. The unlabeled target domain dataset is used for model migration and evaluation, representing the situation when the model is migrated to other hospitals. In order to obtain more complete model migration results, the method of the present invention selects ACNE04, HXSKIN04, and WXSKIN04 as source domain datasets in turn, and uses the remaining two datasets as target domain data each time for model migration and evaluation.

[0057] (2) Design and train an acne grading model: Based on existing research, an acne grading model is designed, consisting of a feature extractor and a classifier. It is trained on source domain (source hospital) data to achieve acne grading results. This model is used as a trained model.

[0058] This section details the design of the acne grading model and the training process on the source domain dataset, which is used as the model to be migrated. According to existing research, the acne grading model usually consists of two parts: a feature extractor and a classifier. The method of the present invention uses the convolutional neural network ResNet50 as the feature extractor, and then processes the feature matrix into a feature vector through global average pooling. Finally, a multi-layer fully connected layer is used as a classifier to obtain the prediction results for each level. The one with the highest probability is the predicted level. The detailed structure and parameters of the model are shown in Figure 2. Figure 4 shown.

[0059] Then the model is trained on the selected source domain acne dataset, which is represented as: The subscript s indicates that the source domain dataset (with labels) is used. represents the input acne image, Indicates the corresponding one-hot label, i.e., the severity level of acne, k indicates that there are k levels, n s Represents the total number of samples in the dataset. The acne grading task is actually classifying acne images, so cross-entropy loss is used as the target loss function. At the same time, due to the similarity between acne images of different severity levels, to prevent the model's overconfidence from affecting generalization, the labels are smoothed based on the cross-entropy loss. The label smoothing formula is as follows:

[0060]

[0061] Note that here and They are all in vector form, where K is a vector of all 1s and its dimension is k. α is a factor that controls the smoothness and its range is between [0,1].

[0062] Acne ImagesX i After the feature extractor and the average pooling of the whole set, the feature vector R is obtained. i As shown below:

[0063]

[0064] ResNet() represents the feature extraction process, R i Image X for Acne i The corresponding eigenvectors, is in component form, and m is the dimension of the vector. Then, after passing through the classifier of multiple layers of fully connected layers, the logit output H is obtained. i After the logit output passes through the softmaxt function, the predicted probability P of the image for each severity level is obtained. i , as shown in the following formula:

[0065]

[0066] Among them, fc represents the processing of the fully connected layer, softmax represents the corresponding function, and K represents the total number of levels.

[0067] Therefore, the prediction result P i With the smoothed label The loss calculation formula is as follows. By optimizing the loss function to reach the minimum value, the model can achieve the best acne grading effect on the source domain dataset.

[0068]

[0069] (3) Adjusting the large model for acne diagnosis: By using the mutual information loss function between the prediction results of the trained acne grading model and the prediction results of the large model, the large model is adjusted to have better prediction results in the acne grading task.

[0070] 31. Overall framework for migrating acne grading models

[0071] The overall structure of the migration process is as follows Figure 5 As shown in the figure, the purpose is to make the model trained on the source domain dataset achieve good classification results on the unlabeled target domain dataset, while not using the source domain dataset in the training process. To represent the target domain dataset, where the subscript t represents the target domain, n t represents the number of samples in the target domain dataset, Represents the corresponding image, noting that only the unlabeled D is used in the migration process t This is also the two major advantages of the method proposed in this invention compared to other methods: no re-labeling is required, and no source data acquisition is required.

[0072] First, through the ability of large models, D t The prediction of acne image grade is performed by using the mutual information loss function L with the prediction output of the trained acne grading model. mi , adjust the grading ability of the large model to adapt it to the acne grading task. Then, in the process of migrating the acne grading model, the target domain data D t To adjust the acne grading model, the prediction results of the large model are used as the supervision of the prediction results of the acne grading model, and the diversity loss function L is used to adjust the acne grading model. divIncrease the diversity of prediction results to avoid collapse during the migration process. At the same time, find the neighbors of each sample through the feature vector output of the acne grading model, and use the neighbor consistency loss L nc To constrain the accuracy of model prediction. Then, based on the model prediction results, the designed class encouragement loss L ce , improving the ability to distinguish similar acne grades. By constraining the three loss functions, the acne grading model can be transferred without labels.

[0073] In order to explain the specific workflow of the proposed model in more detail, its pseudo code is as follows:

[0074]

[0075] 32Adapting the large model to the acne classification task

[0076] In the proposed acne model migration method, the large model corrects the erroneous output of the acne grading model through its grading results, and at the same time avoids the prediction results of the acne grading model from all being predicted to the same category. Because there are labels to complete these tasks during training in the source domain, there are no labels in the target domain. However, the large model is trained by a large amount of image and text data through comparative learning, which provides image and text matching capabilities. It is a general model and has a relatively poor effect when used directly on the acne grading task. The method proposed in the present invention is not to retrain the large model, but to fine-tune the learnable parameters of the text feature extractor in the text encoder of the large model through the mutual information loss based on the output of the acne grading model, so that the text encoder pays more attention to features related to acne grade when converting text into vector expression, thereby improving the prediction effect of the general large model on the acne grading task.

[0077] CLIP acne grading process Figure 4 As shown, a corresponding text is constructed for each acne severity level. The large model's text encoder then extracts the feature vector corresponding to each level. Next, the large model's image encoder generates the corresponding feature vector for the input acne image. The distance between the image vector and the text vector corresponding to each level is calculated, and the distance is processed using the Softmax function to calculate the predicted probability for each severity level. The level with the highest probability is the predicted severity level. This is shown in the following formula:

[0078]

[0079] Where T is the extracted text feature matrix, each column corresponds to each level, q i is the feature vector of the input image, The vector represents the distance between the image vector and each text vector. The vector represents the predicted probability of each level after the distance vector is processed by the Softmaxt function. Where i represents the sample number, K represents the total number of acne levels, k represents the kth level, C is the dimension of the feature vector, and l represents the output data in the large model.

[0080] Then the prediction output of the acne grading model is used as the reference target of the prediction output of the large model, and the mutual information loss function L mi Optimize the learnable parameters of the self-attention module and the fully connected module in the text feature extractor of the large model, so that the large model of the general field can perform better in the acne grading task. mi The loss function is defined as follows. l represents the prediction output of the large model, P t Represents the predicted output of the acne grading model. P represents the joint distribution of the two predicted outputs, which is a two-dimensional matrix with dimensions K×K, where K is the total number of grades, and n t represents the total number of samples, and T represents the transpose of the vector.

[0081]

[0082] 33 Predicting Diversity Loss Design in Migration Tasks

[0083] In the target domain When migrating the acne grading model, since there is no constraint on the label Y, the predicted result P t It will change in the direction of smaller entropy, that is, it will approach the same label. Normal acne grade labels should be in the form of [1,0,0,0], [0,1,0,10]. The purpose of migration is to make the acne grading model t The prediction result P t If the entropy is close to this form, it means that the predicted acne grade is correct and clear. However, if the entropy is small, the acne grade model may predict the form of [1,0,0,0] on all images, which will lose the grade ability. The traditional method is to take the average of the predicted results of all samples. and This fixed vector calculates the cross entropy loss to constrain the collapse of the prediction results. However, there are differences between each acne image, and the number of acne images of each grade is unbalanced, so it is not reasonable to let it approach K. In the method of the present invention, let the prediction P of the acne grading model be t Compared with the prediction of the large model p lBecause the adjustment of the large model and the migration of the acne grading model are performed alternately, the large model is not involved in training, so there is no performance collapse. Furthermore, the prediction results for a single image are more informative, and using this as a diversity constraint achieves better results. The specific loss function is shown below, where K represents the total number of grades. and It is the weight of the acne grading model and the prediction results of the large model at each level.

[0084]

[0085] 34 Neighbor Prediction Consistency Loss Design

[0086] Through the diversity loss constraint, the prediction results of the acne grading model will not explore the same level during the migration process, which ensures the diversity of the prediction results. At the same time, it is close to the prediction results of the large model, which also ensures the correctness of the acne grading model. However, our goal is to make the acne grading model more accurate in the target domain dataset. The prediction effect is better, but at the same time, no labels and source domain datasets are used. Through experiments, it can be noticed that the feature vectors extracted by the feature extractor in the acne grading model have a certain aggregation phenomenon after two-dimensional visualization. That is to say, although the acne grading model has a poor effect on the target domain, due to the synergy of distribution shift, the output feature vectors are still separable. Therefore, the method of the present invention uses this feature to propose a neighbor prediction consistency loss, the principle of which is as follows Figure 6 shown.

[0087] The feature vectors close to each other are called neighbors, so the corresponding prediction results should also be similar. By finding the prediction result P t The prediction results of the m nearest neighbors corresponding to i are brought closer to each other in terms of distance. This makes the prediction results of the neighbors closer to the correct direction, thereby improving the accuracy of the prediction. At the same time, it also improves the model's ability to distinguish the similarities between acne images of different severity. The detailed calculation process of the loss function is as follows. First, calculate X i The corresponding eigenvector H i ∈R 1×c With feature vector memory The distance between each vector in Where c represents the dimension of the feature vector, n t Indicates the total number of samples.

[0088]

[0089] Then from D i Select m with the smallest distance as Hi In practical applications, the value of m is required to cover valid neighbors; in this embodiment, the value of m is the total number of levels minus 1, H i The set of neighbors of i According to the previous theory, H i The corresponding prediction result P i The prediction results corresponding to its neighbors should be consistent, so the calculation formula of the neighbor prediction consistency function is as follows, where ||S i || represents the set S i The number of elements in, k represents the level, p ik and p jk They are the model prediction results P i and neighbor prediction results P j The form of the quantity.

[0090]

[0091] 35 categories of incentive loss design

[0092] Since there are similarities between images of different severity in acne diagnosis, and there are different degrees of variation between acne images of the same severity, a class-incentive loss function is proposed to constrain the prediction results, making it more discriminable between similar classes and between different classes, thereby improving the model's ability to distinguish different grades of acne. The principle is as follows Figure 7 shown.

[0093] The prediction results of the acne grading model are expressed as P i =[p i1 ... p i2 ... p ik ], according to the previous definition, its component p ik It is divided into two parts: the part above the threshold of 0.75 indicates that it is closer to the true level, which is called the positive component, and the part below it indicates that it is not related to the level, which is called the negative component. Category incentives are based on the weight of the positive component more and the weight of the negative component less to adjust the prediction results. At the same time, the change of this weight just coincides with the prediction result, so the prediction result component is used to generate the adaptive weight, avoiding the complexity brought by manual parameters. As shown below, Z + Indicates P i The set of subscripts of the positive components in the equation is used to construct the corresponding weight a through the positive and negative components. i and b i , then calculate the category incentive loss L ce .

[0094] P i =[p i1 ... pi2 ... p ik ]

[0095]

[0096] The final loss function of the acne grading model migration is L = L div +L nc +L ce .

[0097] (4) Acne Grading Model Migration: The main purpose of this step is to migrate the trained acne grading model to the target domain (other hospitals) data so that it still maintains good performance. This mainly includes the mutual information loss between the large model and the acne grading model, and achieves prediction diversity through mutual correction. Then, the neighbor consistency loss and class incentive loss of the acne model output results are used to improve the distinguishability of different acne.

[0098] This part is the core protection point of this patent, which mainly includes the detailed process of the proposed acne grading model migration method and the improvements compared with previous methods.

[0099] This section details the process of transferring the acne grading model to the target domain dataset. The specific implementation includes data encoding, model parameter setting, backpropagation during training, and parameter optimization.

[0100] 41. Data Encoding

[0101] First, the input data for the large model is divided into two parts: text data and image data. The image data consists of facial acne images of patients. Each image is processed to 512×512 pixels as described in Step 22. Then, similar to the input of the acne grading images, each image is enhanced through random cropping, horizontal flipping, and random rotation to create a different image from the original. Normalization is also performed in the channel dimension to accelerate model convergence. For the text data, the doctor's annotated severity level is mapped to "This is an image of facial acne [ ]", with the [ ] filled in with the text of the four levels: "mild", "moderate", "severe", and "very severe". The input of the acne grading images is only the image portion, and the same processing is applied as the image input of the large model.

[0102] 42. Detailed construction of large model and acne grading model

[0103] The acne grading model consists of a feature extractor and a classifier. The feature extractor uses ResNet50, which also achieved the best results in the experiment. The classifier uses three fully connected layers. The construction of the large model CLIP is divided into two parts. The visual encoder part uses the commonly used VIT-B / 32 model. The word embedding part in text processing uses the Byte-pairEmbedding structure. The text encoder part uses the mature transformer / base model in the field of natural language processing. The specific structure is as follows: Figure 8 As shown in FIG, the parameters of the self-attention module and the fully connected module in the text encoder are learnable. The structure of the large model CLIP is well known in the art and will not be described in detail in this invention.

[0104] 43. Loss Function, Backpropagation, and Parameter Optimization

[0105] The loss function is the goal of model optimization. By calculating the gradient of the loss function and backpropagating to optimize the model parameters, we can achieve our ideal effect. Specifically, the gradient is calculated by deriving the loss function, and then the learning rate is used in the gradient direction to update the parameters according to a certain ratio. The model is adjusted to achieve the optimal parameters. The loss function design of the method of the present invention has been described in detail above. Compared with the learning rate part, since the method of the present invention is a migration process of a trained model, a differentiated learning rate setting is adopted. For the feature extractor part of the acne grading model, the learning rate is set to 0.0005, and for the classifier part, it is set to 0.0001. The stochastic gradient optimizer (SGD) is used to select the direction of gradient optimization. For the fine-tuning part of the large model, the learning rate is set to 0.00001 because it already has a good effect. The total number of training rounds is set to 30 rounds, and the amount of data for each batch in each round is set to 64. After each round of training, the model effect is tested according to the evaluation criteria below, and the model with the best classification accuracy is selected as the migrated acne grading model. Because in the migration task, a small number of rounds can achieve good migration results, early stopping is used to avoid excessive migration.

[0106] (5) Evaluation of the migrated model: Select the performance criteria for the acne grading field, and then test and evaluate the performance of the unmigrated acne grading model and the migrated model on multiple target domain (target hospital) datasets.

[0107] The main purpose of the test and evaluation phase is to test the effect of the migrated acne grading model on the target domain acne dataset and evaluate whether it has achieved usable performance. In the method of the present invention, the accuracy (Accuracy), precision (Precision), sensitivity (Sensitivity), specificity (Specificity) and Youden index (YI) commonly used in medical image classification are mainly used to measure the grading effect of the migrated model. The value of Youden index (YI) is (sensitivity + specificity - 1), which can comprehensively measure the effectiveness of model diagnosis by combining sensitivity and specificity. The higher the better for all these indicators. In order to test the completeness of the evaluation, three datasets ACNE04, HXSKIN (abbreviated as HX), and WXSKIN (abbreviated as WX) are used as target datasets, and the remaining two are used as target datasets. "ACNE04->HXSKIN" represents the migration from the source dataset to the target dataset. The specific results are shown in Table 2.

[0108] Table 2 Classification effect of the model after migration

[0109] Dataset metrics Accuracy Accuracy Specificity Sensitivity Youden Index ACE04->WX 0.7099 0.7899 0.8905 0.7563 0.6468 ACE04->HX 0.7561 0.7860 0.9205 0.7661 0.6866 WX->ACNE04 0.8030 0.6809 0.9562 0.8001 0.7564 WX->HX 0.7504 0.6790 0.9428 0.7489 0.6817 HX->ACNE04 0.8490 0.8021 0.9635 0.8188 0.7823 HX->WX 0.6970 0.7900 0.9553 0.8400 0.7953

[0110] To compare the performance of the proposed method, we compared it with other model transfer methods, a non-transfer method, and a labeled retrained model (i.e., retraining the model on the target domain dataset). The results are shown in Table 3. The classification accuracy is mainly used as the evaluation criterion.

[0111] Table 3 Comparison results of the method of the present invention and other methods

[0112] Method Name ACE04->WX ACE04->HX WX->ACNE04 WX->HX HX->ACNE04 HX->WX No migration 0.5343 0.6123 0.4032 0.4122 0.6799 0.5033 Heavy training 0.7132 0.7601 0.8632 0.7601 0.8632 0.7132 DANN 0.6532 0.6943 0.7033 0.6876 0.7067 0.6003 DAN 0.6648 0.6834 0.7189 0.6990 0.7321 0.6342 SHOT 0.6790 0.6855 0.7221 0.6700 0.7201 0.6620 NRC 0.6795 0.7232 0.7399 0.7032 0.8090 0.6540 GKD 0.6832 0.7434 07867 0.7243 0.8390 0.6800 Method of the present invention 0.7099 0.7561 0.8030 0.7504 0.8490 0.6970

[0113] The method of the present invention in Table 3 shows the best migration results. It can be noted that without migration, that is, when the model trained in the source domain is directly used in the target domain, the effect is very poor and cannot achieve usable performance. However, after the migration method, the performance of the model is greatly improved. Among them, the DANN and DAN migration methods still need to obtain source domain data during migration, but the performance is even worse than SHOT, NRC, GKD and the proposed source-free domain method. Therefore, the source domain dataset is not necessary for migration. At the same time, we noticed that the performance of the WX dataset during migration and retraining is very close. Because the scale of the WX dataset is small, in the case of insufficient data, using an existing model for migration is actually a better method. Finally, the large-model-based acne grading model migration method proposed in this paper achieved the best results. On the one hand, it used the prediction results of the large model and the source model to guide each other. On the other hand, the designed diversity loss function, neighbor prediction consistency loss function and category incentive loss function all play a very important role and constrain the model results.

[0114] In addition, in order to demonstrate the proposed migration method, e.g. Figure 9 As shown in the figure, the feature map of the acne grading model is converted into a heat map using the CAM method. The heat map of the retraining method is compared with the heat map of the non-transferred method and the heat map of the proposed migration method to observe the degree of attention to acne lesions. By comparing the visualization of acne feature attention, it can be found that the acne grading model used directly without migration still retains the ability to pay attention to acne features, which also illustrates the feasibility of using existing models. However, there is a phenomenon of missing or misattention to features. This is caused by the data offset between the source domain and the target domain, which is also the reason for the decline in target and classification. Comparing the attention map after migration with the attention map before migration, it can be found that after migration training, the model can better pay attention to the rubbing acne features, which also proves the effectiveness of the proposed method.

[0115] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A large-scale model-based acne grading model migration method, characterized in that: The large model includes a visual encoder and a text encoder; the method also includes establishing an acne grading model, which includes a feature extractor and a classifier; the migration method specifically includes the following steps: S1. The data with acne severity level labels is recorded as the source domain dataset, and the data without acne severity level labels is recorded as the target domain dataset; the initial acne grading model is the acne grading model that has been trained on the source domain dataset; S2. Divide the target domain dataset into multiple batches, each batch including several target domain data; S3. During the current iteration, the target domain data of the current batch in the current iteration is input into the large model and the acne grading model respectively. The visual encoder in the large model outputs visual features based on the input target domain data of the current batch; the text encoder in the large model outputs text features based on the input target domain data of the current batch; the large model outputs the large model prediction results corresponding to the target domain data of the current batch based on the similarity between the visual features and the text features; the feature extractor in the acne grading model outputs the feature vector of the target domain data of the current batch, and the classifier in the acne grading model outputs the prediction results corresponding to the target domain data of the current batch based on the feature vector; S4. The large model calculates a mutual trust loss function corresponding to the target domain data of the current batch based on the prediction results output by the large model and the prediction results output by the current acne grading model; and adjusts the text encoder in the large model based on the mutual trust loss function; S5. The acne grading model calculates a diversity loss function based on the prediction result output by the current large model and the prediction result output by the current acne grading model; Calculate the neighbor consistency loss based on the feature vector output by the feature extractor in the current acne grading model and the prediction result output by the current acne grading model; Calculate the category incentive loss based on the prediction results output by the current acne grading model and the corresponding weights; The total loss function corresponding to the target domain data of the current batch is calculated based on the sum of the diversity loss function, the neighbor consistency loss, and the category incentive loss. The current acne grading model is updated using the gradient descent method based on the total loss function. S6. Calculate the classification accuracy of the acne grading model corresponding to each batch of target domain data during the current iteration, and retain the acne grading model with the highest classification accuracy. If all batches of iterative training have been completed, the acne grading model with the highest current classification accuracy is used as the migrated acne grading model; otherwise, return to step S3.

2. The method for migrating an acne grading model based on a large model according to claim 1, characterized in that: The feature extractor of the acne grading model is as follows: the convolutional neural network ResNet50 is used to extract the feature matrix of the input data, and then the feature matrix is ​​processed into a feature vector through global average pooling.

3. The method for migrating an acne grading model based on a large model according to claim 2, characterized in that: The acne grading model uses multiple fully connected layers as classifiers.

4. The method for migrating an acne grading model based on a large model according to claim 3, characterized in that: The large model specifically uses the CLIP model, and its text encoder uses the transformer / base model. The text encoder specifically includes: a text generation unit, a word embedding unit, and a text feature extraction unit. The text generation unit generates text descriptions of acne of different severities based on input data. The word embedding unit performs word segmentation and mapping processing on the text representation output by the text generation unit to obtain a text vector. The text feature extraction unit extracts text features based on the text vector output by the word embedding unit.

5. The method for migrating an acne grading model based on a large model according to claim 4, characterized in that: The visual encoder of the large model uses the VIT-B / 32 model to extract image features.

6. The method for migrating an acne grading model based on a large model according to claim 5, characterized in that: The mutual information loss function is denoted as L mi , the expression is: Among them, p ij The elements in the matrix P representing the joint distribution of the prediction results of the large model and the prediction results of the acne grading model, The superscript T indicates transposition, P l Represents the prediction result of the large model, P t represents the prediction result of the acne grading model, Represents the i-th element in the prediction result of the large model, represents the jth element in the prediction result of the acne grading model, c represents the dimension of the feature vector, n t represents the total number of samples in P.

7. The method for migrating an acne grading model based on a large model according to claim 6, characterized in that: The diversity loss function is denoted as L div , L div The calculation expression is: Where K represents the total number of acne severity levels, represents the component of the prediction result of the acne grading model at the kth level, Represents the component of the prediction result of the large model at the kth level.

8. The method for migrating an acne grading model based on a large model according to claim 7, characterized in that: The feature vector output by the feature extractor in the acne grading model corresponding to the current batch of target domain data is recorded as H i , H i The prediction result of the corresponding acne grading model is recorded as P i , set the feature vector memory to store the feature vector output by the feature extractor in the acne grading model corresponding to each batch of target domain data, and calculate H i The distance between it and the rest of the feature vectors in the feature vector memory; Sort the distances by smallest and largest, and select the eigenvectors corresponding to the first several distances as H i The neighbors of i ; Neighbor consistency loss is denoted as L nc , L nc The calculation expression is: Among them, p ik Indicates P i The kth component of jk Indicates H i The kth component of the prediction result of the acne grading model corresponding to the jth neighbor of ||S i || represents the set S i The number of elements in .

9. The method for migrating an acne grading model based on a large model according to claim 8, characterized in that: Set the threshold value and set P i The component in is greater than the threshold value and is recorded as a positive component, and the component less than or equal to the threshold value is recorded as a negative component; the category excitation loss is recorded as L ce , L ce The calculation formula is: Among them, Z + Indicates P i The set of subscripts of the positive components in a i According to P i The weight of the positive component, b i According to P i The weight of the negative component construction.

Images