Acne grading model migration method based on large model

Abstract

The invention discloses an acne grading model migration method based on a large model, which is applied to the crossing field of computers and medicine and aims at solving the problem that an existing acne grading model based on deep learning is usually trained on data collected by a specific hospital. Therefore, when the model is migrated to a new hospital or other regions, the effect is generally reduced due to the offset of data distribution; the method comprises the following steps: firstly, collecting acne grading data from a plurality of different hospitals; then designing an acne grading model, and selecting hospital data as source data for training and testing; secondly, data sets of other hospitals serve as a target domain (namely a target hospital), meanwhile, the data are not labeled, and a large model is adjusted on the target hospital data set so that the target hospital data set can adapt to an acne grading task; adjusting the acne grading model on a target hospital data set by means of the output of the large model and the designed loss function to complete a model migration process; and finally carrying out model testing.

Description

Technical Field

[0001] The invention belongs to the intersection field of computer and medicine, and particularly relates to an acne grading model migration technology. Background Art

[0002] Acne is one of the most common skin diseases. In addition to adolescents being prone to acne due to changes in hormone levels during puberty, the unhealthy lifestyles of modern young people, such as staying up late and eating spicy and greasy foods, can also lead to facial acne. Therefore, there are a large number of patients who are in urgent need of treatment. Usually, dermatologists will grade the severity of acne based on the number of acne lesions and the overall facial condition of the patient, combined with medical standards and their own experience. Commonly used grading standards such as the Hayashi grading method divide the patient's facial images into four levels, including: "mild", "moderate", "severe" and "very severe". Doctors judge the severity based on the number of lesions on the patient's face and their own experience, and then design specific treatment plans for the patient. In order to reduce the burden on doctors, many studies have begun to use deep learning models to judge the severity of patients' acne as an aid to doctors' diagnosis.

[0003] However, due to the differences in shooting equipment in different hospitals and for different patients, there are large differences between the data obtained and the data used to train the original model. Therefore, the trained model often does not perform well when it is migrated to other hospitals for use. Figure 1 The datasets used for acne grading in different studies are shown. It can be noticed that the angle, position, and style of the patient's face in the three different datasets a, b, and c are different. Therefore, the accuracy of the model trained on one dataset is usually poor when directly diagnosing on another dataset.

[0004] Existing acne grading model migration technologies include the following: 1. Based on the robustness of the trained model itself, it is used directly in other hospitals or regions without migration. 2. After migrating the model to other hospitals, re-collect the data of the new hospital and label and train it. 3. Adopt the transfer learning method, adjust the feature extractor part of the model based on adversarial learning, and adjust the positive model to adapt to the conditions of the new region and hospital by using the original labeled data and the collected unlabeled new data at the same time. 4. Adopt the passive domain adaptation method, use the original model to predict labels for the collected new data, and then migrate the model through the new data and predicted labels.

[0005] Method 1 relies on the model's ability to adapt to data changes. This method has a certain effect in the field of natural images. However, for medical images and acne grading, acne lesions are relatively small, and changes in different image styles will seriously affect the grading effect. Therefore, directly using the trained model in other hospitals is not effective.

[0006] Method 2 is to re-collect and annotate data in a new hospital, and then train the model based on the trained acne grading model using this supplementary data. However, re-collecting and annotating data requires a lot of manpower and resources, and the amount of data collected may not be enough for training, which has limitations.

[0007] Method 3 considers the idea of ​​using contrastive learning. By comparing the original data with the new data, the acne grading model is adjusted to adapt to the new hospital data distribution and achieve better migration effect. However, in many cases, in the medical field, the data used to train the model involves user privacy, so the new hospital cannot obtain this data. This method lacks practical practicability.

[0008] Method 4 migrates the acne grading model based on passive domain adaptation, which only requires obtaining unlabeled data from the new hospital. However, the pseudo-label strategy introduces noise and cannot distinguish the target domain (new hospital) data well, so the migration effect is poor.

[0009] The disadvantages of the above four acne severity grading model migration methods are: methods without migration usually perform poorly in the field of acne grading. The method of re-collecting data for training fails to utilize the existing model, and collecting data and labeling requires a lot of manpower and lacks versatility. On the other hand, the transfer learning method either requires obtaining data from the original hospital or is based on pseudo-labels, which will bring noise. These methods fail to perform model migration well, and also fail to deal well with the characteristics of high inter-class similarity and large intra-class changes in acne grading, and the migration effect is limited.

[0010] Acne image grading models can automatically complete acne diagnosis, assist doctors in diagnosis, and provide patients with a way to understand their own condition. However, acne grading models based on deep learning are usually trained on data collected in a specific hospital. Therefore, when the model is migrated to a new hospital or other region, the shift in data distribution usually leads to a decrease in effect, so measures need to be taken to ensure that the model can be reasonably migrated to other environments. In addition, the existing methods either need to re-collect data training in the target hospital, which consumes a lot of manpower and material resources, or need to obtain the original training data during migration, which usually involves patient privacy and ethics in the field of medical imaging. In addition, some migration methods that do not require original data ignore the characteristics of the acne grading task, so the migration effect is poor.

[0011] In summary, the shortcomings of the existing acne grading model and the technical problems to be solved by this method are summarized as follows:

[0012] 1) The acne grading model relies on training data, and its performance deteriorates in new regions or hospitals due to the shift in data distribution.

[0013] 2) Some methods improve the robustness of the model by smoothing labels during model training to adapt to new data, however this does not work well in the field of acne grading.

[0014] 3) Some methods collect and annotate data from new hospitals, perform fine-tuning and supplement training to achieve model migration, which consumes a lot of manpower and material resources, and under some conditions, it is impossible to collect sufficient data for training.

[0015] 4) When acquiring new unlabeled data, existing migration methods also need to obtain labeled data from the original hospital, which usually violates patient privacy and ethics in the field of medical imaging and lacks practical utility.

[0016] 5) Some migration methods use pseudo-labeling for new data to migrate the acne grading model, but the noise caused by the pseudo-labeling will reduce the model effect.

[0017] 6) The model transfer ignores the characteristics of the acne grading task, the similarities between different acne severities, and the changes within the same severity. Summary of the invention

[0018] In order to solve the above technical problems, the present invention proposes a large-model-based acne grading model migration method.

[0019] The technical solution adopted by the present invention is: a method for migrating an acne grading model based on a large model, wherein the large model includes a visual encoder and a text encoder; and further includes establishing an acne grading model, wherein the acne grading model includes a feature extractor and a classifier; the migration method specifically includes the following steps:

[0020] S1. The data with acne severity level labels are recorded as the source domain data set, and the data without acne severity level labels are recorded as the target domain data set; the initial acne grading model is the acne grading model that has been trained with the source domain data set;

[0021] S2, dividing the target domain dataset into multiple batches, each batch including a number of target domain data;

[0022] S3. During the iteration of the current batch, the current target domain data in the current batch is input into the large model and the acne grading model respectively. The visual encoder in the large model outputs a visual vector according to the input target domain data; the text encoder in the large model outputs a text vector according to the input target domain data; the large model outputs the large model prediction result corresponding to the current target domain data according to the similarity between the visual vector and the text vector; the feature extractor in the acne grading model outputs the feature vector of the current target domain data, and the classifier in the acne grading model outputs the prediction result corresponding to the current target domain data according to the feature vector;

[0023] S4, the large model calculates the mutual trust loss function corresponding to the current target domain data according to the prediction result output by the large model and the prediction result output by the current acne grading model; and adjusts the text encoder in the large model based on the mutual trust loss function;

[0024] S5, the acne grading model calculates a diversity loss function according to the prediction result output by the current large model and the prediction result output by the current acne grading model;

[0025] Calculate the neighbor consistency loss based on the feature vector output by the feature extractor in the current acne grading model and the prediction result output by the current acne grading model;

[0026] Calculate the category incentive loss based on the prediction results output by the current acne grading model and the corresponding weights;

[0027] The total loss function corresponding to the current target domain data in the current batch is calculated according to the sum of the diversity loss function, the neighbor consistency loss, and the category incentive loss; and the current acne grading model is updated using the gradient descent method based on the total loss function;

[0028] S6. Calculate the classification accuracy of the acne grading model corresponding to each target domain data during the iteration of the current batch, retain the acne grading model with the highest classification accuracy, and if all batches of iterative training have been completed, use the acne grading model with the highest current classification accuracy as the migrated acne grading model; otherwise, return to step S3.

[0029] Beneficial effects of the present invention: Therefore, this method is based on the research on domain adaptation in deep learning, and proposes a migration method for acne grading models based on large models. This method can solve the problem of performance degradation when the acne grading model is migrated to a new hospital. At the same time, the proposed method does not need to obtain the data of the original hospital during migration training, and also avoids ethical and privacy issues when obtaining the original data. In the new hospital, only an appropriate amount of unlabeled data needs to be collected to complete the migration, and a good effect is achieved, which greatly reduces the data difficulty in the migration process of the acne grading model. At the same time, according to the characteristics of the acne grading model, the proposed neighbor prediction consistency loss and inter-class discrimination loss also improve the diagnostic performance of the migration model for acne. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Acne images from different datasets;

[0031] Figure 2 for Figure 2 CLIP overall structure;

[0032] Figure 3 is a flow chart of the method of the present invention;

[0033] Figure 4 This is a structural diagram of the acne classification model;

[0034] Figure 5 This is the overall structure diagram of the model migration method;

[0035] Figure 6 Schematic diagram of consistency loss for neighbor prediction;

[0036] Figure 7 Schematic diagram of category incentive loss;

[0037] Figure 8 This is the structure diagram of the CLIP submodule of the large model;

[0038] Fig. 9 Focus visualization for acne features. DETAILED DESCRIPTION

[0039] In order to facilitate those skilled in the art to understand the technical content of the present invention, the following technical terms are first explained:

[0040] 1. Deep Neural Net (DNN)

[0041] Deep neural networks are a type of feedforward neural networks, and the convolutional neural network (CNN) model is usually used in the field of computer vision. For the acne severity grading task, the image features are usually extracted through the convolution kernel of the convolutional neural network, and then the classifier layer is trained based on the obtained image features. Through a certain amount of data training, the parameters of the feature extraction layer and the classification layer are continuously adjusted to achieve the grading task.

[0042] 2. Large Model

[0043] The large model is usually a deep neural network model with hundreds of millions to tens of billions of parameters. It is trained unsupervised and self-supervised through a large amount of unlabeled data to learn rich representations from the massive data. It has stronger versatility and can often perform well on multiple tasks (such as image recognition, natural language processing, etc.), and can classify images without any labels without any training. The method of the present invention is to utilize the general classification ability of the CLIP (Contrastive Language-Image Pre-training, a pre-training model based on contrasting text-image pairs) large model to achieve the migration of the acne grading model on the unlabeled target data set, so that the acne grading model can also be accurately graded on the target data set. The structure of the CLIP model is as follows Figure 2 shown.

[0044] 3. Domain Adaptation

[0045] Domain adaptation is an important research direction of transfer learning, and its purpose is to transfer a model trained on one dataset to another dataset while still achieving the original effect. This is because the model trained on a specific dataset often performs worse when it is actually deployed or transferred to similar tasks due to changes in data distribution. Re-collecting data in the target domain for annotation usually consumes additional resources, and in some cases, the target domain cannot collect enough data to retrain the model. The task of transfer learning is to study the use of trained models and perform a small amount of training on the target domain dataset to achieve a better effect.

[0046] The method proposed in the present invention first uses the trained small model to adjust the large model so that it can adapt to the field of acne grading. Then, the zero-shot grading ability of the large model is used to grade the unlabeled data collected by the target hospital as a guide for model migration. At the same time, the model's ability to distinguish between different severity levels of acne is further improved through the designed neighbor prediction consistency loss and inter-class discrimination loss. Finally, without the need to carefully label the new data, the trained acne grading model can be migrated to the new hospital with better acne grading ability.

[0047] The present invention proposes a complete acne grading model migration method, which can be divided into four parts: (1) Collecting acne grading data from multiple different hospitals; (2) Designing an acne grading model and selecting a hospital data as the source data for training and testing; (3) Using the data sets of other hospitals as the target domain (i.e., the target hospital), and these data are all unlabeled, the large model is adjusted on the target hospital data set to adapt it to the acne grading task. (4) Using the output of the large model and the designed loss function on the target hospital data set, the acne grading model is adjusted to complete the model migration process. (5) Model testing. The overall flow chart is as follows: Figure 3 As shown:

[0048] (1) Prepare acne image data from multiple hospitals: Collect acne image data from multiple hospitals or different shooting methods, and use each category as an acne grading dataset. Then, professional dermatologists annotate the acne image data, and then divide these multiple datasets into source domains and target domains. The source domain is used to train the acne grading model to simulate existing research. The target domain represents the data of the new hospital when using the model.

[0049] 11. Collect datasets from different hospitals or different styles

[0050] There are three data sets used in this embodiment: the first data set a is the public acne severity grading data set ACNE04, which has a picture style of changing angles. The shooting equipment includes mobile phones and cameras, and focuses on shooting the acne locations on the patient's face. A total of 1,457 patient pictures are included. The second data set b is professional diagnostic pictures taken by VISIA equipment collected in cooperation with dermatologists. The picture contains the complete part of the patient's face and retains the high-definition details of the face. The data set is named HXSKIN04, with a total of 1,540 pictures. The third data set c is the data uploaded by patients collected through the mini program. The patients took and uploaded the pictures with their own mobile phones. In addition to the patient's facial area, there are also interferences from the environmental background and clothing background. A total of 656 pictures are included, and the data set is named WXSKIN04. Some examples of these three data sets can be found Figure 1Finally, with the help of dermatologists, the data in each dataset were classified into four different levels according to the severity of acne; the severity level served as the label of the corresponding image.

[0051] 12. Data Processing

[0052] Data processing is a prerequisite for model training and model migration. The acne images used in this embodiment have three dimensions: length, width, and channel. During training and migration, the bilinear interpolation method is used to uniformly scale them to 520×520×3, and then they are cropped to 512×512×3 by center. Then channel normalization, random flipping, random rotation and other enhancements are performed. The severity of acne corresponding to the image, that is, the label part, is first encoded into the corresponding number, and then processed into one-hot form, as shown in Table 1. It should be noted that labels are only used when the dataset is used as the source domain, and there are no labels when it is used as the target domain.

[0053] Table 1. Acne severity coding results

[0054] Severity level Corresponding code One-hot Tags Mild 0 [1,0,0,0] Moderate 1 [0,1,0,0] serious 2 [0,0,1,0] Urgent need for medical attention 3 [0,0,0,1]

[0055] 13. Divide the source domain and target domain datasets

[0056] The model migration task is different from the usual deep learning process. It is necessary to evaluate the effect of migrating a model trained on a certain dataset to other datasets of similar tasks. Therefore, the method of the present invention uses three acne grading datasets with different data distributions: ACNE04, HXSKIN04, and WXSKIN04, which are divided into source domain and target domain datasets respectively. The source domain dataset refers to the acne grading model trained on this dataset using labeled data, representing a general grading method. The unlabeled target domain dataset is used for model migration and evaluation, representing the situation when the model is migrated to other hospitals. In order to obtain a more complete model migration result, the method of the present invention selects ACNE04, HXSKIN04, and WXSKIN04 as source domain datasets in turn, and uses the remaining two datasets as target domain data each time for model migration and evaluation.

[0057] (2) Design and train an acne grading model: Based on existing research, an acne grading model is designed, which consists of a feature extractor and a classifier. It is trained on the source domain (source hospital) data to achieve the acne grading effect. This model is used as a trained model.

[0058] This section introduces in detail the design of the acne grading model and the training process on the source domain dataset, which is used as the model to be migrated. According to existing research, the acne grading model usually consists of two parts: a feature extractor and a classifier. In the method of the present invention, the convolutional neural network ResNet50 is used as the feature extractor, and then the feature matrix is ​​processed into a feature vector through global average pooling. Finally, a multi-layer fully connected layer is used as a classifier to obtain the prediction results for each level. The one with the highest probability is the predicted level. The detailed structure and parameters of the model are shown in Figure 4 shown.

[0059] Then the model is trained on the selected source domain acne dataset, which is represented as: The subscript s indicates that the source domain dataset (with labels) is used. represents the input acne image, represents the corresponding one-hot label, i.e., the severity level of acne, k represents a total of k levels, n s Represents the total number of samples in the dataset. The acne grading task is actually to classify acne images, so the cross entropy loss is used as the target loss function. At the same time, since there are similarities between acne images of different severity levels, in order to prevent the overconfidence of the model from affecting the generalization, the labels are smoothed based on the cross entropy loss. The processing formula for label smoothing is as follows:

[0060]

[0061] Note that and They are all in vector form, where K is a vector of all 1s and its dimension is k. α is a factor that controls the smoothness and its range is between [0,1].

[0062] Acne ImagesX i After the feature extractor and the average pooling of the whole set, the feature vector R is obtained. i As shown below:

[0063]

[0064] ResNet() represents the feature extraction process, R i For acne images X i The corresponding eigenvectors are is in component form, and m is the dimension of the vector. Then, after passing through the classifier of multiple fully connected layers, the logit output H is obtained. i After the logit output passes through the softmaxt function, the predicted probability P of the image for each severity level is obtained. i , as shown in the following formula:

[0065]

[0066] Among them, fc represents the processing of the fully connected layer, softmax represents the corresponding function, and K represents the total number of levels.

[0067] Therefore, the prediction result P i With smoothed labels The loss calculation formula is as follows. By optimizing the loss function to reach the minimum value, the model can achieve the best acne grading effect on the source domain dataset.

[0068]

[0069] (3) Adjusting the large model for acne diagnosis: By using the mutual information loss function between the prediction results of the trained acne grading model and the prediction results of the large model, the large model is adjusted to have better prediction results in the acne grading task.

[0070] 31. Overall framework of acne grading model migration

[0071] The overall structure of the migration process is as follows Figure 5 As shown in the figure, the purpose is to make the model trained on the source domain dataset achieve good classification effect on the unlabeled target domain dataset, and the source domain dataset is not used in the training process. To represent the target domain dataset, the subscript t represents the target domain, n t represents the number of samples in the target domain dataset, Denotes the corresponding image. Note that only the unlabeled D is used in the migration process. t And the trained acne grading model. This is also the two major advantages of the method proposed in the present invention compared with other methods, no need to re-label, no need to obtain source data.

[0072] First, through the ability of large models, D t The prediction of acne image grade in the training acne grading model is used to perform mutual information loss function L mi , adjust the grading ability of the large model to adapt it to the acne grading task. Then, in the process of migrating the acne grading model, the target domain data D t To adjust the acne grading model, the prediction results of the large model are used as the supervision of the prediction results of the acne grading model, and the diversity loss function L divIncrease the diversity of prediction results to avoid collapse during the migration process. At the same time, find the neighbors of each sample through the feature vector output of the acne grading model, and use the neighbor consistency loss L nc To constrain the accuracy of model prediction. Then, based on the model prediction results, the designed class encouragement loss L ce , improve the discrimination of similar acne grades. Through the constraints of three loss functions, the acne grading model can be transferred without labels.

[0073] In order to explain the specific workflow of the proposed model in more detail, its pseudo code form is as follows:

[0074]

[0075] 32Adapting the large model to the acne grading task

[0076] In the proposed acne model migration method, the large model corrects the erroneous output of the acne grading model through its grading results, and avoids the prediction results of the acne grading model from all predicting the same category. Because there are labels to complete these tasks during training in the source domain, there are no labels in the target domain. However, the large model is composed of a large amount of image and text data, and is trained through comparative learning, which provides image and text matching capabilities. It is a general model and has a relatively poor effect when used directly on the acne grading task. The method proposed in the present invention is not to retrain the large model, but to fine-tune the learnable parameters of the text feature extractor in the text encoder of the large model through the output of the acne grading model through mutual information loss, so that the text encoder pays more attention to features related to acne grade when converting text into vector expressions, thereby improving the prediction effect of the general large model on the acne grading task.

[0077] CLIP acne grading process Figure 4 As shown in the figure, first construct the corresponding text for each acne severity, and then use the large model text encoder to extract the feature vector corresponding to each level of text. Then, for the input acne image, use the large model image encoder to obtain the corresponding feature vector, calculate the distance between the image vector and the text vector corresponding to each level, and use the Softmax function to process it as the predicted probability of each severity level. The level corresponding to the highest probability is the predicted severity level. As shown in the following formula:

[0078]

[0079] Where T is the extracted text feature matrix, each column corresponds to each level, q i is the feature vector of the input image, The vector represents the distance between the image vector and each text vector. The vector represents the predicted probability of each level after the distance vector is converted by the Softmaxt function. Where i represents the sample number, K represents the total number of acne levels, k represents the kth level, C is the dimension of the feature vector, and l represents the output data in the large model.

[0080] The predicted output of the acne grading model is then used as the reference target for the predicted output of the large model, and the mutual information loss function L mi Optimize the learnable parameters of the self-attention module and the fully connected module in the text feature extractor of the large model, so that the large model of the general field can perform better in the acne grading task. mi The definition of the loss function is as follows. l represents the prediction output of the large model, P t represents the predicted output of the acne grading model. P represents the joint distribution of the two predicted outputs as a two-dimensional matrix with dimensions K×K, where K is the total number of grades, and n t represents the total number of samples, and T represents the transpose of the vector.

[0081]

[0082] 33 Prediction Diversity Loss Design in Migration Task

[0083] In the target domain When migrating the acne grading model, since there is no constraint on label Y, the predicted result P t It will change in the direction of smaller entropy, that is, it will approach the same label. Normal acne grade labels should be in the form of [1,0,0,0], [0,1,0,10]. The purpose of migration is to make the acne grading model in D t The prediction result P t If the entropy is close to this form, it means that the predicted acne grade is correct and clear. However, if the entropy is small, the acne grading model may predict the form of [1,0,0,0] on all images, which will lose the grading ability. The traditional method is to take the average of the predicted results of all samples. and This fixed vector calculates the cross entropy loss to constrain the collapse of the prediction results. However, there are differences between each acne image, and the number of acne images of each grade is unbalanced, so it is not reasonable to let it approach K. In the method of the present invention, let the prediction P of the acne grading model t Compared with the prediction of the large model lBecause the adjustment of the large model and the migration of the acne grading model are performed alternately, the large model does not participate in the training at this time, so there will be no performance collapse, and the prediction results for a single image are more referenceable. Taking it as a constraint target for diversity has achieved better results. The specific loss function is shown below, where K represents the total number of grades. and It is the weight of the acne grading model and the prediction results of the large model at each level.

[0084]

[0085] 34 Neighbor Prediction Consistency Loss Design

[0086] Through the diversity loss constraint, the prediction results of the acne grading model will not explore the same level during the migration process, which ensures the diversity of the prediction results. At the same time, it is close to the prediction results of the large model, which also ensures the correctness of the acne grading model. However, our goal is to make the acne grading model in the target domain dataset The prediction effect is better, but at the same time, no labels and source domain datasets are used. Through experiments, it can be noticed that the feature vectors extracted by the feature extractor in the acne grading model have a certain aggregation phenomenon after two-dimensional visualization. That is to say, although the acne grading model has a poor effect on the target domain, due to the synergy of distribution shift, the output feature vectors are still separable. Therefore, the method of the present invention uses this feature to propose a neighbor prediction consistency loss, and its principle is as follows Figure 6 shown.

[0087] The feature vectors close to each other are called neighbors, so the corresponding prediction results should also be similar. By finding the prediction result P t The prediction results of the nearest m neighbors corresponding to i are made closer to each other in terms of distance. This makes the prediction results of the neighbors closer to the correct direction, thereby improving the accuracy of the prediction. At the same time, the model's ability to distinguish the similarities between acne images of different severity is also improved. The detailed calculation process of the loss function is as follows. First, calculate X i The corresponding eigenvector H i ∈R 1×c With feature vector memory The distance between each vector in Where c represents the dimension of the feature vector, n t Represents the total number of samples.

[0088]

[0089] Then from D i Select m with the smallest distance as Hi In practical applications, the value of m is required to cover valid neighbors; in this embodiment, the value of m is the total number of levels minus 1, H i The set of neighbors of i According to the previous theory, H i The corresponding prediction result P i The prediction results corresponding to its neighbors should be consistent, so the calculation formula of the neighbor prediction consistency function is as follows, where ||S i || represents the set S i The number of elements in, k represents the level, p ik and p jk They are the model prediction results P i and the neighbor prediction result P j The quantity form.

[0090]

[0091] 35 Category Incentive Loss Design

[0092] Since there are similarities between images of different severity in acne diagnosis, and there are different degrees of variation between acne images of the same severity, a class-incentive loss function is proposed to constrain the prediction results, so that there is greater differentiation between similar categories and between different categories, thereby improving the model's ability to distinguish between different levels of acne. The principle is as follows Figure 7 shown.

[0093] The prediction results of the acne grading model are expressed as P i =[p i1 ... p i2 ... p ik ], according to the previous definition, its component p ik It is divided into two parts, namely, those above the threshold of 0.75 indicate that it is closer to the true level, called the positive component, and those below the threshold indicate that it is not related to the level, called the negative component. Category incentives are based on the adjustment of the prediction results based on the weight of the positive component and the weight of the negative component. At the same time, the change of this weight just coincides with the prediction result, so the prediction result component is used to generate the adaptive weight, avoiding the complexity brought by manual parameters. As shown below, Z + Indicates P i The set of subscripts of the positive components in the equation, and the corresponding weight a is constructed by the positive and negative components. i and b i , then calculate the category incentive loss L ce .

[0094] P i =[p i1 ... pi2 ... p ik ]

[0095]

[0096] The final loss function of the acne grading model migration is L = L div +L nc +L ce .

[0097] (4) Acne grading model migration: The main purpose of this step is to migrate the trained acne grading model to the data of the target domain (other hospitals) so that it still maintains good performance. It mainly includes the mutual information loss between the large model and the acne grading model, and achieves prediction diversity through mutual correction. Then, the neighbor consistency loss and class incentive loss of the acne model output results are used to improve the distinguishability of different acne.

[0098] This part is the core protection point of this patent, which mainly includes the detailed process of the proposed acne grading model migration method and the improvement compared with previous methods.

[0099] This section details the process of transferring the acne grading model to the target domain dataset. The specific implementation includes data encoding, model parameter setting, back propagation during training, and parameter optimization process.

[0100] 41. Data Encoding

[0101] First, the input data of the large model is divided into two parts: text data and image data. The image data is the patient's facial acne image. Each image is processed to 512×512 pixels in size according to the introduction of step 22. Then, similar to the input of the acne grading image, each image is enhanced by random cropping, horizontal flipping, and random rotation to form a form different from the original image. And normalization is performed in the channel dimension to accelerate the convergence process of the model. For text data, the doctor's annotated severity level annotation is mapped to "This is a facial acne image of [ ]", and the [ ] is filled with the text of four levels of "mild", "moderate", "severe", and "very severe". The input of the acne grading image only has the image part, which is processed in the same way as the large model image input.

[0102] 42. Detailed construction of large model and acne grading model

[0103] The acne grading model consists of a feature extractor and a classifier. The feature extractor uses ResNet50, which is also the best result obtained in the experiment. The classifier is composed of three layers of fully connected layers. The construction of the large model CLIP is divided into two parts. The visual encoder part uses the commonly used VIT-B / 32 model. The word embedding part in text processing uses the Byte-pairEmbedding structure. The text encoder part uses the mature transformer / base model in the field of natural language processing. The specific structure is as follows: Figure 8 As shown, the parameters in the self-attention module and the fully connected module in the text encoder are learnable. The structure of the large model CLIP is a well-known technology in the art, and the present invention will not be described in detail here.

[0104] 43. Loss function, back propagation and parameter optimization

[0105] The loss function is the goal of model optimization. By calculating the gradient of the loss function, the parameters of the model are optimized by back propagation to achieve our ideal effect. Specifically, the gradient is calculated by deriving the loss function, and then the learning rate is used in the gradient direction to update the parameters according to a certain proportion, and the model is adjusted continuously to achieve the optimal parameters. The loss function design of the method of the present invention has been described in detail in the previous text. Compared with the learning rate part, since the method of the present invention is a migration process for a trained model, a differentiated learning rate setting is adopted. For the feature extractor part of the acne grading model, the learning rate is set to 0.0005, and for the classifier part, it is set to 0.0001, and the stochastic gradient optimizer (SGD) is used to select the direction of gradient optimization. For the fine-tuning part of the large model, the learning rate is set to 0.00001, because it already has a good effect. The total number of training rounds is set to 30 rounds, and the amount of data for each batch in each round is set to 64. After each round of training is completed, the model effect is tested according to the evaluation criteria below, and the model with the best grading accuracy (Accuracy) is selected as the migrated acne grading model. Because in the migration task, a small number of rounds can achieve a good migration effect, early stopping is used to avoid excessive migration.

[0106] (5) Evaluation of the migrated model: Select the performance criteria in the field of acne grading, and then test and evaluate the performance of the unmigrated acne grading model and the migrated model on multiple target domain (target hospital) datasets.

[0107] The main purpose of the test evaluation stage is to test the effect of the migrated acne grading model on the target domain acne data set to evaluate whether the available performance has been achieved. In the method of the present invention, the accuracy (Accuracy), precision (Precision), sensitivity (Sensitivity), specificity (Specificity) and Youden index (YI) commonly used in medical image classification are mainly used to measure the grading effect of the migrated model. The value of Youden index (YI) is (sensitivity + specificity -1), which can comprehensively measure the effectiveness of model diagnosis by combining sensitivity and specificity. All these indicators are as high as possible. In order to test the integrity of the evaluation, three data sets ACNE04, HXSKIN (abbreviated as HX), and WXSKIN (abbreviated as WX) are used as target data sets respectively, and the remaining two are used as target data sets. "ACNE04->HXSKIN" represents the migration from the source data set to the target data set. The specific results are shown in Table 2.

[0108] Table 2 Classification effect of the model after migration

[0109] Dataset Metrics Accuracy Accuracy Specificity Sensitivity Youden Index ACE04->WX 0.7099 0.7899 0.8905 0.7563 0.6468 ACE04->HX 0.7561 0.7860 0.9205 0.7661 0.6866 WX->ACNE04 0.8030 0.6809 0.9562 0.8001 0.7564 WX->HX 0.7504 0.6790 0.9428 0.7489 0.6817 HX->ACNE04 0.8490 0.8021 0.9635 0.8188 0.7823 HX->WX 0.6970 0.7900 0.9553 0.8400 0.7953

[0110] In order to compare the performance of the proposed method, the method of the present invention is compared with other model transfer methods, non-transfer methods and labeled retraining models (i.e., retraining models on target domain datasets), and the results are shown in Table 3. Here, the classification accuracy is mainly used as the evaluation criterion.

[0111] Table 3 Comparison results of the method of the present invention with other methods

[0112] Method Name ACE04->WX ACE04->HX WX->ACNE04 WX->HX HX->ACNE04 HX->WX No migration 0.5343 0.6123 0.4032 0.4122 0.6799 0.5033 Heavy training 0.7132 0.7601 0.8632 0.7601 0.8632 0.7132 DANN 0.6532 0.6943 0.7033 0.6876 0.7067 0.6003 DAN 0.6648 0.6834 0.7189 0.6990 0.7321 0.6342 SHOT 0.6790 0.6855 0.7221 0.6700 0.7201 0.6620 NRC 0.6795 0.7232 0.7399 0.7032 0.8090 0.6540 GKD 0.6832 0.7434 07867 0.7243 0.8390 0.6800 Method of the present invention 0.7099 0.7561 0.8030 0.7504 0.8490 0.6970

[0113] The method of the present invention in Table 3 shows the best migration results. It can be noted that when there is no migration, that is, the model trained in the source domain is directly used in the target domain, the effect is very poor and the available performance cannot be achieved. However, after the migration method, the performance of the model has been greatly improved. Among them, the DANN and DAN migration methods also need to obtain source domain data during migration, but the performance is even worse than SHOT, NRC, GKD and the proposed source-free domain method, so the source domain data set is not necessary during migration. At the same time, we noticed that the performance of the WX data set during migration and training is very close. Because the scale of the WX data set is small, it is a better method to use existing models for migration when the data is insufficient. Finally, the acne grading model migration method based on a large model proposed in this paper achieved the best results. On the one hand, it used the prediction results of the large model and the source model to guide each other. On the other hand, the designed diversity loss function, neighbor prediction consistency loss function and category incentive loss function all play a very important role and constrain the results of the model.

[0114] In addition, in order to demonstrate the proposed migration method, Fig. 9 As shown in the figure, the feature map of the acne grading model is converted into a heat map using the CAM method, and the heat map of the retraining method is compared with the heat map of the non-migrated method and the heat map of the proposed migration method to observe the degree of attention to acne lesions. By comparing the visualization of acne feature attention, it can be found that the acne grading model used directly without migration still retains the ability to pay attention to acne features, which also illustrates the feasibility of using existing models. However, there is a phenomenon of missing or mis-attention to attention features, which is caused by the data offset between the source domain and the target domain, and is also the reason for the decrease in target and gradability. Comparing the attention map after migration with the attention map before migration, it can be found that after migration training, the model can better pay attention to the rubbing acne features, which also proves the effectiveness of the proposed method.

[0115] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for migrating acne grading models based on a large model, characterized in that: The large model based on it includes a visual encoder and a text encoder; it also includes establishing an acne grading model, which includes a feature extractor and a classifier; the migration method specifically includes the following steps: S1. The data with acne severity level labels are recorded as the source domain data set, and the data without acne severity level labels are recorded as the target domain data set; the initial acne grading model is the acne grading model that has been trained with the source domain data set; S2, dividing the target domain dataset into multiple batches, each batch including a number of target domain data; S3. In the current iteration process, the target domain data of the current batch in the current iteration are respectively input into the large model and the acne grading model. The visual encoder in the large model outputs visual features according to the input target domain data of the current batch; the text encoder in the large model outputs text features according to the input target domain data of the current batch; the large model outputs the large model prediction results corresponding to the target domain data of the current batch according to the similarity between the visual features and the text features; the feature extractor in the acne grading model outputs the feature vector of the target domain data of the current batch, and the classifier in the acne grading model outputs the prediction results corresponding to the target domain data of the current batch according to the feature vector; S4, the large model calculates the mutual trust loss function corresponding to the target domain data of the current batch according to the prediction result output by the large model and the prediction result output by the current acne grading model; and adjusts the text encoder in the large model based on the mutual trust loss function; S5, the acne grading model calculates a diversity loss function according to the prediction result output by the current large model and the prediction result output by the current acne grading model; Calculate the neighbor consistency loss based on the feature vector output by the feature extractor in the current acne grading model and the prediction result output by the current acne grading model; Calculate the category incentive loss based on the prediction results output by the current acne grading model and the corresponding weights; The total loss function corresponding to the target domain data of the current batch is calculated according to the sum of the diversity loss function, the neighbor consistency loss, and the category incentive loss; and the current acne grading model is updated using the gradient descent method based on the total loss function; S6. Calculate the classification accuracy of the acne grading model corresponding to each batch of target domain data in the current iteration process, retain the acne grading model with the highest classification accuracy, and if all batches of iterative training have been completed, use the acne grading model with the highest current classification accuracy as the migrated acne grading model; otherwise, return to step S3.

2. The method for migrating an acne grading model based on a large model according to claim 1, characterized in that: The feature extractor of the acne grading model is as follows: the convolutional neural network ResNet50 is used to extract the feature matrix of the input data, and then the feature matrix is ​​processed into a feature vector through global average pooling.

3. The method for migrating an acne grading model based on a large model according to claim 2, characterized in that: The acne grading model uses multiple fully connected layers as classifiers.

4. The method for migrating an acne grading model based on a large model CLIP according to claim 3, characterized in that: The large model specifically adopts the CLIP model, and its text encoder adopts the transformer / base model. The text encoder specifically includes: a text generation unit, a word embedding unit, and a text feature extraction unit; the text generation unit generates text descriptions of acne of different severities according to the input data; The word embedding unit performs word segmentation and mapping processing on the text representation output by the text generation unit to obtain a text vector, and the text feature extraction unit extracts text features based on the text vector output by the word embedding unit.

5. The method for migrating an acne grading model based on a large model according to claim 4, characterized in that: The visual encoder of the large model uses the VIT-B / 32 model to extract image features.

6. The method for migrating an acne grading model based on a large model according to claim 5, characterized in that: The mutual information loss function is denoted as L mi , the expression is: Among them, p ij The elements in the matrix P representing the joint distribution properties of the prediction results of the large model and the prediction results of the acne grading model, The superscript T indicates transposition, P l represents the prediction result of the large model, P t represents the prediction result of the acne grading model, Represents the i-th element in the prediction result of the large model, represents the jth element in the prediction result of the acne grading model, c represents the dimension of the feature vector, n t represents the total number of samples in P.

7. The method for migrating an acne grading model based on a large model according to claim 6, characterized in that: The diversity loss function is denoted as L div , L div The calculation expression is: Where K represents the total number of acne severity levels, represents the component of the prediction result of the acne grading model at the kth level, Represents the component of the prediction result of the large model at the kth level.

8. The method for migrating an acne grading model based on a large model according to claim 7, characterized in that: The feature vector output by the feature extractor in the acne grading model corresponding to the current batch of target domain data is H i , H i The prediction result of the corresponding acne grading model is recorded as P i , set the feature vector memory to store the feature vectors output by the feature extractor in the acne grading model corresponding to each batch of target domain data, and calculate H i The distance between the feature vectors and the rest of the feature vectors in the feature vector memory; Sort the distances from smallest to smallest, and select the eigenvectors corresponding to the first several distances as H i The neighbors of i ; Neighbor consistency loss is denoted as L nc , L nc The calculation expression is: Among them, p ik Indicates P i The kth component of jk Indicates H i The kth component of the prediction result of the acne grading model corresponding to the jth neighbor of ||S i || represents the set S i The number of elements in .

9. The method for migrating an acne grading model based on a large model according to claim 8, characterized in that: Set the threshold value and set P i The component in is greater than the threshold value and recorded as positive component, and the component less than or equal to the threshold value is recorded as negative component; the category excitation loss is recorded as L ce , L ce The calculation formula is: Among them, Z + Indicates P i The set of subscripts for the positive components in a i According to P i The weight of the positive component in the construction, b i According to P i The weight of the negative component construction.

Images