An automatic acne severity grading system based on multi-task learning

Abstract

The application discloses a kind of based on multi-task learning's acne severity automatic grading system, it belongs to medical image processing field.The present application solves the accuracy rate and the low efficiency problem of existing acne severity grading method.The application can construct suitable sequence tree by designing sequence tree construction method, and greatly reduce the time cost of finding suitable sequence tree, improve the efficiency of grading.By introducing label distribution learning, and combining label distribution learning with sequence tree, adjacent acne severity can be better distinguished, the problem that adjacent acne severity is difficult to distinguish is solved, the effect of overall acne severity grading is significantly improved, and the accuracy of acne severity grading is guaranteed.The method of the application can be applied to acne severity grading.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing, specifically relating to an automatic acne severity grading system based on multi-task learning. Background Technology

[0002] Acne vulgaris, also known as pimples, is a common inflammatory skin condition related to the hair follicles and sebaceous glands, with the highest incidence in adolescents. Because acne is a chronic condition, it can persist for years, negatively impacting a patient's quality of life, self-esteem, and mood, and increasing the risk of anxiety and depression. Therefore, timely and effective diagnosis and treatment of acne are crucial. Accurate acne severity grading is a key step in developing an appropriate treatment plan. The Hayashi grading system is commonly used by dermatologists to classify acne severity into four levels: mild, moderate, severe, and very severe, based on the number of lesions. Traditional assessment methods involve dermatologists observing and then evaluating the severity of acne; this method is time-consuming, relies heavily on the dermatologist's expertise, and struggles to provide accurate acne severity assessments. Therefore, an accurate computer-aided grading system is essential for developing a suitable acne treatment plan. Summary of the Invention

[0003] The purpose of this invention is to address the problems of low accuracy and low efficiency in existing acne severity grading methods, and to propose an automatic acne severity grading system based on multi-task learning.

[0004] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0005] An automatic acne severity grading system based on multi-task learning, the system comprising a label distribution generation module, a semantic similarity calculation module, a sequence tree module, a decision sequence generation module, a multi-hot label encoding module, and a neural network module; wherein:

[0006] The label distribution generation module is used to convert the real acne severity labels of the input image into a discrete label distribution;

[0007] The semantic similarity calculation module is used to calculate the semantic similarity of the label distribution corresponding to adjacent acne severity labels;

[0008] The sequence tree module is used to construct a sequence tree based on the semantic similarity of the distribution of labels corresponding to adjacent acne severity labels;

[0009] The decision sequence generation module is used to convert each acne severity label into a binary decision sequence based on the sequence tree;

[0010] The multi-hot tag encoding module is used to generate multi-hot tag encodings for each acne severity tag based on the sequence tree;

[0011] The neural network module includes a label distribution branch and a multi-hot label encoding branch. The label distribution branch is trained using the output of the label distribution generation module, and the multi-hot label encoding branch is trained using the output of the decision sequence generation module and the output of the multi-hot label encoding module.

[0012] The trained label distribution branch and multi-hot label encoding branch are used to classify the severity of acne in the images to be graded.

[0013] Furthermore, the working process of the label distribution generation module is as follows:

[0014] Acne severity is divided into Z levels, and the labels for the Z levels of acne severity are denoted as follows: , … ,Label , … Together they form a set of acne severity labels;

[0015] For the input image Input image The true severity of acne is labeled as follows: Tags are generated based on a Gaussian function and a set of acne severity labels. The label distribution, where the first Acne severity label For the input image The descriptive degree is:

[0016]

[0017] in, It is the [number]th in the acne severity label set. Acne severity labels It's a hyperparameter. It is a regularization factor;

[0018] The true severity label for acne The corresponding label distribution is , It is the size of the acne severity label set.

[0019] Furthermore, the regularization factor for:

[0020]

[0021] in, It is the [number]th in the acne severity label set. Acne severity labels.

[0022] Furthermore, the semantic similarity calculation module operates as follows:

[0023] Step S11: Calculate the similarity matrix :

[0024]

[0025] in, It is a similarity matrix The Middle Line number Column elements, , = 2, ... , = 2, ... , This represents taking the absolute value. ;

[0026] Step S12: Group the acne severity labels into the first... The label distribution corresponding to each acne severity label is denoted as . The severity of acne is categorized into the first category. The label distribution corresponding to each acne severity label is denoted as . Then the first The label distribution corresponding to the first acne severity label and the first The semantic similarity of the label distribution corresponding to each acne severity label is as follows:

[0027]

[0028] in, It is the first The label distribution corresponding to the first acne severity label and the first The semantic similarity of the label distribution corresponding to each acne severity label, with the superscript T indicating transposition.

[0029] Furthermore, the working process of the sequence tree module is as follows:

[0030] Step S21: Given a set of leaf nodes and initialize the node set. ;

[0031] Step S22: Select a set of nodes Partition node index :

[0032]

[0033] in, Representatives utilize nodes For sets The ratio of samples in the left subtree to the right subtree after the partitioning;

[0034]

[0035] in, The labels representing the actual acne severity in the training set are: The corresponding number of image samples;

[0036] After the partitioning operation, two new sets of nodes are generated. and ;

[0037] Step S23: Determine whether there exists a set with more than 1 nodes in the node set obtained from step S22.

[0038] If there exists a set with more than 1 node, then return to step S22 for each set with more than 1 node.

[0039] If there is no set with more than 1 node, the sequence tree is constructed.

[0040] Furthermore, the working process of the decision sequence generation module is as follows:

[0041] For any acne severity label According to the sequence tree root node to the label The path of the corresponding child node is used to generate acne severity labels. The corresponding binary decision sequence;

[0042] The generated binary decision sequence Represented as:

[0043]

[0044] Where H represents the height of the sequence tree, Acne severity level label The corresponding path encoding sequence from the root node to the leaf node, h = 1, 2, ..., H;

[0045] binary decision sequence Shift the elements in the binary decision sequence one position to the right and fill the first position with the start marker to obtain the final binary decision sequence. ;

[0046] .

[0047] Furthermore, the operation process of the multi-hotspot encoding module is as follows:

[0048] Acne severity label Label the severity of acne The corresponding node is the th in the sequence tree The multi-hot tag encoding of the layer is denoted as , ,in, and obtain the first In the layer, the label is related to the severity of acne. The corresponding node is a node within the same set, and the retrieved node is... The value of the corresponding element is also 1. The values ​​of all other elements in the array are 0;

[0049] Similarly, obtain acne severity labels After the corresponding nodes are encoded with multi-hot labels at each level of the sequence tree, the acne severity label is obtained. Multi-hot tag encoding sequence ;

[0050]

[0051] Similarly, the multi-hot tag encoding sequence for each acne severity label is obtained.

[0052] Furthermore, the label distribution branch and the multi-hot label encoding branch are jointly trained. The specific process of joint training is as follows:

[0053] Step S31: After preprocessing the input image, input the preprocessed image into the feature extraction network to extract image features;

[0054] Step S32: Input the image features extracted in step S31 into the label distribution branch. In the label distribution branch, the extracted image features first pass through a fully connected layer for feature dimensionality reduction, and then the dimensionality-reduced features pass through a Softmax layer to obtain the predicted label distribution. ;

[0055] Calculate the loss between the predicted label distribution and the label distribution generated by the label distribution generation module. :

[0056]

[0057] in, It is the distribution of the true labels corresponding to the input image. express and The cumulative JS divergence;

[0058] The cumulative JS divergence is calculated as follows:

[0059]

[0060] in, It is the cumulative density function. It is JS divergence. , It is the distribution of labels The Middle The values ​​corresponding to the severity labels of acne. , It is the distribution of labels The Middle The value corresponding to each acne severity label;

[0061]

[0062] in, ;

[0063] Step S33: Input the image features extracted in step S31 into the multi-hot label coding branch. In the multi-hot label coding branch, the extracted image features are first encoded by the Transformer encoder to obtain the encoding result; then, based on the encoding result and the final binary decision sequence, the multi-hot label codes predicted in each step are obtained. , t=1,2,…,H;

[0064] Calculate the loss between the predicted multi-hot tag encoded sequence and the multi-hot tag encoded sequence generated by the multi-hot tag encoding module. ;

[0065]

[0066] in, It is a binary cross-entropy function. express The first in One element, express The first in One element, = ;

[0067] Step S34 combines the losses from steps S32 and S33, and then backpropagates the joint loss to train the model; whereby the joint loss... for: ;

[0068] in, It is a balance loss and The hyperparameters of the weights.

[0069] Furthermore, the multi-hot tag encoding predicted at each step is obtained based on the encoding results and the final binary decision sequence. Specifically:

[0070] Step S331: Initialize the number of steps t=1;

[0071] The encoding result and the final binary decision sequence are input into the selective mask decision decoder. The working process within the selective mask decision decoder is as follows:

[0072] Step S331: Initialize the number of steps t=1;

[0073] Step S332: Encode the position of the t-th element in the binary decision sequence corresponding to the input image;

[0074] Step S333: Input the position encoding result and the image features after passing through the Transformer encoder into the Transformer decoder to obtain the decoder output. ;

[0075] Step S334 After passing through the Sigmoid layer, the predicted probability sequence is obtained. ;

[0076] Step S335: Based on the probability sequence Generate mask sequence Specifically:

[0077] Based on the node partitioning results of the t-th level of the sequence tree, the probability sequence is... The elements in the data are divided into sets. In the partitioning results, for a set containing only one element, the value of the element is directly used as the value of the set. For a set containing two or more elements, the average value of all elements in the set needs to be calculated. That is, after obtaining the average value of each set, the average value is used as the value of the corresponding set.

[0078] The values ​​of each partitioned set are compared, and the set corresponding to the largest value is selected. This yields the acne severity label corresponding to the element in the selected set. The obtained acne severity label is then used in the mask sequence. The element corresponding to the given value is 1, and all other elements are 0.

[0079] Step S336: If t=1, then the multi-hot tag code predicted in step t is... = ;

[0080] If t > 1, then determine and Is the intersection of the two sets an empty set?

[0081] If it is not an empty set, then the multi-hot tag encoding predicted at step t is... = ;

[0082] If it is an empty set, then the multi-hot tag encoding predicted in step t is... = ;

[0083] Step S337: Determine whether the number of steps t has reached H;

[0084] If the number of steps t reaches H, then the iteration ends;

[0085] If step t has not reached H, then let t = t + 1 and return to execute step S332.

[0086] Furthermore, the step of using the trained label distribution branch and multi-hot label encoding branch to classify the severity of acne in the image to be classified is specifically as follows:

[0087] Step S41: Crop, rotate, and standardize the image to be graded to obtain the processed image;

[0088] Step S42: After extracting features from the processed image, input the extracted image features into the trained label distribution branch to obtain the predicted label distribution. ;

[0089] Step S43: Input the extracted image features into the multi-hot label coding branch, and then combine the binary decision results from each step. Obtain multi-hot tag encodings for each step of the prediction ;

[0090] Step S44, according to And the final step of predicting multi-hot tag encoding The acne severity grading results for the images to be graded were obtained. :

[0091] .

[0092] The beneficial effects of this invention are:

[0093] The sequence tree construction method presented in this invention can construct suitable sequence trees, significantly reducing the time cost of finding suitable sequence trees and improving the efficiency of grading. Furthermore, the introduction of label distribution learning, combined with sequence trees, can better distinguish the severity of adjacent acne lesions, solving the problem of difficulty in differentiating the severity of adjacent acne lesions, significantly improving the effectiveness of acne severity grading, and ensuring the accuracy of acne severity grading. Attached Figure Description

[0094] Figure 1 This is a flowchart of an automatic acne severity grading system based on multi-task learning according to the present invention;

[0095] Figure 2a This is a diagram illustrating tag conversion;

[0096] Figure 2b This is a schematic diagram of multi-hot tag encoding generation;

[0097] Figure 3 This is a structural diagram of the selective mask decision decoder;

[0098] Figure 4 This is a flowchart of selective masking decisions. Detailed Implementation

[0099] Specific implementation method one: Combining Figure 1 This embodiment describes an automatic acne severity grading system based on multi-task learning. In this invention, multi-task learning refers to the learning tasks of label distribution branches and multi-hot label encoding branches. The system includes a label distribution generation module, a semantic similarity calculation module, a sequence tree module, a decision sequence generation module, a multi-hot label encoding module, and a neural network module; wherein:

[0100] The label distribution generation module is used to convert the real acne severity labels of the input image into a discrete label distribution;

[0101] The semantic similarity calculation module is used to calculate the semantic similarity of the label distribution corresponding to adjacent acne severity labels;

[0102] The sequence tree module is used to construct a sequence tree based on the semantic similarity of the distribution of labels corresponding to adjacent acne severity labels;

[0103] The decision sequence generation module is used to convert each acne severity label into a binary decision sequence based on the sequence tree;

[0104] The multi-hot tag encoding module is used to generate multi-hot tag encodings for each acne severity tag based on the sequence tree;

[0105] The neural network module includes a label distribution branch and a multi-hot label encoding branch. The label distribution branch is trained using the output of the label distribution generation module, and the multi-hot label encoding branch is trained using the output of the decision sequence generation module and the output of the multi-hot label encoding module.

[0106] The trained label distribution branch and multi-hot label encoding branch are used to classify the severity of acne in the images to be graded.

[0107] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the working process of the tag distribution generation module is as follows:

[0108] Acne severity is divided into Z levels, and the labels for the Z levels of acne severity are denoted as follows: , … ,Label , … Together they form a set of acne severity labels;

[0109] In this invention, the value of Z is 4 as an example. =0, This indicates mild acne. 1, This represents moderate acne. 2, This indicates severe acne. 3, This indicates very severe acne;

[0110] For the input image Input image The true severity of acne is labeled as follows: Tags are generated based on a Gaussian function and a set of acne severity labels. The label distribution, where the first Acne severity label For the input image The descriptive degree is:

[0111]

[0112] in, It is the [number]th in the acne severity label set. Acne severity labels Represents the Gaussian function. These are hyperparameters used to control the sharpness of the generated distribution. It is a regularization factor used to normalize the descriptive power of acne severity labels on the input image;

[0113] The true severity label for acne The corresponding label distribution is , It is the size of the acne severity label set.

[0114] The other steps and parameters are the same as in Specific Implementation Method 1.

[0115] This invention uses the acne severity label distribution as the label corresponding to the acne image, which can take advantage of the label uncertainty information inherent in the acne severity classification task to further improve the classification performance.

[0116] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the regularization factor... for:

[0117]

[0118] in, It is the [number]th in the acne severity label set. Acne severity labels.

[0119] Other steps and parameters are the same as in specific implementation method one or two.

[0120] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the semantic similarity calculation module operates as follows:

[0121] Step S11: Calculate the similarity matrix :

[0122]

[0123] in, It is a similarity matrix The Middle Line 1 Column elements, , = 2, ... , = 2, ... , This represents taking the absolute value. ;

[0124] Step S12: Group the acne severity labels into the first... The label distribution corresponding to each acne severity label is denoted as . , , The real label is the first Images of acne severity levels, grouping acne severity labels into the first... The label distribution corresponding to each acne severity label is denoted as . Then the first The label distribution corresponding to the first acne severity label and the first The semantic similarity of the label distribution corresponding to each acne severity label is as follows:

[0125]

[0126] in, It is the first The label distribution corresponding to the first acne severity label and the first The semantic similarity of the label distribution corresponding to each acne severity label, with the superscript T indicating transposition.

[0127] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0128] In the process of constructing a sequence tree, considering the semantic similarity between the label distributions corresponding to different acne severity levels, a sequence tree can be constructed that progresses from easy to difficult in binary decision-making, thereby alleviating the problem of error propagation in sequence prediction.

[0129] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the working process of the sequence tree module is as follows:

[0130] Step S21: Given a set of leaf nodes and initialize the node set. ;

[0131] Step S22: Select a set of nodes Partition node index :

[0132]

[0133] in, Representatives utilize nodes For sets The ratio of samples in the left subtree to the right subtree after the partitioning;

[0134]

[0135] in, The labels representing the actual acne severity in the training set are: The corresponding number of image samples;

[0136] After the partitioning operation, two new sets of nodes are generated. and ;

[0137] Step S23: Determine whether there exists a set with more than 1 nodes in the node set obtained from step S22.

[0138] If there exists a set with more than 1 node, then return to step S22 for each set with more than 1 node.

[0139] If there is no set with more than 1 node, it means that all generated node sets contain only one node, and the construction of the sequence tree is complete.

[0140] It should be noted that, starting from the second iteration, we only need to partition the nodes in the sets whose left and right subtrees obtained in the previous iteration contain more than one node. Figure 2a Taking the right subtree obtained in the first iteration as an example, at this time the node set of the right subtree only includes the second to fourth nodes;

[0141] When partitioning again, use the nodes The method for calculating the sample ratio of the left subtree and right subtree generated by the partition is as follows:

[0142]

[0143] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0144] Specific Implementation Method Six: Combination Figure 2a This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to five is that the working process of the decision sequence generation module is as follows:

[0145] For any acne severity label According to the sequence tree root node to the label The path of the corresponding child node is used to generate acne severity labels. The corresponding binary decision sequence (in the constructed sequence tree, each node has two possible paths: the left subtree and the right subtree, represented by 0 and 1 respectively).

[0146] The generated binary decision sequence Represented as:

[0147]

[0148] Where E represents the acne severity rating conversion function, and H represents the height of the sequence tree. Acne severity level label The corresponding path encoding sequence from the root node to the leaf node, h = 1, 2, ..., H;

[0149] =0 indicates that the current node in the path is located in the left subtree of the node above it. = This indicates that the current node in the path is located in the right subtree of the node at the next higher level;

[0150] binary decision sequence Shift the elements in the binary decision sequence one position to the right and fill the first position with the start marker to obtain the final binary decision sequence. ;

[0151]

[0152] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0153] This invention transforms the prediction of acne severity grading into a binary decision sequence prediction, which allows for progressive refinement and thus improves grading performance.

[0154] Specific implementation method seven: Combining Figure 2b This embodiment is described below. The difference between this embodiment and one of the specific embodiments one through six is ​​that the operation process of the multi-hotspot encoding module is as follows:

[0155] Acne severity label Label the severity of acne The corresponding node is the th in the sequence tree The multi-hot tag encoding of the layer is denoted as , ,in, and obtain the first In the layer, the label is related to the severity of acne. The corresponding node is a node within the same set; the retrieved node is... The value of the corresponding element is also 1 (after obtaining the node, you can get the acne severity label corresponding to the node, and then set the value of the element corresponding to these acne severity labels to 1). The values ​​of all other elements in the array are 0;

[0156] Similarly, obtain acne severity labels After the corresponding nodes are encoded with multi-hot labels at each level of the sequence tree, the acne severity label is obtained. Multi-hot tag encoding sequence ;

[0157]

[0158] Similarly, the multi-hot tag encoding sequence for each acne severity label is obtained.

[0159] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0160] Transforming the prediction of binary decision sequences into the prediction of multi-hot-label encoded sequences allows the model to utilize more information during the prediction process, thereby achieving better performance.

[0161] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One through Seven in that the label distribution branch and the multi-hot label coding branch are jointly trained. The specific process of joint training is as follows:

[0162] Step S31: After preprocessing the input image, input the preprocessed image into a feature extraction network (the feature extraction network that can be used includes, but is not limited to, the PVTv2 network) to extract image features;

[0163] Step S32: Input the image features extracted in step S31 into the label distribution branch. In the label distribution branch, the extracted image features first pass through a fully connected layer for feature dimensionality reduction, and then the dimensionality-reduced features pass through a Softmax layer to obtain the predicted label distribution. ;

[0164] Calculate the loss between the predicted label distribution and the label distribution generated by the label distribution generation module. :

[0165]

[0166] in, It is the distribution of the true labels corresponding to the input image. express and The cumulative JS divergence (CJS);

[0167] The cumulative JS divergence is calculated as follows:

[0168]

[0169] in, It is the cumulative density function. It is JS divergence. , It is the distribution of labels The Middle The values ​​corresponding to the severity labels of acne. , It is the distribution of labels The Middle The value corresponding to each acne severity label;

[0170]

[0171] in, ;

[0172] Step S33: Input the image features extracted in step S31 into the multi-hot label coding branch. In the multi-hot label coding branch, the extracted image features are first encoded by the Transformer encoder to obtain the encoding result; then, based on the encoding result and the final binary decision sequence, the multi-hot label codes predicted in each step are obtained. , t=1,2,…,H;

[0173] Calculate the loss between the predicted multi-hot tag encoded sequence and the multi-hot tag encoded sequence generated by the multi-hot tag encoding module. ;

[0174]

[0175] in, It is a binary cross-entropy function. express The first in One element, express The first in One element, = ;

[0176] Step S34 combines the losses from steps S32 and S33, and then backpropagates the joint loss to train the model; whereby the joint loss... for: ;

[0177] in, It is a balance loss and The hyperparameters of the weights.

[0178] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0179] This implementation is only for a single input image sample; the process of this implementation is performed for each image sample in the training set.

[0180] Specific Implementation Method Nine: Combining Figure 3 and Figure 4 This embodiment describes a method that differs from specific embodiments one through eight in that the step of obtaining the multi-hot tag codes for each step based on the encoding results and the final binary decision sequence is performed differently. Specifically:

[0181] Step S331: Initialize the number of steps t=1;

[0182] The encoding result and the final binary decision sequence are input into the selective mask decision decoder. The working process within the selective mask decision decoder is as follows:

[0183] Step S331: Initialize the number of steps t=1;

[0184] Step S332: Encode the position of the t-th element in the binary decision sequence corresponding to the input image;

[0185] Step S333: Input the position encoding result and the image features after passing through the Transformer encoder into the Transformer decoder to obtain the decoder output. ;

[0186] Step S334 After passing through the Sigmoid layer, the predicted probability sequence is obtained. ;

[0187] Step S335: Based on the probability sequence Generate mask sequence Specifically:

[0188] Based on the node partitioning results of the t-th level of the sequence tree, the probability sequence is... The elements in the probability sequence are divided into sets (the number of elements in the probability sequence is the same as the size of the acne severity label set, i.e., the same as the number of nodes. For example, in the first layer, if the second to fourth nodes are in one set, then the second to fourth elements in the probability sequence are divided into one set). In the partitioning results, for sets containing only one element, the value of the element is directly used as the value of the set; for sets containing two or more elements, the average value of all elements in the set needs to be calculated, i.e., after obtaining the average value of each set, the average value is used as the value of the corresponding set.

[0189] That is, for each set, calculate the average of the elements in the set, and the average value is the value of the corresponding set;

[0190] The values ​​of each partitioned set are compared, and the set corresponding to the largest value is selected. This yields the acne severity label corresponding to the element in the selected set. The obtained acne severity label is then used in the mask sequence. The element corresponding to the given value is 1, and all other elements are 0.

[0191] It should be noted that the mask sequence The number of elements in the set is the same as the size of the acne severity label set. For example, the acne severity label corresponding to each element in the selected set is... , and Then the mask sequence The second, third, and fourth elements are 1;

[0192] Step S336: If t=1, then the multi-hot tag code predicted in step t is... = ,make It is a sequence of all 1s;

[0193] If t > 1, then determine and Whether the intersection of the two sets is an empty set; that is, determine whether the intersection of the two sets is an empty set. and Check if the values ​​of the elements at corresponding positions are equal. If there are equal elements, then the set is not empty; otherwise, the set is empty.

[0194] If it is not an empty set, then the multi-hot tag encoding predicted at step t is... = ;

[0195] This can suppress the interference of categories that have been removed in the previous time step on the loss of the current time step;

[0196] If it is an empty set, then the multi-hot tag encoding predicted in step t is... = ;

[0197] Step S337: Determine whether the number of steps t has reached H;

[0198] If the number of steps t reaches H, then the iteration ends;

[0199] If step t has not reached H, then let t = t + 1 and return to execute step S332.

[0200] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0201] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that the method of using the trained label distribution branch and multi-hot label encoding branch to classify the severity of acne in the image to be classified is as follows:

[0202] Step S41: Crop, rotate, and standardize the image to be graded to obtain the processed image;

[0203] Step S42: After extracting features from the processed image, input the extracted image features into the trained label distribution branch to obtain the predicted label distribution. ;

[0204] Step S43: Input the extracted image features into the multi-hot label coding branch, and then combine the binary decision results from each step. Obtain multi-hot tag encodings for each step of the prediction ;

[0205] Specifically, this refers to binary decision sequences. The first element is positionally encoded, and the positionally encoded result, along with the image features after passing through the Transformer encoder, is input into the Transformer decoder to obtain the decoder's output. And then according to Obtain the multi-hot tag code predicted in step 1. The prediction method is the same as the training process;

[0206] by Figure 2a For example, the partitioning result of the first iteration is: the left subtree includes node 1, and the right subtree includes nodes 2-4. Then, the following is calculated: The mean of the second to fourth elements in the middle is then compared with the calculated mean. If the mean is greater than the value of the first element, then... The value is 1, otherwise The value is 0;

[0207] First, compare the sets with the largest average values. If the set with the largest average value is located on the right branch of the previous level, then... The value is 1, otherwise The value is 0;

[0208] Again Position encoding is performed, and the position encoding result, along with the image features after passing through the Transformer encoder, is input into the Transformer decoder to obtain the decoder output. And then according to Obtain the multi-hot tag code predicted in step 2. ;

[0209] by Figure 2a For example, the second iteration partitions the data into three sets. The average value of elements within each set is compared, and the set with the largest average value is selected. If the largest average value corresponds to the left subtree of the node at the next higher level, then... The value is 0, otherwise The value is 1; it should be noted that if the maximum average value has no branches relative to the previous level node, that is, if there is only one node at that position in the previous level, then The value is 0.

[0210] And so on; finally That is , … The sequence formed, That is , … The sequence formed.

[0211] Step S44, according to And the final step of predicting multi-hot tag encoding The acne severity grading results for the images to be graded were obtained. :

[0212]

[0213] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0214] According to this implementation method, an acne severity index can be obtained, and the obtained index is the acne severity classification result.

[0215] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An automatic acne severity grading system based on multi-task learning, characterized in that, The system includes a label distribution generation module, a semantic similarity calculation module, a sequence tree module, a decision sequence generation module, a multi-hot label encoding module, and a neural network module; wherein: The label distribution generation module is used to convert the real acne severity labels of the input image into a discrete label distribution; The working process of the label distribution generation module is as follows: Acne severity is divided into Z levels, and the labels for the Z levels of acne severity are denoted as follows: , … ,Label , … Together they form a set of acne severity labels; For the input image Input image The true severity of acne is labeled as follows: Tags are generated based on a Gaussian function and a set of acne severity labels. The label distribution, where the first Acne severity label For the input image The descriptive degree is: in, It is the [number]th in the acne severity label set. Acne severity labels It's a hyperparameter. It is a regularization factor; in, It is the [number]th in the acne severity label set. Individual acne severity labels; The true severity label for acne The corresponding label distribution is , It is the size of the acne severity label set; The semantic similarity calculation module is used to calculate the semantic similarity of the label distribution corresponding to adjacent acne severity labels; The semantic similarity calculation module works as follows: Step S11: Calculate the similarity matrix : in, It is a similarity matrix The Middle Line number Column elements, , = 2, ... , = 2, ... , This represents taking the absolute value. ; Step S12: Group the acne severity labels into the first... The label distribution corresponding to each acne severity label is denoted as . The severity of acne is categorized into the first category. The label distribution corresponding to each acne severity label is denoted as . Then the first The label distribution corresponding to the first acne severity label and the first The semantic similarity of the label distribution corresponding to each acne severity label is as follows: in, It is the first The label distribution corresponding to the first acne severity label and the first The semantic similarity of the label distribution corresponding to each acne severity label, with the superscript T indicating transposition; The sequence tree module is used to construct a sequence tree based on the semantic similarity of the distribution of labels corresponding to adjacent acne severity labels; The sequence tree module works as follows: Step S21: Given a set of leaf nodes and initialize the node set. ; Step S22: Select a set of nodes Partition node index : in, Representatives utilize nodes For sets The ratio of samples in the left subtree to the right subtree after the partitioning; in, The labels representing the actual acne severity in the training set are: The corresponding number of image samples; After the partitioning operation, two new sets of nodes are generated. and ; Step S23: Determine whether there exists a set with more than 1 nodes in the node set obtained from step S22. If there exists a set with more than 1 node, then return to step S22 for each set with more than 1 node. If there is no set with more than 1 node, the sequence tree construction is complete. The decision sequence generation module is used to convert each acne severity label into a binary decision sequence based on the sequence tree; The working process of the decision sequence generation module is as follows: For any acne severity label According to the sequence tree root node to the label The path of the corresponding child node is used to generate acne severity labels. The corresponding binary decision sequence; The generated binary decision sequence Represented as: Where H represents the height of the sequence tree, Acne severity level label The corresponding path encoding sequence from the root node to the leaf node, h = 1, 2, ..., H; binary decision sequence Shift the elements in the binary decision sequence one position to the right and fill the first position with the start marker to obtain the final binary decision sequence. ; The multi-hot tag encoding module is used to generate multi-hot tag encodings for each acne severity tag based on the sequence tree; The neural network module includes a label distribution branch and a multi-hot label encoding branch. The label distribution branch is trained using the output of the label distribution generation module, and the multi-hot label encoding branch is trained using the output of the decision sequence generation module and the output of the multi-hot label encoding module. The trained label distribution branch and multi-hot label encoding branch are used to classify the severity of acne in the images to be graded.

2. The automatic acne severity grading system based on multi-task learning according to claim 1, characterized in that, The working process of the multi-hot tag encoding module is as follows: Acne severity label Label the severity of acne The corresponding node is the th in the sequence tree The multi-hot tag encoding of the layer is denoted as , ,in, and obtain the first In the layer, the label is related to the severity of acne. The corresponding node is a node within the same set; the retrieved node is... The value of the corresponding element is also 1. The values ​​of all other elements in the array are 0; Similarly, obtain acne severity labels After the corresponding nodes are encoded with multi-hot labels at each level of the sequence tree, the acne severity label is obtained. Multi-hot tag encoding sequence ; Similarly, the multi-hot tag encoding sequence for each acne severity label is obtained.

3. The automatic acne severity grading system based on multi-task learning according to claim 2, characterized in that, The label distribution branch and the multi-hot label encoding branch are jointly trained. The specific process of joint training is as follows: Step S31: After preprocessing the input image, input the preprocessed image into the feature extraction network to extract image features; Step S32: Input the image features extracted in step S31 into the label distribution branch. In the label distribution branch, the extracted image features first pass through a fully connected layer for feature dimensionality reduction, and then the dimensionality-reduced features pass through a Softmax layer to obtain the predicted label distribution. ; Calculate the loss between the predicted label distribution and the label distribution generated by the label distribution generation module. : in, It is the distribution of the true labels corresponding to the input image. express and The cumulative JS divergence; The cumulative JS divergence is calculated as follows: in, It is the cumulative density function. It is JS divergence. , It is the distribution of labels The Middle The values ​​corresponding to the severity labels of acne. , It is the distribution of labels The Middle The value corresponding to each acne severity label; in, ; Step S33: Input the image features extracted in step S31 into the multi-hot label coding branch. In the multi-hot label coding branch, the extracted image features are first encoded by the Transformer encoder to obtain the encoding result; then, based on the encoding result and the final binary decision sequence, the multi-hot label codes predicted in each step are obtained. , t=1,2,…,H; Calculate the loss between the predicted multi-hot tag encoded sequence and the multi-hot tag encoded sequence generated by the multi-hot tag encoding module. ; in, It is a binary cross-entropy function. express The first in One element, express The first in One element, = ; Step S34 combines the losses from steps S32 and S33, and then backpropagates the joint loss to train the model; whereby the joint loss... for: ; in, It is a balance loss and The hyperparameters of the weights.

4. The automatic acne severity grading system based on multi-task learning according to claim 3, characterized in that, The multi-hot tag encoding for each step is obtained based on the encoding results and the final binary decision sequence. Specifically: Step S331: Initialize the number of steps t=1; The encoding result and the final binary decision sequence are input into the selective mask decision decoder. The working process within the selective mask decision decoder is as follows: Step S331: Initialize the number of steps t=1; Step S332: Encode the position of the t-th element in the binary decision sequence corresponding to the input image; Step S333: Input the position encoding result and the image features after passing through the Transformer encoder into the Transformer decoder to obtain the decoder output. ; Step S334 After passing through the Sigmoid layer, the predicted probability sequence is obtained. ; Step S335: Based on the probability sequence Generate mask sequence ; Specifically: Based on the node partitioning results of the t-th level of the sequence tree, the probability sequence is... The elements in the data are divided into sets. In the partitioning results, for a set containing only one element, the value of the element is directly used as the value of the set. For a set containing two or more elements, the average value of all elements in the set needs to be calculated. That is, after obtaining the average value of each set, the average value is used as the value of the corresponding set. The values ​​of each partitioned set are compared, and the set corresponding to the largest value is selected. This yields the acne severity label corresponding to the element in the selected set. The obtained acne severity label is then used in the mask sequence. The element corresponding to the given value is 1, and all other elements are 0. Step S336: If t=1, then the multi-hot tag code predicted in step t is... = ; If t > 1, then determine and Is the intersection of the two sets an empty set? If it is not an empty set, then the multi-hot tag encoding predicted at step t is... = ; If it is an empty set, then the multi-hot tag encoding predicted in step t is... = ; Step S337: Determine whether the number of steps t has reached H; If the number of steps t reaches H, then the iteration ends; If step t has not reached H, then let t = t + 1 and return to execute step S332.

5. The automatic acne severity grading system based on multi-task learning according to claim 4, characterized in that, The method of classifying acne severity in the image to be classified using the trained label distribution branch and multi-hot label encoding branch is as follows: Step S41: Crop, rotate, and standardize the image to be graded to obtain the processed image; Step S42: After extracting features from the processed image, input the extracted image features into the trained label distribution branch to obtain the predicted label distribution. ; Step S43: Input the extracted image features into the multi-hot label coding branch, and then combine the binary decision results from each step. Obtain multi-hot tag encodings for each step of the prediction ; Step S44, according to And the final step of predicting multi-hot tag encoding The acne severity grading results for the images to be graded were obtained. : 。

Images