A skin disease recognition method based on hierarchical feature memory learning
Through the method based on hierarchical feature memory learning, combined with time attention and hierarchical attention mechanism, the problem of insufficient feature extraction ability in skin disease image processing in the prior art is solved, and accurate feature capture and diagnosis of skin disease images is achieved. Increased rate.
Patent Information
- Application Number
- CN202411423796.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-10-12
AI Technical Summary
The existing neural network models have limited feature extraction capabilities in skin disease image processing, are susceptible to background noise interference, and are difficult to effectively process diverse and complex skin disease data, resulting in low diagnostic accuracy.
Using a hierarchical feature memory learning method, the feature learning module, feature storage module and time-hierarchical feature processing module are combined with time attention and hierarchical attention mechanism to extract features in image data, and introduce adaptive reweighted losses and Cross-entropy loss function, dynamically enhances attention to a few categories, and builds identification models.
Significantly improve the model's feature capture ability of skin disease images, enhance the understanding of time series data and network hierarchy, dynamically balance the judgment of different categories, multiple performance indicators are better than existing classification methods, and improve the accuracy of diagnosis .
Smart Images

Figure CN118941833B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a skin disease recognition method based on hierarchical feature memory learning. Background Art
[0002] Due to the wide variety of skin diseases currently available, as well as problems such as inconsistent identification standards, a relatively complicated diagnostic process, and reliance on experienced doctors, image processing technology is applied to the auxiliary diagnosis of skin diseases to reduce the misdiagnosis of skin diseases due to observation errors or lack of experience, greatly improving the accuracy of skin disease diagnosis.
[0003] In recent years, computer-aided diagnosis technology has been increasingly used to analyze and process skin-related images. For example, traditional neural network models are used for processing, but this method has limited feature extraction capabilities and is easily interfered by background noise. Existing neural network models cannot meet the requirements of skin disease image data processing. Summary of the invention
[0004] The present invention aims at solving the problems in the prior art and provides a skin disease recognition method based on hierarchical feature memory learning.
[0005] The technical solution adopted by the present invention is:
[0006] A skin disease recognition method based on hierarchical feature memory learning comprises the following steps:
[0007] Step 1: Obtain skin disease image data and preprocess the data to obtain a data set;
[0008] Step 2: Construct a recognition model based on hierarchical feature memory learning;
[0009] The recognition model includes a feature learning module, a feature storage module, and a time-level feature processing module;
[0010] Feature learning module, used to extract features from image data;
[0011] The feature storage module is used to obtain each layer of data in the process of feature learning module processing, and obtain feature data including time dimension, layer dimension and feature size dimension. M ;
[0012] The time-level feature processing module is used to process the time dimension data in the feature data based on the time attention mechanism to obtain the time attention output data. IT , the layer dimension data in the feature data is processed based on the layer attention mechanism to obtain the layer attention output TO ;right IT , TO andM Add and get the output result;
[0013] Step 3: Train the recognition model to obtain a trained model; input the image data into the trained model to obtain the required recognition result.
[0014] Furthermore, the temporal attention output IT The acquisition process is as follows:
[0015] For feature data M Obtained by substitution M E , after three linear layers, the corresponding data Q is obtained e , K e and V e ;
[0016] Where Q e , K e After the tensor product is calculated and processed by the normalization function, it is equal to V e Calculate the tensor product; after linear processing; introduce the weight factor after permutation to calculate the tensor product to get the output IT .
[0017] Furthermore, the hierarchical attention output TO The acquisition process is as follows:
[0018] For feature data M Replace M L , after three linear layers, the corresponding data Q is obtained l , K l and V l ;
[0019] Where Q l , K l After the tensor product is calculated and processed by the normalization function, it is equal to V l Calculate the tensor product; after linear processing; introduce the weight factor after permutation to calculate the tensor product to get the output TO .
[0020] Furthermore, the temporal attention output IT The acquisition process is as follows:
[0021]
[0022] Where: is a linear layer, P e For replacement, is the weight factor; , , For feature data MThe eigenvectors of the permuted three-dimensional tensor data after processing by three linear layers; T represents transpose, d k is the hidden layer dimension, and softmax is the normalization function.
[0023] Furthermore, the hierarchical attention output TO The acquisition process is as follows:
[0024]
[0025] Where: is a linear layer, P l For replacement, is the weight factor; , , For feature data M The eigenvectors of the permuted three-dimensional tensor data after processing by three linear layers; T represents transpose, d k is the hidden layer dimension, and softmax is the normalization function.
[0026] Furthermore, the loss function in the training process of step 3 includes the cross entropy loss generated by the feature storage module and the adaptive reweighted loss generated by the time-hierarchical feature processing module;
[0027]
[0028] Where: L is the loss function, L AR is the adaptive reweighted loss function, L M is the cross entropy loss, is a hyperparameter.
[0029] Furthermore, the cross entropy loss is as follows:
[0030]
[0031] Where: c is the sample size, i is the sample serial number, y i is the true probability distribution, The first i Output probability distribution of samples;
[0032] The adaptive reweighted loss is as follows:
[0033]
[0034] Where: is a hyperparameter, is the cumulative learning coefficient, is the clustering weight, is the sample counting weight, C is the number of categories after clustering, j is the category number, is the predicted probability of the model.
[0035] Furthermore, the output of each layer in the feature learning module in step 2 is processed by the global average pooling layer and the fully connected layer to obtain processed data, which is then stored in the feature storage module.
[0036] The beneficial effects of the present invention are:
[0037] (1) The present invention constructs a time-hierarchical feature processing module, which can deeply explore the complex relationship between features through time attention and hierarchical attention, and enhance the model's ability to understand time series data and network hierarchical structure;
[0038] (2) The present invention relates to dynamically enhancing attention to minority categories based on adaptive reweighted loss function and cross entropy loss function while taking into account visual features and sample quantity;
[0039] (3) The present invention can accurately capture image features and is superior to existing classification methods in multiple performance indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a structural schematic diagram of the recognition model based on hierarchical feature memory learning of the present invention.
[0041] Figure 2 This is the structure of the time-level feature processing module in the present invention.
[0042] Figure 3 This is a visualized heat map obtained by using the algorithm of the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0044] A skin disease recognition method based on hierarchical feature memory learning comprises the following steps:
[0045] Step 1: Obtain skin disease image data and preprocess the data to obtain a data set;
[0046] Step 2: Construct a recognition model based on hierarchical feature memory learning; the structure is as follows Figure 1shown.
[0047] The recognition model includes a feature learning module, a feature storage module, and a time-level feature processing module;
[0048] The feature learning module is used to extract features from image data; it can use the ResNet network model or other network models as long as it can meet the requirements. The output of each layer is processed by the global average pooling layer and the fully connected layer to obtain the processed data, which is stored in the feature storage module.
[0049] The feature storage module is used to obtain each layer of data in the process of feature learning module processing, and obtain feature data including time dimension, layer dimension and feature size dimension. M ; ,in N e is the duration number, i.e. the time dimension data, N l is the number of layers, i.e., hierarchical dimension data, N f is the feature number, i.e., the feature dimension data, R For data collection.
[0050] Temporal layer Transformer (TLFormer) is a temporal layer feature processing module with the following structure: Figure 2 As shown, it is used to process the time dimension data in the feature data based on the time attention mechanism to obtain the time attention output data IT , the layer dimension data in the feature data is processed based on the layer attention mechanism to obtain the layer attention output TO ;right IT , TO and M Add and get the output result;
[0051]
[0052] Where: y m It is the final output representation of the recognition model.
[0053] Temporal attention focuses on the mutual influence of features at different time points, while hierarchical attention focuses on the interaction between features at different levels of the network. The combination of the two attention mechanisms enables the model to capture key information in the image more accurately.
[0054] Temporal Attention Output IT The acquisition process is as follows:
[0055] For feature data M The time dimension data in is replaced byM E , After three linear layers, the corresponding data Q is obtained e , K e and V e ;
[0056] Where Q e , K e After the tensor product is calculated and processed by the normalization function, it is equal to V e Find the tensor product; after linear processing;
[0057] After permutation, the weight factor is introduced to obtain the tensor product to get the output IT .
[0058] Temporal Attention Output IT The acquisition process is as follows:
[0059]
[0060] Where: is a linear layer, P e For replacement, is the weight factor; , , For feature data M The eigenvectors of the permuted three-dimensional tensor data after processing by three linear layers; T represents transpose, d k is the hidden layer dimension, and softmax is the normalization function.
[0061] Hierarchical Attention Output TO The acquisition process is as follows:
[0062] For feature data M The hierarchical dimension data in is replaced by M L , After three linear layers, the corresponding data Q is obtained l , K l and V l ;
[0063] Where Q l , K l After the tensor product is calculated and processed by the normalization function, it is equal to V l Calculate the tensor product; after linear processing; introduce the weight factor after permutation to calculate the tensor product to get the output TO .
[0064] The specific acquisition process is as follows:
[0065]
[0066] Where: is a linear layer, P e For replacement, is the weight factor; , , For feature data M The eigenvectors of the permuted three-dimensional tensor data after processing by three linear layers; T represents transpose, d k is the hidden layer dimension, and softmax is the normalization function.
[0067] Step 3: Train the recognition model to obtain a trained model; input the image data into the trained model to obtain the required recognition result.
[0068] To solve the problem of data imbalance, we introduce a new loss function that dynamically enhances the focus on minority classes through a weighted focal loss mechanism based on image appearance and sample number. The loss function designed by this invention considers both the visual features of the image and the number of samples in each mini-batch, effectively solving the challenges brought by unbalanced datasets.
[0069] A two-stage cumulative learning method is used to further improve the performance of the model. In the first stage, the model is trained on an unbalanced dataset (a dataset is divided according to the number of data in the dataset, and the number of data in each category is different, which is an unbalanced dataset. Most of the datasets in this field are unbalanced datasets) to initially learn the skewed distribution of the data. As the training progresses, the model will smoothly transition to a balanced dataset, thereby ensuring its ability to effectively generalize in various data distributions.
[0070] To understand the distribution of data, it is necessary to understand the similarity or distance between data samples. Extracting image feature embeddings in a low-dimensional space to calculate the distance between image frames is a key step in this process. These features are then clustered; the relative size of each cluster provides the probability of a sample appearing in a specific cluster in the dataset. Subsequently, the relative likelihood of each cluster is calculated compared to the largest cluster in the dataset. The weights of the clusters are calculated as follows:
[0071]
[0072] Where: For the j clusters, For the j The number of samples for each cluster, and are all hyperparameters,P It is the same for gatherings.
[0073] In order to achieve weighting based on the number of samples, the weighting coefficient is dynamically adjusted according to the number of each category in each mini-batch. The weights based on the sample technology are as follows:
[0074]
[0075] Where: C For categories, N c For each mini-batch c The number of samples in the class.
[0076] Introducing cumulative learning, the adaptive reweighted loss is as follows:
[0077]
[0078] Where: is a hyperparameter, is the cumulative learning coefficient, is the clustering weight, is the sample counting weight, C is the number of categories after clustering, j is the category number, is the predicted probability of the model.
[0079] in
[0080]
[0081] Where: E 1 and E 2 is the hyperparameter threshold of the epoch during training. The value will gradually change from 0 to , and then the weighting factor will gradually increase from 1.
[0082] During the specific training process, integrating the adaptive reweighted loss with the cross entropy loss of the feature storage module can enhance the model's ability to judge different categories.
[0083]
[0084] Where: L is the loss function, L AR is the adaptive reweighted loss function, L M is the cross entropy loss, is a hyperparameter.
[0085] The cross entropy loss is as follows:
[0086]
[0087] Where: c is the sample size, i is the sample serial number, y i is the true probability distribution, The first i Output probability distribution of samples.
[0088] Example
[0089] In order to illustrate the performance of the recognition model obtained by the present invention, experiments were conducted on the skin disease benchmark dataset ISIC-2017, and the results were presented in the form of two binary classification tasks, namely melanoma classification and seborrheic keratosis classification. In order to quantitatively evaluate the classification performance of the proposed method, accuracy (ACC), sensitivity (SE), specificity (SP), average precision (AP) and Area Under the ROC Curve (AUC) were used as performance indicators to compare the recognition model of the present invention with the existing classification methods. Among them, the average AUC of the two subtasks is the target indicator.
[0090] The existing classification methods are selected from the following methods: ARL-CNN, SSAC, SDL, Taxonomies, MWNL-CLS, DermaKNet, RENI, Alea-Jacta-Est, EResNet, DLSL. The results are shown in Table 1.
[0091] Table 1. Performance comparison between the proposed model and existing methods
[0092]
[0093] As can be seen from Table 1, the method of the present invention achieves the best performance in terms of AUC, AP, ACC in melanoma classification, AUC, AP, ACC in seborrheic keratosis classification, and the average AUC of the two subtasks.
[0094] Figure 3 This is a visualized heat map of the method of the present invention, from which we can see the attention paid by the method of the present invention to different areas of the input image during the classification process. The method of the present invention can well focus on the key areas of the lesion and realize the accurate capture and analysis of key features.
[0095] The recognition model constructed by the method of the present invention includes a feature learning module, a feature storage module, and a time-hierarchical feature processing module; wherein the time-hierarchical feature processing module deeply mines the complex relationship between features through time attention and hierarchical attention. This mechanism can enhance the model's ability to understand time series data and network hierarchical structures. In addition, the loss function constructed by the present invention dynamically enhances attention to minority categories by simultaneously considering the visual features and sample quantity of the image. The method of the present invention has been shown to be superior to existing classification methods in multiple performance indicators through experiments, demonstrating its superior performance.
Claims
1. A skin disease recognition method based on hierarchical feature memory learning, characterized in that: The following steps are involved: Step 1: Obtain skin disease image data and preprocess the data to obtain a data set; Step 2: Construct a recognition model based on hierarchical feature memory learning; The recognition model includes a feature learning module, a feature storage module, and a time-level feature processing module; Feature learning module, used to extract features from image data; The feature storage module is used to obtain each layer of data in the process of feature learning module processing, and obtain feature data including time dimension, layer dimension and feature size dimension. M ; The time-level feature processing module is used to process the time dimension data in the feature data based on the time attention mechanism to obtain the time attention output data. EA , the layer dimension data in the feature data is processed based on the layer attention mechanism to obtain the layer attention output LA ; right EA , LA and M Add and get the output result; Step 3: Train the recognition model to obtain a trained model; input the image data into the trained model to obtain the required recognition result; The loss function during training includes the cross entropy loss generated by the feature storage module and the adaptive reweighted loss generated by the time-hierarchical feature processing module; Where: L is the loss function, L AR is the adaptive reweighted loss function, L M is the cross entropy loss, is a hyperparameter; The cross entropy loss is as follows: Where: c is the sample size, i is the sample serial number, y i is the true probability distribution, The first i Output probability distribution of samples; The adaptive reweighted loss is as follows: Where: is a hyperparameter, is the cumulative learning coefficient, is the clustering weight, is the sample counting weight, C is the number of categories after clustering, j is the category number, is the predicted probability of the model; in, Where: For the j clusters, For the j The number of samples for each cluster, and are all hyperparameters, P For gathering similarity; Where: C For categories, N c For each mini-batch c The number of samples in the class.
2. A skin disease recognition method based on hierarchical feature memory learning according to claim 1, characterized in that: The temporal attention output EA The acquisition process is as follows: For feature data M Obtained by substitution M E , after three linear layers, the corresponding data Q is obtained e , K e and V e ; Where Q e , K e After the tensor product is calculated and processed by the normalization function, it is equal to V e Find the tensor product; After linear processing; after permutation, the weight factor is introduced to calculate the tensor product to get the output EA .
3. The skin disease recognition method based on hierarchical feature memory learning according to claim 1 is characterized in that: The hierarchical attention output LA The acquisition process is as follows: For feature data M Replace M L , after three linear layers, the corresponding data Q is obtained l , K l and V l ; Where Q l , K l After the tensor product is calculated and processed by the normalization function, it is equal to V l Find the tensor product; After linear processing; after permutation, the weight factor is introduced to calculate the tensor product to get the output LA .
4. The skin disease recognition method based on hierarchical feature memory learning according to claim 2 is characterized in that: The temporal attention output EA The acquisition process is as follows: Where: is a linear layer, P e For replacement, is the weight factor; , , For feature data M The eigenvectors of the permuted three-dimensional tensor data after processing by three linear layers; T represents transpose, d k is the hidden layer dimension, and softmax is the normalization function.
5. The skin disease recognition method based on hierarchical feature memory learning according to claim 3 is characterized in that: The hierarchical attention output LA The acquisition process is as follows: Where: is a linear layer, P l For replacement, is the weight factor; , , For feature data M The eigenvectors of the permuted three-dimensional tensor data after processing by three linear layers; T represents transpose, d k is the hidden layer dimension, and softmax is the normalization function.
6. The skin disease recognition method based on hierarchical feature memory learning according to claim 1, characterized in that: The output of each layer in the feature learning module in step 2 is processed by the global average pooling layer and the fully connected layer to obtain processed data, which is then stored in the feature storage module.
