Skin disease classification and detection method, system and equipment based on multi-modal data

Abstract

The invention discloses a skin disease classification and detection method, system and equipment based on multi-modal data. Firstly, skin images of all parts of a patient are collected; the skin images of all the parts are classified through a pre-trained skin classification model, and the skin classification model comprises a CNN convolution model and a DeiT model; if the skin image does not belong to the healthy image, acquiring a skin electric signal of the patient; performing feature extraction on the skin electric signals to obtain an electric signal feature set; analyzing according to the electric signal feature set to determine the disease severity of the patient. According to the method, the skin diseases are classified based on the skin images through the CNN and the DeiT, and the skin diseases can be classified quickly and accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of skin disease diagnosis, and more specifically, to a skin disease classification and detection method, system and device based on multi-modal data. Background Art

[0002] Skin diseases are one of the most common diseases in the world, used to describe abnormal skin tissues. More than three thousand acute or chronic subclasses have been discovered. Although skin diseases are not fatal, the itching and pain caused by this disease will bring a heavy burden to the quality of life of patients.

[0003] At present, the diagnosis of skin diseases is mainly based on the experience of experts. However, dermatologists cannot quickly conduct large-scale skin disease screening and diagnosis. Skin diseases have complex types and diverse manifestations, and are extremely prone to missed diagnosis and misdiagnosis. Facing a complex disease spectrum and a large patient group, how to quickly and accurately classify and detect skin diseases. Summary of the Invention

[0004] In view of this, the present invention provides a skin disease classification and detection method, system and device based on multi-modal data, which can quickly and accurately classify and detect skin diseases based on skin images and skin electrical signals.

[0005] To achieve the above object, the following solutions are proposed:

[0006] A skin disease classification and detection method based on multi-modal data, comprising:

[0007] Collect skin images of various parts of the patient;

[0008] Classify the skin images of various parts through a pre-trained skin classification model, wherein the skin classification model includes a CNN convolutional model and a DeiT model;

[0009] If the skin image does not belong to a healthy image, collect the skin electrical signal of the patient;

[0010] Extract features from the skin electrical signal to obtain an electrical signal feature set;

[0011] Analyze according to the electrical signal feature set to determine the severity of the patient's illness.

[0012] Preferably, the electrical signal feature set includes: skin conductance response, skin conductance level, average value, variance, standard deviation, rise time, kurtosis and skewness.

[0013] Preferably, before extracting features from the skin electrical signal, the method further includes:

[0014] Filter, smooth, and integrate the skin electrical signals in terms of dimensions.

[0015] Preferably, the process of analyzing according to the set of electrical signal features includes:

[0016] Extract the relationship between the features in the set of electrical signal features and the skin disease itching information through 8 types of dimensional algorithms;

[0017] Determine the degree of the patient's illness according to the feature relationship.

[0018] Preferably, before classifying based on each skin image, the method further includes:

[0019] Normalize each skin image through the mean and standard deviation;

[0020] Adjust each skin image to a unified size.

[0021] Preferably, skin disease classification includes: eczema, viral infection, melanoma, atopic dermatitis, basal cell carcinoma, melanocytic nevus, benign keratosis-like lesion, psoriasis, seborrheic keratosis, and fungal infection.

[0022] A skin disease classification and detection system based on multimodal data, which implements the aforementioned skin disease classification and detection method based on multimodal data. The system includes:

[0023] An image acquisition module, used to acquire skin images of various parts of the patient;

[0024] A skin disease classification module, used to classify the skin images of various parts through a pre-trained skin classification model, where the skin classification model includes a CNN convolutional model and a DeiT model;

[0025] An electrical signal acquisition module, used to acquire the patient's skin electrical signals when the skin image does not belong to a healthy image;

[0026] An electrical signal feature extraction module, used to extract features from the skin electrical signals to obtain a set of electrical signal features;

[0027] An electrical signal feature calculation module, used to analyze according to the set of electrical signal features to determine the severity of the patient's illness.

[0028] A skin disease classification and detection device based on multimodal data, including: a memory and a processor;

[0029] The memory is used to store programs;

[0030] The processor is used to execute the program to implement each step of the aforementioned skin disease classification and detection method based on multimodal data.

[0031] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0032] For the skin disease classification and detection method based on multi-modal data provided by the present invention, first, skin images of various parts of the patient are collected; the skin images of various parts are classified by a pre-trained skin classification model, wherein the skin classification model includes a CNN convolutional model and a DeiT model; if the skin image does not belong to a healthy image, the skin electrical signal of the patient is collected; feature extraction is performed on the skin electrical signal to obtain an electrical signal feature set; according to the analysis of the electrical signal feature set, the severity of the patient's illness is determined. The present invention can classify skin diseases quickly and accurately through the classification of skin diseases based on skin images by CNN and DeiT.

[0033] The present invention also introduces the skin electrical signal as a detection index for assisting in judging skin inflammation, and analyzes the degree of illness of the patient through the correlation between the skin electrical signal and the severity of the patient's illness. It can better provide reference for doctors and further improve the efficiency of skin disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0035] Figure 1 It is a flowchart of a skin disease classification and detection method based on multi-modal data provided by an embodiment of the present invention;

[0036] Figure 2 It is a schematic structural diagram of a skin classification model provided by an embodiment of the present invention;

[0037] Figure 3 It is a schematic structural diagram of a skin disease classification and detection system based on multi-modal data provided by an embodiment of the present invention;

[0038] Figure 4 It is a block diagram of the hardware structure of a skin disease classification and detection device based on multi-modal data provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] First, in combination with Figure 1 a skin disease classification and detection method based on multimodal data provided by an embodiment of the present invention will be introduced. As Figure 1 shown, it includes:

[0041] Step S01, collect skin images of various parts of the patient.

[0042] Specifically, obtain the input skin images of various parts of the patient. The skin images of various parts of the patient can be captured by a camera.

[0043] Step S02, classify the skin images of various parts through a pre-trained skin classification model.

[0044] Specifically, as Figure 2 shown, the skin classification model includes a CNN convolutional model and a DeiT model. Among them, the skin classification model based on the CNN and DeiT models includes a feature extraction unit and a classification prediction unit.

[0045] The feature extraction unit repeatedly extracts the local position features of the image through the structure of multi-layer convolution operations, batch normalization operations, and ReLU activation functions, and reduces the computational complexity through max pooling operations and adaptive average pooling. The multi-layer convolution operations can be three groups of 3×3 convolutional layers connected in sequence. Then, the extracted image features are sent into the classification prediction unit for training. The classification prediction unit connects multiple stacked Transformer blocks, normalization operations, multi-head self-attention mechanisms, feed-forward neural networks, and residual blocks to integrate the extracted feature information, and finally realizes the classification judgment of the image through normalization operations and fully connected layers.

[0046] If the skin image is a diseased skin image, then execute the following step S03; if the skin image is a healthy skin image, it proves that there is no disease, and the detection and analysis are stopped.

[0047] Step S03, collect the skin electrical signals of the patient

[0048] Specifically, the skin electrical signals of the diseased position of the patient can be collected through an electrical signal collection device such as an electrode patch.

[0049] Step S04, extract features from the skin electrical signals.

[0050] Specifically, eight characteristic values of the skin conductance signal, including skin conductance response, skin conductance level, mean, variance, standard deviation, rise time, kurtosis and skewness, can be extracted.

[0051] F (skin conductance response) = G pealk -G baseline , where G peak represents the peak value of skin conductance, G baseline represents the baseline skin conductance value.

[0052] Among them, G i represents the skin conductance value at the i-th moment, and N represents the total number of measured time points.

[0053] Among them, x i represents the i-th measurement value, and N represents the total number of measured time points.

[0054] With x i The difference is that G i is the average level over a period of time, and x i It is any point of global data.

[0055] Here, μ represents the mean of the data.

[0056]

[0057] F(rise time) = t peak -t onset , t peak The time point when the skin conductance response reaches its peak, t onset The point in time when the skin conductance response begins to rise.

[0058]

[0059]

[0060] Step S05: performing analysis based on the electrical signal feature set.

[0061] Specifically, the relationship between skin electrical signal features and skin itch information can be determined in advance using skin electrical signal features collected from patients with atopic dermatitis, psoriasis, alopecia areata, and urticaria. An eight-dimensional algorithm is used to extract the relationship between features in the electrical signal feature set and skin itch information. Based on this relationship, the severity of the patient's illness is determined.

[0062] Box plots and violin plots can be used to compare the distribution differences of sympathetic nerve activity-related indicators between chronic patients and healthy controls. By observing the distributions of these indicators in the two groups, it is possible to intuitively understand whether there are abnormalities in sympathetic nerve activity in patients with chronic pruritus. A correlation heat map is used to analyze the internal association patterns of sympathetic nerve indicators in the patient group and the healthy control group. Further, a random forest model is adopted to identify the key physiological characteristics that distinguish chronic pruritus patients from healthy controls. Using disease status (N = control group, P = patient group) as the dependent variable and sympathetic nerve indicators such as SCR (skin conductance response), SCL (skin conductance level), and mean of skin electrophysiological signals as independent variables, a random forest model is trained and the feature importance is calculated. And the model parameters are further explained by calculating Shapley Additive explanation (SHAP) to observe the contribution degree of each feature to a specific prediction. And the principal component analysis method (PCA) is used to reduce the high-dimensional information of the model and simplify the data structure, so as to evaluate the potential of each feature as a biomarker. Through a variety of analysis methods, we seek to explain the differences in sympathetic nerve activity between chronic pruritus patients and non-diseased control groups from different perspectives, in order to reveal the potential physiological mechanisms of chronic pruritus.

[0063] In the embodiment of the present invention, a skin disease classification and detection method based on multimodal data includes collecting skin images of various parts of a patient's body; inputting the patient's skin images into a pre-trained deep learning model based on CNN and DeiT to classify the diseases suffered by the patient; continuing to collect skin electrophysiological signals for patients who meet the criteria of skin inflammation and diseases; and then extracting features from the collected electrophysiological signal data to quantify the degree of skin inflammation of the patient. The skin disease classification model based on CNN and DeiT of the present invention can achieve more accurate skin disease classification tasks. In addition, skin electrophysiological signals are introduced as detection indicators for assisting in judging skin inflammation, which has the advantages of comprehensiveness and objectivity compared with single or visual evaluation methods.

[0064] In addition, in order to enable the model to better classify and predict based on skin images and skin electrical signals, in step S02, before classifying based on each skin image, the following steps can also be performed:

[0065] Normalize each skin image through the mean and standard deviation to ensure that all images have the same brightness and contrast levels.

[0066] Adjust each skin image to a unified size.

[0067] In step S04, before extracting features from the skin electrical signals, the following steps can also be performed:

[0068] Filter the skin electrical signals, smooth the signals, and perform dimensional integration operations to clean the data. Clean the skin electrical signals by extracting the skin electrical signal information at 100 Hz, smoothing it, and integrating it into data groups of 30 s in dimension.

[0069] Next, the training process of the skin disease classification model in the embodiments of the present invention is introduced as follows:

[0070] The first step is data collection and processing:

[0071] Collect images of ten types of skin diseases, namely eczema, viral infection, melanoma, atopic dermatitis, basal cell carcinoma, melanocytic nevus, benign keratosis-like lesion, psoriasis, seborrheic keratosis, and fungal infection. Perform image cropping and image size unification operations on the image data, and finally place them in different folders according to categories. Use the preprocessed image data as training data, and each skin image uses the skin disease type as a label.

[0072] Randomly select 80% of the data as the training set, randomly select 10% of the data as the validation set, and the remaining 10% of the data as the test set; among them, the training set is used for model training; the validation set is used for model selection and hyperparameter tuning; the test set is used for final evaluation of the model performance.

[0073] The second step is model construction and training:

[0074] The overall pipeline of CDeiT is as Figure 2 shown. For the original data, before performing block data feature extraction, first use the CNN feature extraction module to perform preliminary data feature extraction on the data to enhance the local feature extraction ability and improve the global information modeling ability of the Transformer, so as to improve the accuracy and stability of the classification task.

[0075] The CNN module first uses multiple convolutional operations to extract low-level edge features and high-level semantic features, and combines batch normalization to improve the stability of the feature distribution. At the same time, the ReLU activation function is used to introduce non-linear transformation to enhance the expressive power of the model. In addition, the model reduces the computational complexity through max pooling and adaptive average pooling, and compresses the features in the spatial dimension to reduce information redundancy and improve computational efficiency. Through this simple CNN module, the overall feature extraction ability of the model for data can be increased. The extracted features are then input into the Transformer encoder module in the CDeiT model, which is composed of multiple stacked Transformer blocks. Each Transformer block contains layer normalization, multi-head self-attention mechanism, feed-forward neural network and residual connection. Among them, the self-attention mechanism is used to model global information to ensure long-range dependencies between features, while the MLP further enhances the feature expression ability through non-linear mapping. The introduction of residual connection effectively alleviates the problem of gradient disappearance in deep networks, enabling information to be efficiently propagated between layers. After the Transformer calculation is completed, the model undergoes a normalization operation and then enters the fully connected layer for classification prediction.

[0076] Compared with the standard DeiT structure, the skin disease classification model combines the advantages of CNN and Transformer, improves the performance of the classification task while maintaining a low computational cost, and can effectively improve the quality of feature extraction and enhance the generalization ability of the model.

[0077] The third step is model evaluation:

[0078] After the skin classification model is trained, the classification effect is evaluated on an independent test set. The top-1 accuracy, top-3 accuracy, recall rate, F1 value and confusion matrix are used to evaluate the performance of the model on the test set.

[0079] Next, the skin disease classification and detection system based on multi-modal data provided by the embodiments of the present invention will be described. The skin disease classification and detection system based on multi-modal data described below can be correspondingly referred to the skin disease classification and detection method based on multi-modal data described above.

[0080] First, in combination with Figure 3 , the skin disease classification and detection system based on multi-modal data will be introduced. As Figure 3 shown, the skin disease classification and detection system based on multi-modal data may include:

[0081] An image acquisition module 100, configured to acquire skin images of various parts of a patient;

[0082] The skin disease classification module 200 is used to classify skin images of each part through a pre-trained skin classification model, where the skin classification model includes a CNN convolutional model and a DeiT model;

[0083] The electrical signal acquisition module 300 is used to acquire the skin electrical signals of the patient when the skin image does not belong to a healthy image;

[0084] The electrical signal feature extraction module 400 is used to extract features from the skin electrical signals to obtain an electrical signal feature set;

[0085] The electrical signal feature calculation module 500 is used to analyze according to the electrical signal feature set to determine the severity of the patient's illness.

[0086] The skin disease classification and detection system based on multi-modal data provided by the embodiments of the present invention can be applied to skin disease classification and detection devices based on multi-modal data. Figure 4 The hardware structure block diagram of the skin disease classification and detection device based on multi-modal data is shown. Refer to Figure 4 , the hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0087] In the embodiments of the present invention, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete mutual communication through the communication bus 4;

[0088] The processor 1 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0089] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0090] Among them, the memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the foregoing skin disease classification and detection solution based on multi-modal data;

[0091] The embodiments of the present invention also provide a storage medium, which can store a program suitable for execution by a processor, and the program is used to implement each processing flow in the foregoing skin disease classification and detection solution based on multi-modal data.

[0092] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For the same or similar parts among the various embodiments, reference may be made to each other.

[0094] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A skin disease classification and detection method based on multimodal data, characterized in that, Including: Collecting skin images of various parts of the patient; Classifying the skin images of various parts through a pre-trained skin classification model, where the skin classification model includes a CNN convolutional model and a DeiT model; If the skin image does not belong to a healthy image, collecting the patient's skin electrical signal; Extracting features from the skin electrical signal to obtain an electrical signal feature set; Analyzing according to the electrical signal feature set to determine the severity of the patient's illness.

2. The method for classifying and detecting skin diseases based on multi-modal data according to claim 1, wherein, The electrical signal feature set includes: skin conductance response, skin conductance level, mean value, variance, standard deviation, rise time, kurtosis, and skewness.

3. The skin disease classification and detection method based on multimodal data according to claim 1, characterized in that, Before extracting features from the skin electrical signal, the method further includes: Performing filtering, signal smoothing, and dimension integration operations on the skin electrical signal.

4. The method for classifying and detecting skin diseases based on multimodal data according to claim 1, wherein The process of analyzing according to the electrical signal feature set includes: Extracting the relationship between the features in the electrical signal feature set and the skin disease itching information through 8 types of dimension algorithms; Determining the degree of the patient's illness according to the feature relationship.

5. The method for classifying and detecting skin diseases based on multimodal data according to claim 1, wherein Before classifying based on each skin image, the method further includes: Normalizing each skin image through the mean value and standard deviation; Adjusting each skin image to a unified size.

6. The method for classifying and detecting skin diseases based on multi-modal data according to any one of claims 1-5, characterized in that, Skin disease classification includes: eczema, viral infection, melanoma, atopic dermatitis, basal cell carcinoma, melanocytic nevus, benign keratosis-like lesion, psoriasis, seborrheic keratosis, and fungal infection.

7. A skin disease classification and detection system based on multimodal data, characterized in that, Implementing the skin disease classification and detection method based on multi-modal data according to claim 1, the system includes: An image acquisition module for collecting skin images of various parts of the patient; A skin disease classification module for classifying the skin images of various parts through a pre-trained skin classification model, where the skin classification model includes a CNN convolutional model and a DeiT model; An electrical signal acquisition module for collecting the patient's skin electrical signal when the skin image does not belong to a healthy image; An electrical signal feature extraction module for extracting features from the skin electrical signal to obtain an electrical signal feature set; An electrical signal feature calculation module for analyzing according to the electrical signal feature set to determine the severity of the patient's illness.

8. A skin disease classification and detection device based on multimodal data, characterized in that, Including: A memory and a processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the skin disease classification and detection method based on multi-modal data according to any one of claims 1-6.

Images