Children atopic dermatitis severity intelligent evaluation system and computer equipment

Abstract

The invention discloses a child atopic dermatitis severity intelligent evaluation system and computer equipment, and the system comprises a data collection module which is used for collecting a multispectral image, a common image, electronic medical record data and wearable equipment data of a child atopic dermatitis patient; the data registration module is used for registering the multispectral image and the common image through an improved feature point matching algorithm; the data fusion module is used for fusing the registered image with the electronic medical record data and the wearable equipment data; the classification module comprises a deep learning model, the input of an input layer is a multi-modal data fusion feature, and the output of a classification layer is a normal classification result, a mild classification result and a moderate-moderate-severe classification result; and the training module is used for training and evaluating the deep learning model in the classification module by using the multi-modal data set. By utilizing the method, the severity of the atopic dermatitis can be automatically identified and classified, and the method has relatively high accuracy and practicability.

Description

Technical Field

[0001] The present invention relates to the field of medical artificial intelligence, and particularly to an intelligent evaluation system for the severity of childhood atopic dermatitis based on multimodal fusion and a computer device. Background Art

[0002] As a common chronic inflammatory skin disease, accurately evaluating the severity of childhood atopic dermatitis plays a crucial role in the treatment and prognosis of patients. Traditional diagnostic methods mainly rely on visual inspection by dermatologists and clinical experience to judge. However, this evaluation process is highly subjective, extremely vulnerable to human factors, and has significant limitations. Due to the subjectivity of visual inspection, different doctors' judgments on the same lesion may vary greatly. Moreover, the symptoms of early childhood atopic dermatitis are often not typical enough, and it is very difficult to accurately distinguish lesions of different severities solely by visual inspection, thus easily leading to misdiagnosis and missed diagnosis.

[0003] The Chinese patent document with the publication number CN114947756A discloses an intelligent evaluation and decision-making system for the severity of atopic dermatitis based on skin images. The system includes: an identity information management module for creating and managing the identity recognition information of patients; a collection module for intelligently collecting the clinical images of atopic dermatitis of patients and matching the dermatitis clinical images with the identity recognition information; an identification module for segmenting the dermatitis clinical images into multiple human secondary regions, and identifying and obtaining the skin lesion types, skin lesion grades, and skin lesion areas of each human secondary region; an evaluation module connected to the identification module for receiving the health interference indicators input by the patient according to their subjective feelings, and comprehensively scoring according to the health interference indicators, skin lesion types, skin lesion grades, and skin lesion areas to obtain scoring results in multiple dimensions, and sending them to the terminal for visual display.

[0004] The Chinese patent document with the publication number CN119048526A discloses a method for segmenting atopic dermatitis images based on the SAM model, including the following steps: S1, data preparation and preprocessing to achieve high-precision dermatitis image segmentation; S2, fine-tuning the SAM model to adapt to the characteristics of atopic dermatitis image segmentation; S3, using the trained SAM model to segment the dermatitis lesion in the dermatitis image data; S4, evaluating and optimizing the dermatitis image segmentation results.

[0005] Currently, the evaluation of related technologies is mostly limited to a single data source, lacking comprehensive and overall considerations. Although existing multi-modal data fusion technologies have been applied in the fields of medical imaging and pathological diagnosis, their application in the specific field of childhood atopic dermatitis is still scarce. Within the scope of diagnosing childhood atopic dermatitis, effectively utilizing multi-modal data faces numerous challenges. On the one hand, how to efficiently integrate multi-modal data with different sources and characteristics, deeply mine and fully utilize their complementary advantages has become a key problem that urgently needs to be solved. On the other hand, existing diagnostic methods have room for improvement in terms of both accuracy and efficiency when dealing with complex skin lesion images, and it is difficult to truly meet the actual clinical needs.

[0006] Therefore, developing an intelligent detection and diagnosis method that can fuse multi-modal data has important clinical value and practical significance. Summary of the Invention

[0007] The present invention provides an intelligent evaluation system and computer device for the severity of childhood atopic dermatitis based on multi-modal fusion, which can automatically identify and classify the severity of atopic dermatitis, and has high accuracy and practicality.

[0008] An intelligent evaluation system for the severity of childhood atopic dermatitis, comprising:

[0009] A data acquisition module for acquiring multi-spectral images, ordinary images, electronic medical record data, and wearable device data of childhood atopic dermatitis patients;

[0010] A data registration module for registering multi-spectral images and ordinary images through an improved feature point matching algorithm to ensure the spatial consistency of images of different modalities;

[0011] A data fusion module for fusing the registered images with electronic medical record data and wearable device data to form a multi-modal data set composed of multi-modal data fusion features;

[0012] A classification module, including a deep learning model. The deep learning model includes an input layer and a classification layer. The input of the input layer is multi-modal data fusion features, and the output of the classification layer is a three-classification result of normal, mild, and moderate to severe; the input layer contains 3 front-end network modules, each network module contains 2 convolutional layers and 1 pooling layer, and an attention layer is inserted between two network modules; the classification layer includes a Transformer layer, a fully connected layer, and a Dropout layer. The Transformer layer captures long-range dependencies through self-attention mechanisms. The fully connected layer makes the final decision output based on the features extracted and integrated by the previous layers. The Dropout layer randomly discards part of the neuron outputs during the training process to increase the uncertainty of the model;

[0013] A training module that trains and evaluates a deep learning model in a classification module using a multi-modal dataset.

[0014] Furthermore, the electronic medical record data includes structured data and unstructured data of patients, and the wearable device data includes the temperature, humidity, and pH value of the skin lesion area.

[0015] In the data fusion module, the structured data in the electronic medical record data is preprocessed using a multi-layer perceptron, and the unstructured data in the electronic medical record data is preprocessed using natural language processing technology. The preprocessed structured data, unstructured data, registered image data, and wearable device data are fused to form a unified feature vector.

[0016] In the data registration module, through an improved feature point matching algorithm, the feature points of the multi-spectral image and the ordinary image are extracted, and the similarity measure between the feature points is calculated to achieve the precise registration of the two.

[0017] The improved feature point matching algorithm has the following specific working process:

[0018] Feature point detection: Obtain the SIFT feature points of the multi-spectral image I t (x, y) and the ordinary image I r (x, y), and establish a feature point matching set C 1 ∩C 2 ; C 1 represents the set of feature points in the multi-spectral image, and C 2 represents the set of feature points in the ordinary image;

[0019] Macroscopic matching: For the feature point matching set C 1 ∩C 2 , first assume that the sampling based on the uniform kernel function in the multi-spectral image is A i , and the matching feature points in the ordinary image are B i , and the sampling points (A 1 , A 2 , A 3 , ……, A i ) and (B 1 , B 2 , B 3 , ……, B i ) are obtained after sampling. The corresponding sampled gray value arrays are represented as (x 1 , x 2 , x 3 , ……, x i ) and (y 1 , y 2 , y 3 , ……, yi ); then normalize (x 1 , x 2 , x 3 , ……, x i ) and (y 1 , y 2 , y 3 , ……, y i ) so that ‖x‖ = ‖y‖ = 1; define d to represent the line segment correlation:

[0020]

[0021] where x and y represent the normalized sampled point gray level arrays, and n represents the number of sampled points; calculate the correlation scores of the two determined line segments, compare with the threshold T, use the voting method to obtain the scores of each pair of feature point matches, sort them according to the scores from high to low, delete the matches with lower scores, and retain the matches with higher scores;

[0022] Microscopic matching: On the basis of macroscopic matching, for the retained matches with higher scores, randomly select 4 pairs of pairings, and calculate the projection change T of the multispectral image I t (x, y) and the ordinary image I r (x, y); then apply the formula to calculate the global information similarity measure, and perform screening after comparison according to the maximum global similarity measure;

[0023] After multiple rounds of matching of the images, pair the lesion areas in the ordinary image and the multispectral image; when the skin area in the ordinary image shows lesion characteristics, the corresponding area in the multispectral image should present corresponding characteristics; in this way, accurately combine the corresponding lesion areas in the two modalities to form comprehensive skin lesion characteristics, and finally complete image registration.

[0024] During the macroscopic matching process, in order to further improve the probability of correct feature point matching, considering that there are similar feature point distributions around the matching feature points, reconstruct the coordinate system according to the distribution of feature points. For any pair of feature point matches (A, A ′ ), after coordinate constraint, obtain Calculate the coordinate difference of each pair of feature points, and the formula is as follows:

[0025]

[0026] Delete the matches with a distance too large exceeding the threshold D as incorrect matches; the threshold D is related to the number N of input feature point matches, and the relationship is: D = α * N.

[0027] The formula for calculating the global information similarity measure is as follows:

[0028]

[0029] Among them, represents the gradient of the j-th pixel, T represents the operation of performing a projection transformation, represents the degree of coincidence of the edge pixels of the i-th region, S represents the global similarity, and N t represents the number of pixels in each region; represents the number of edge pixels at 0 degrees; represents the number of edge pixels at 45 degrees; represents the number of edge pixels at 90 degrees; represents the number of edge pixels at 135 degrees; I Γ (x, y) represents the T-transformed test image.

[0030] A computer device includes a memory and a processor. The memory stores a computer program. It is characterized in that when the processor executes the computer program, it realizes the functions of the above-mentioned intelligent evaluation system for the severity of childhood atopic dermatitis.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The system of the present invention provides an objective and accurate evaluation method by combining multi-spectral images, conventional images, electronic medical record data, and wearable device data, avoiding human biases in traditional methods; the system can automatically identify and classify the severity of atopic dermatitis, with high accuracy and practicality, and can provide strong auxiliary decision-making support for clinicians.

[0033] 2. The present invention adopts an improved feature point matching algorithm, which combines macro matching and micro matching. The role of macro matching is: 1. Using all available feature points (i.e., the intersection of C1 and C2), the feature points are roughly matched by a simple and fast method, providing a basis for more refined micro matching. 2. Using C1∩C2 in the macro stage can ensure that only the feature points that are definitely present in both images are used to estimate the approximate correspondence between the images, which can enhance the reliability and accuracy of subsequent fine matching. Then, in the micro matching stage, the results of macro matching can be optimized and finely adjusted: 1. Micro matching involves more complex algorithms, such as local search and optimization of feature points, and uses more advanced models to accurately calculate the best correspondence between feature points. 2. At this stage, the feature point pairs that have been determined to correspond through macro matching (i.e., the feature point pairs in C1∩C2) will be analyzed and verified more carefully to ensure that their matching is accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1This is the implementation flowchart of an intelligent evaluation system for the severity of childhood atopic dermatitis according to the present invention.

[0035] Figure 2 This is the structural schematic diagram of the image registration module in the present invention.

[0036] Figure 3 This is the structural schematic diagram of the deep learning model in the present invention. Detailed implementation manners

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0038] As Figure 1 shown, an intelligent evaluation system for the severity of childhood atopic dermatitis mainly includes a data acquisition module, a data registration module, a data fusion module, a classification module, and a training module.

[0039] The data acquisition module is used to collect multispectral images, ordinary images, electronic medical record data, and wearable device data of children with atopic dermatitis.

[0040] The data registration module is used to register the multispectral image with the ordinary image, and ensure the spatial consistency of different modality images through an improved feature point matching algorithm.

[0041] The data fusion module is used to fuse the registered images with the electronic medical record data and wearable device data to form a multimodal data set composed of multimodal data fusion features.

[0042] The classification module includes a deep learning model. The deep learning model includes an input layer and a classification layer. The input of the input layer is the multimodal data fusion feature, and the output of the classification layer is a three-classification result of normal, mild, and moderate to severe.

[0043] The training module uses the multimodal data set to train the deep learning model in the classification module, and uses techniques such as cross-validation for evaluation to ensure the accuracy, sensitivity, and reliability of the model.

[0044] In the data acquisition module, the ordinary image is taken by a mobile phone or a camera, mainly recording the appearance of the skin and the lesion area; the multispectral image is collected by a professional device, used to extract more skin information, including data in bands such as infrared and ultraviolet; the electronic medical record data includes the structured data and unstructured data of the patient, where the structured data includes basic information, chief complaint, etc., and the unstructured data includes examination data such as blood routine and urine routine; the wearable device collects physiological parameters such as skin surface temperature, humidity, and PH of the patient through sensors.

[0045] In the data fusion module, before fusion, the structured data in the electronic medical record data needs to be preprocessed using a multi-layer perceptron, and the unstructured data in the electronic medical record data needs to be preprocessed using natural language processing technology. The preprocessed structured data, unstructured data, registered image data, and wearable device data are fused to form a unified feature vector.

[0046] As Figure 2 shown, the main working process of the image registration module is as follows:

[0047] (1) Feature point detection: The multi-spectral - AD image is represented as I t (x, y), and is used to represent the SIFT feature points obtained by testing. The SIFT descriptor of is represented as Let the AD image be represented as I r (x, y), and is used to represent the SIFT feature points obtained by testing. The SIFT descriptor of is represented as If and satisfy the following formula:

[0048]

[0049] According to the BBF rule, 1 and 2 are a pair of feature point matches in the initial matching set. The feature point matching sets established by the above formula are C 1 and C 2 . The feature point matching set established based on the BBF rule is C 1 ∩C 2 .

[0050] Microscopic matching is to select the feature points in C 1 and C 2 for more refined matching on the basis of macroscopic matching. The intersection of C 1 and C 2 (i.e., their common feature points) is used to calculate the relationship between images and further optimize the matching through global similarity. This intersection is used to ensure the precise alignment of the registered images, especially for the precise combination of the lesion areas.

[0051] (2) Macroscopic matching: First, assume that the sampling based on the uniform kernel function in the multi-spectral - AD image is A i , and the matching feature points in the AD image are B i . After sampling, the sampling points (A 1 , A 2 , A 3 , ……, Ai ), and (B, B 2 , B 3 , ……, B i ), the corresponding sampled grayscale arrays are represented as (x 1 , x 2 , x 3 , ……, x i ) and (y 1 , y 2 , y 3 , ……, y i ). Then, for (x 1 , x 2 , x 3 , ……, x i ) and (y 1 , y 2 , y 3 , ……, y i ), normalization processing is performed so that ‖x‖ = ‖y‖ = 1. Define d to represent the line segment correlation:

[0052]

[0053] where x and y represent the normalized sampled point grayscale arrays, and n represents the number of sampled points. Calculate the correlation scores of the two determined line segments, compare them with the threshold T, use the voting method to obtain the scores of each pair of feature point matches, sort them according to the scores from high to low, delete the lower-scoring matches, and retain the higher-scoring matches.

[0054] To further increase the probability of correct feature point matching, considering that there should be a similar distribution of feature points around the matching feature points, reconstruct the coordinate system according to the distribution of feature points. For any pair of feature point matches (A, A ′ ), after coordinate constraint, we get Calculate the coordinate differences of each pair of feature points. The formula is as follows:

[0055]

[0056] Delete the matches with too large distances as incorrect matches. Among them, the threshold is D, which is related to the number N of input feature point matches, and the relationship is as follows:

[0057] D = α * N

[0058] (3) Microscopic matching: On the basis of macroscopic matching, obtain more accurate feature point matches. For the pair C (i.e., the retained higher-scoring matches), randomly select 4 pairs of matches Calculate the projection transformation T of the image I t (x, y) and I r (x, y) according to the matches. The formula is as follows:

[0059]

[0060] After that, the global information similarity measure is calculated using the following formula:

[0061]

[0062] Where represents the gradient of the j-th pixel, represents the degree of coincidence of the edge pixels of the i-th region, and S represents the global similarity. The maximum global similarity measure obtained is defined and compared for screening.

[0063] After multiple rounds of matching of the images, the lesion regions in the AD image and the multi-spectral - AD image are paired. When the skin region in the AD image shows lesion characteristics, the corresponding region in the multi-spectral - AD image should present corresponding characteristics. In this way, the corresponding lesion regions in the two modalities are accurately combined to form comprehensive skin lesion characteristics, so as to more accurately infer the severity of skin inflammation. Finally, image registration is completed.

[0064] As Figure 3 shown, in the classification module, the deep learning model (BADnet network) mainly includes a convolutional layer, a pooling layer, an attention layer, a Transformer layer, a fully connected layer, and a Dropout layer.

[0065] There are 3 modules at the front end of the network, and each module consists of 2 convolutional layers and 1 pooling layer. The convolutional layer is responsible for initially extracting spatial or temporal features from the input data. It performs a convolution operation on the input data by sliding a window, effectively capturing local dependencies, reducing the complexity of the model through parameter sharing, and having good spatial or temporal information preservation ability, providing a basis for subsequent feature extraction and analysis.

[0066] The pooling layer is then after the convolutional layer. It downsamples the features extracted by the convolutional layer, reducing the spatial dimension of the features, reducing the amount of calculation and the number of parameters, while retaining the main information of the features, enhancing the robustness of the features, and making the model more invariant to changes such as translation and scaling of the input data.

[0067] An attention layer (SENet) is inserted between the convolutions in multiple stages to enhance the network's attention to key features. In relatively complex multi-modal data, the attention layer focuses on the feature regions. By assigning different weights to different features, the network can pay more attention to important feature information, thereby improving the expressiveness and interpretability of the model, and enabling the model to better understand and process complex multi-modal data.

[0068] The Transformer layer captures long-range dependencies through the self-attention mechanism, considers the relationships between all positions in the input sequence, provides better parallel processing capabilities, and can capture long-range dependencies more efficiently compared to traditional structures such as recurrent neural networks, which helps the model better understand the global information and complex patterns in the data.

[0069] The fully connected layer makes the final decision output based on the features extracted and integrated by the previous layers. By learning the non-linear combinations between features, it maps the high-dimensional features extracted by the previous layers to the output space and outputs the prediction result of the severity of childhood atopic dermatitis.

[0070] The Dropout layer randomly "drops" the outputs of some neurons during the training process. In this way, it increases the uncertainty of the model, prevents the model from relying too much on certain specific neurons during training, thereby enhancing the generalization ability of the model to further prevent overfitting, enabling the model to have better performance and stability when facing new and unseen data.

[0071] The above-described embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, supplements, and equivalent replacements made within the scope of the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent assessment system for the severity of atopic dermatitis in children, characterized in that: include: A data acquisition module, which is used to collect multispectral images, general images, electronic medical record data, and wearable device data from children with atopic dermatitis; Data registration module, used to register multispectral images with ordinary images through an improved feature point matching algorithm to ensure the spatial consistency of images of different modalities; The data fusion module is used to fuse the registered image with the electronic medical record data and the wearable device data to form a multimodal data set consisting of multimodal data fusion features; The classification module includes a deep learning model. The deep learning model includes an input layer and a classification layer. The input of the input layer is a multimodal data fusion feature, and the output of the classification layer is a three-classification result of normal, mild, and moderately severe. The input layer includes three front-end network modules, each of which includes two convolutional layers and one pooling layer, and an attention layer is inserted between the two network modules. The classification layer includes a Transformer layer, a fully connected layer, and a Dropout layer. The Transformer layer captures long-distance dependencies through a self-attention mechanism. The fully connected layer makes a final decision output based on the features extracted and integrated by the previous layer. The Dropout layer randomly discards some neuron outputs during training to increase the uncertainty of the model. The training module uses multimodal datasets to train and evaluate the deep learning model in the classification module.

2. The intelligent assessment system for severity of atopic dermatitis in children according to claim 1, characterized in that: Electronic medical record data includes structured and unstructured data of patients, and wearable device data includes temperature, humidity, and pH value of skin lesion areas.

3. The intelligent assessment system for severity of atopic dermatitis in children according to claim 1, characterized in that: In the data fusion module, the structured data in the electronic medical record data is preprocessed using a multi-layer perceptron, and the unstructured data in the electronic medical record data is preprocessed using natural language processing technology. The preprocessed structured data and unstructured data are fused with the registered image data and wearable device data to form a unified feature vector.

4. The intelligent assessment system for severity of atopic dermatitis in children according to claim 1, characterized in that: In the data registration module, the feature points of the multispectral image and the ordinary image are extracted through an improved feature point matching algorithm, and the similarity measure between the feature points is calculated to achieve accurate registration of the two.

5. The intelligent assessment system for severity of atopic dermatitis in children according to claim 4, characterized in that: The improved feature point matching algorithm has the following specific working process: Feature Point Detection: Acquiring Multispectral Images I t (x,y), normal image I r The SIFT feature points of (x, y) are used to establish a feature point matching set C1∩C2 based on the BBF rule; C1 represents the feature point set in the multispectral image, and C2 represents the feature point set in the ordinary image; Macro matching: For the feature point matching set C1∩C2, first assume that the sampling based on the uniform kernel function in the multispectral image is A i , the matching feature point in the ordinary image is B i After sampling, we get the sampling points (A1, A2, A3, ..., A i ) and (B1,B2,B3,……,B i ), the corresponding sampled grayscale array is expressed as (x1, x2, x3, ..., x i ) and (y1,y2,y3,……,y i ); then (x1,x2,x3,……,x i ) and (y1,y2,y3,……,y i ) is normalized so that ‖x‖=‖y‖=1; Define d as the segment correlation: Where x, y represent the normalized grayscale array of sampling points, and n represents the number of sampling points. The correlation scores of the two determined line segments are calculated and compared with the threshold T. The matching scores of each pair of feature points are obtained by voting, and the matching scores are sorted according to the scores. The matching scores with lower scores are deleted, and the matching scores with higher scores are retained. Micro matching: Based on the macro matching, select 4 pairs of matches with higher scores from the retained ones, and calculate the multispectral image I according to the pairs. t (x,y) and the normal image I r The projection change T of (x, y); then the formula is applied to calculate the global information similarity measure, and the maximum global similarity measure is compared and screened; After multiple rounds of image matching, the relationship between the lesion area in the ordinary image and the multispectral image is matched; when the skin area in the ordinary image shows lesion characteristics, the area in the corresponding multispectral image should show corresponding characteristics; in this way, the corresponding lesion areas in the two modalities are accurately combined to form a comprehensive skin lesion feature, and finally complete the image registration.

6. The intelligent assessment system for severity of atopic dermatitis in children according to claim 5, characterized in that: In the macro matching process, in order to further improve the probability of correct feature point matching, considering that there are similar feature points distributed around the matching feature points, the coordinate system is reconstructed according to the distribution of the feature points. For any pair of feature point matching (A, A ′ ), according to the coordinate constraints, we get Calculate the coordinate difference of each pair of feature points using the following formula: Matches whose distance is too large and exceeds the threshold D are considered to be wrong matches and are deleted; the threshold D is related to the number of feature point matches N input, and the relationship is: D = α*N.

7. The intelligent assessment system for severity of atopic dermatitis in children according to claim 5, characterized in that: The formula for calculating the global information similarity measure is as follows: in, express The gradient of the jth pixel, T represents the projection transformation operation, represents the degree of overlap of edge pixels in the ith region, S represents the global similarity, and N t Indicates the number of pixels in each area; Indicates the number of edge pixels at 0 degrees; Indicates the number of 45-degree edge pixels; Indicates the number of 90-degree edge pixels; Indicates the number of edge pixels at 135 degrees; I Γ (x,y) represents the T-transformed test image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the functions of the intelligent assessment system for the severity of atopic dermatitis in children according to any one of claims 1 to 7 are realized.

Images