Skin disease auxiliary system based on computer vision

Abstract

The invention discloses a skin disease auxiliary system based on computer vision, and particularly relates to the technical field of skin disease assistance, which comprises a multi-modal data acquisition module, an image preprocessing module, a deep learning diagnosis module and a clinical decision support module, according to the method, integrated hardware for synchronously collecting visible light and polarized light images and temperature and humidity parameters is adopted, a focus area is automatically segmented through self-adaptive filtering denoising and an improved U-Net network, and a Dice loss function optimization model is adopted; the method comprises the following steps: constructing a double-branch CNN model, extracting lesion image features by a first branch, normalizing physiological parameters by a second branch, obtaining a disease classification result through a feature fusion layer, storing a skin disease clinical knowledge base, matching similar cases and treatment schemes according to diagnosis results, and generating a structured diagnosis report; if the diagnosis confidence is high, matching according to disease categories; otherwise, expanding the range to assist identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dermatosis auxiliary, more particularly, the present application relates to a computer vision-based dermatosis auxiliary system. BACKGROUND

[0002] In the field of dermatosis clinical diagnosis and treatment, the incidence of skin diseases continues to rise worldwide. According to statistics from the World Health Organization, the prevalence rate of common skin diseases (such as eczema, psoriasis, and acne) has exceeded 20%. Due to the difficulty in early diagnosis, the mortality rate of some rare skin diseases (such as melanoma) remains high.

[0003] In recent years, artificial intelligence (AI) and computer vision technology have developed rapidly, providing a revolutionary solution to the above problems. Convolutional neural networks (CNN) and other deep learning models have shown superior performance in image classification, object detection, and segmentation tasks, making them successful in medical image analysis (such as lung nodule CT image recognition and fundus lesion screening). Skin lesion images, as a direct visual information, are very suitable for processing using computer vision technology.

[0004] However, in actual use, there are still some shortcomings, such as more than 3000 types of skin diseases, highly heterogeneous clinical manifestations, traditional diagnosis relying on doctors' subjective experience, low consistency in diagnosis by different qualified doctors, and long-term image follow-up for chronic skin diseases, which is time-consuming and lacks quantitative indicators. SUMMARY

[0005] To overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a computer vision-based dermatosis auxiliary system to solve the problems raised in the background art.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] Multi-modal data acquisition module: for synchronously acquiring visible light images, polarized light images, and skin surface temperature and humidity physiological parameters of a skin lesion area;

[0008] Image preprocessing module: for removing image noise through adaptive Gaussian filtering and combining an improved U-Net network for automatic segmentation of the lesion area;

[0009] Deep learning diagnosis module: a dual-branch CNN model is constructed based on transfer learning, the first branch extracts features from the segmented lesion image, the second branch normalizes the physiological parameters, and the outputs of the two branches are obtained through a feature fusion layer to obtain the disease classification result;

[0010] Clinical decision support module: Stores a clinical knowledge base of dermatology, automatically matches similar cases and treatment plans based on diagnostic results, and generates structured diagnostic reports, supporting doctors to manually adjust parameters to optimize diagnostic results.

[0011] Preferably, the multimodal data acquisition module adopts an integrated hardware architecture, and synchronously acquires visible light images, polarized light images and physiological parameters of skin surface temperature / humidity through a central control unit. The acquisition time deviation is ≤50ms, ensuring the consistency of multi-dimensional data in the time dimension and avoiding the problem of misalignment of lesion features due to asynchronous acquisition.

[0012] The system employs a CMOS image sensor with a resolution of ≥5 million pixels, equipped with a fixed-focus lens, and features automatic white balance adjustment. It automatically corrects image color deviations based on ambient light color temperature to ensure accurate color reproduction of lesion areas, with an image color reproduction accuracy of ΔZ≤3 (ΔZ is the color difference between the standard color and the acquired color in the CIELAB color space; the smaller the value, the higher the reproduction accuracy). Additionally, a ring-shaped supplementary lighting component is included. The supplementary lighting source uses high color rendering index LED beads, and the supplementary lighting intensity is adjusted via a PWM signal. This allows for adaptive adjustment of the supplementary lighting power based on skin tone, avoiding overexposure due to strong light or increased image noise due to weak light, ensuring a signal-to-noise ratio of ≥35dB for the visible light image.

[0013] In addition to acquiring visible light images, the polarized light image acquisition system integrates a polarizer assembly and a 550-650nm band filter assembly. The polarizer features a rotatable design with a rotation angle range of 0°-180° and an adjustment accuracy of 1°. The central control unit controls a motor to drive the polarizer's rotation, acquiring lesion images at different polarization angles: 0°, 45°, 90°, and 135°, thereby extracting the polarization characteristics of the lesion area. The 550-650nm band filter assembly employs narrowband interference filtering technology with a filtering bandwidth ≤20nm. This filters stray light outside this band, reducing interference from ambient light on the polarized image contrast. Furthermore, this band is a characteristic absorption band of hemoglobin in skin tissue, enhancing the grayscale difference between blood vessels in the lesion area and surrounding normal skin tissue, making lesion features such as vascular dilation and bleeding easier to identify. After processing by this filter assembly, the polarized image shows improved contrast between the lesion and the background.

[0014] Preferably, in the image preprocessing module, the adaptive Gaussian filtering unit adopts a modal adaptive filtering strategy based on the noise characteristics of different modal images: for visible light images, it first traverses the image through a sliding window and calculates the mean gray value μ and variance σ of the pixels within each window. 2 If the variance within the window σ 2If the noise level is greater than the preset threshold E, the window is considered to have strong noise interference, and the standard deviation σ of the Gaussian filter kernel is adjusted accordingly. g Adjust to 1.5-2.0 to enhance noise filtering; if the variance σ within the window... 2 If σ ≤ E, then the window noise interference is considered weak, and σ is set to... g Adjusted to 0.8-1.4, while removing noise, the edge details of the lesion are preserved to the greatest extent. After this adaptive Gaussian filtering process, the peak signal-to-noise ratio of the visible light image is improved to ≥38dB, and the PSNR of the polarized light image is improved to ≥35dB.

[0015] The window size is automatically switched between 3×3, 5×5, and 7×7 according to the image noise density. The noise density is determined by calculating the standard deviation of the image gray value. When the standard deviation is >30, a 7×7 window is selected; when 15 < standard deviation ≤30, a 5×5 window is selected; and when the standard deviation ≤15, a 3×3 window is selected.

[0016] The value of A is between 25 and 40, and the optimal value is determined through training with 10,000 clinical dermatology image samples;

[0017] The preprocessed visible light image feature map, polarized light image feature map, and normalized skin temperature and humidity parameters are then fused. First, feature mapping is performed on the temperature and humidity parameters; the specific calculation method is as follows:

[0018]

[0019] Among them, F T Let W be a temperature feature vector, δ represent the ReLU activation function, and W be a temperature feature vector. T Represented as a weighting factor, T skin Represented as a real-time measured skin temperature value, T avg The average skin temperature, T std Expressed as temperature standard deviation, b T Represented as a bias term;

[0020]

[0021] Among them, F H Let W be a humidity feature vector, where δ represents the ReLU activation function. H Represented as a weighting factor, H skin H represents the real-time measured skin moisture value. avg H represents average skin moisture. std Expressed as the standard deviation of humidity, b H Represented as a bias term;

[0022] F T F H With FV F P The channel attention-weighted feature maps are concatenated to obtain a multimodal fusion feature map. The specific calculation method is as follows:

[0023] F fuse =Concat(F cV F cP F T F H )

[0024] Among them, F fuse This is represented as a multimodal fusion feature map, where Concat(·) represents the concatenation operation, and F cV Represented as a channel-weighted feature map of a visible light image, F cP Represented as a channel-weighted feature map of a polarized light image, F T Represented as a temperature eigenvector, F H This is represented as a humidity feature vector.

[0025] Preferably, in the deep learning diagnostic module, the dual-branch CNN model uses transfer learning as its core framework. The input to the first branch is the segmented lesion image output by the image preprocessing module mentioned above, and the input to the second branch is the skin temperature and humidity acquired by the multimodal data acquisition module.

[0026] The first branch is the image feature branch, which uses a ResNet-50 network pre-trained on ImageNet as the basic backbone. Since the segmented image corresponding to the predicted region of the previous model retains the detailed features of the lesions (such as the scale texture of psoriasis and the edge of the exudate area of ​​eczema), an attention mechanism consistent with the encoding end of the improved U-Net network is added after the 3rd and 4th convolutional blocks of ResNet-50 to enhance the ability to extract key features of the lesions. The final output is an image feature vector F with a dimension of 2048×1×1. img ;

[0027] The second branch is the physiological parameter branch. To take advantage of the low dimensionality of physiological parameters, a 3-layer fully connected network structure is designed, and the input layer normalizes the physiological parameters to ensure that the parameter distribution is adapted to the image feature vector.

[0028] The physiological parameter normalization process in the second branch introduces an error compensation term into the Dice loss function. This normalization process needs to be optimized in conjunction with error characteristics. The specific formula derivation is as follows:

[0029] Raw parameter preprocessing: First, outlier acquired values ​​are removed to obtain valid parameters; error compensation normalization introduces an acquisition error coefficient to normalize the valid parameters. The specific calculation method is as follows:

[0030]

[0031] Among them, T norm T represents the normalized temperature. valid T represents the currently valid real-time measurement value. avg The average temperature, T max This represents the upper limit of the temperature acquisition range, T. min This represents the lower limit of the temperature acquisition range, and α represents the error coefficient.

[0032]

[0033] Among them, H norm Humidity expressed as normalized value, H valid H represents the currently valid real-time measurement value. avg The average humidity, H max H represents the upper limit of the humidity sampling range. min This represents the lower limit of the humidity sampling range;

[0034] Normalized T norm and H norm The range of values ​​for all values ​​is controlled within [-1, 1].

[0035] Normalized T norm and H norm Concatenate into a 2D vector P raw Feature mapping is performed using a 3-layer fully connected network. The specific calculation methods for each layer of the network are as follows:

[0036] First layer fully connected: F p1 =δ(W1×P raw +b1), where F p1 It is represented as a 128-dimensional feature vector output by the first layer, W1 is represented as a weight matrix of dimension 128×2, and b1 is represented as a bias of dimension 128×1;

[0037] Second layer fully connected: F p2 =δ(W2×F p1 +b2), where F p2 W1 represents the 256×128 weight matrix of the second layer output, W2 represents the 256×128 weight matrix, and b2 represents the 256×1 bias.

[0038] Third layer fully connected F phy =W3×F p2 +b3, where F phy The final output is represented by a 2048-dimensional physiological parameter feature vector, where W3 represents a weight matrix of dimension 2048×256, and F... p2The weight matrix is ​​represented as 256×128, which is the output of the second layer. b3 represents the bias with a dimension of 2048×1. There is no activation function. The final output is combined with the first branch image feature vector F. img A 2048-dimensional physiological parameter feature vector F with consistent dimensions phy .

[0039] Preferably, in the clinical decision support module, a clinical knowledge base for dermatology is first constructed, comprising a case database, a treatment plan database, and a diagnostic criteria database. The case database stores images of lesions acquired by the multimodal acquisition module and segmented images output by the image preprocessing module, as well as classification results and confidence levels from the deep learning diagnostic module. Each case is labeled with the doctor's final diagnosis, pathology report, and follow-up data. The treatment plan database stores clinical guideline recommended treatments (such as topical corticosteroid regimens for eczema and phototherapy regimens for psoriasis) according to the N types of dermatological diseases covered by the deep learning diagnostic module. The database includes statistics on drug contraindications and efficacy, and incorporates individual parameters such as patient allergy history. The diagnostic criteria database integrates the correspondence between ICD-11 dermatology codes and multimodal features (such as hemangioma features under 550-650nm polarized light, and the correlation between humidity >60%RH and fungal infection). It also adopts an "automatic incremental update + manual review" mechanism. After new cases are confirmed by doctors, the database is updated by extracting the segmented lesion image features, physiological parameter features, and classification results through a federated learning framework. The latest clinical guidelines are accessed quarterly and updated with treatment plan databases after expert review to ensure the timeliness and accuracy of the knowledge base.

[0040] In the process of matching similar cases with treatment plans, the module triggers different logic based on the confidence level (Conf) of the deep learning diagnosis results: if Conf ≥ 85%, the preliminary disease category is used as the screening condition, and the similarity is calculated using the "weighted feature distance" algorithm; if Conf < 85%, the matching range is expanded to similar disease categories to assist doctors in differential diagnosis.

[0041] The technical effects and advantages of this invention are as follows:

[0042] This invention employs integrated hardware to simultaneously acquire visible / polarized light images and temperature and humidity parameters, and achieves lesion segmentation through adaptive Gaussian filtering and an improved U-Net. A dual-branch CNN based on transfer learning fuses image features and physiological indicators, and a dynamic weighted attention mechanism optimizes classification accuracy. The clinical module integrates a knowledge base, intelligently matches similar cases and treatment plans based on confidence levels, generates structured reports, and supports manual intervention and adjustments. The system automatically outputs diagnostic results when a confidence level of 85% or higher is maintained; otherwise, a dynamic learning update mechanism is triggered.

[0043] Compared to traditional single-modal auxiliary systems, this invention integrates visual and physiological parameters, and deeply coordinates the acquisition and processing of data from each modality to characterize lesions from multiple dimensions, reducing the rate of missed diagnoses and misdiagnoses. Furthermore, it combines an improved U-Net network to achieve high-precision lesion segmentation, overcoming the limitations of traditional methods in areas with blurred boundaries and small lesions, and significantly improving segmentation accuracy. The dual-branch CNN model fuses image and physiological features, and uses a weighted attention mechanism to dynamically highlight discriminative features. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the module connection of the present invention.

[0045] Figure 2 This is a schematic diagram of the dual-branch CNN workflow of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figure 1 As shown, the present invention provides a computer vision-based dermatology assistance system, including a multimodal data acquisition module, an image preprocessing module, a deep learning diagnostic module, and a clinical decision support module.

[0048] Multimodal data acquisition module: used to simultaneously acquire visible light images, polarized light images, and physiological parameters of skin surface temperature and humidity in the skin lesion area;

[0049] The multimodal data acquisition module adopts an integrated hardware architecture and synchronously acquires visible light images, polarized light images, and physiological parameters of skin surface temperature / humidity through a central control unit. The acquisition time deviation is ≤50ms, ensuring the consistency of multi-dimensional data in the time dimension and avoiding the problem of misalignment of lesion features due to asynchronous acquisition.

[0050] The system employs a CMOS image sensor with a resolution of ≥5 million pixels, equipped with a fixed-focus lens, and features automatic white balance adjustment. It automatically corrects image color deviations based on ambient light color temperature to ensure accurate color reproduction of lesion areas, with an image color reproduction accuracy of ΔZ≤3 (ΔZ is the color difference between the standard color and the acquired color in the CIELAB color space; the smaller the value, the higher the reproduction accuracy). Additionally, a ring-shaped supplementary lighting component is included. The supplementary lighting source uses high color rendering index LED beads, and the supplementary lighting intensity is adjusted via a PWM signal. This allows for adaptive adjustment of the supplementary lighting power based on skin tone, avoiding overexposure due to strong light or increased image noise due to weak light, ensuring a signal-to-noise ratio of ≥35dB for the visible light image.

[0051] In addition to acquiring visible light images, the polarized light image acquisition system integrates a polarizer assembly and a 550-650nm band filter assembly. The polarizer features a rotatable design with a rotation angle range of 0°-180° and an adjustment accuracy of 1°. The central control unit controls a motor to drive the polarizer's rotation, acquiring lesion images at different polarization angles: 0°, 45°, 90°, and 135°, thereby extracting the polarization characteristics of the lesion area. The 550-650nm band filter assembly employs narrowband interference filtering technology with a filtering bandwidth ≤20nm. This filters stray light outside this band, reducing interference from ambient light on the polarized image contrast. Furthermore, this band is a characteristic absorption band of hemoglobin in skin tissue, enhancing the grayscale difference between blood vessels in the lesion area and surrounding normal skin tissue, making lesion features such as vascular dilation and bleeding easier to identify. After processing by this filter assembly, the polarized image shows improved contrast between the lesion and the background.

[0052] The contrast ratio is calculated as: (average gray value of lesion area - average gray value of background area) / (average gray value of background area × 100%).

[0053] The skin surface temperature and humidity physiological parameter acquisition unit adopts a contact sensing scheme; the skin surface temperature uses a high-precision thermocouple sensor with a temperature measurement range of 25-45℃ (covering the normal temperature range of human skin and the abnormal temperature range of lesions). The influence of ambient temperature (15-35℃) on the measurement results is corrected by a temperature compensation algorithm. The compensation formula is: T0=T1+k(T2-25), where T0 represents the actual skin surface temperature, T1 represents the temperature directly measured by the sensor, T2 represents the ambient temperature, and k represents the temperature compensation coefficient, with a value range of 0.02-0.05.

[0054] The skin surface humidity is collected using a capacitive humidity sensor with a measurement range of 20% to 90% relative humidity. By coating the sensor probe with an anti-sweat interference coating, the influence of skin sweat on humidity measurement is avoided. At the same time, a periodic calibration mechanism is adopted (the calibration process is automatically triggered every 100 uses). The calibration process corrects the sensor drift error by comparing it with the standard humidity environment generated by a standard humidity generator.

[0055] Image preprocessing module: used to remove image noise through adaptive Gaussian filtering and to automatically segment lesion regions by combining with an improved U-Net network;

[0056] In the image preprocessing module, the adaptive Gaussian filtering unit employs a modal adaptive filtering strategy tailored to the noise characteristics of images of different modalities: For visible light images, it first traverses the image through a sliding window, calculating the mean gray value μ and variance σ of each pixel within the window. 2 If the variance within the window σ 2 If the noise level is greater than the preset threshold E, the window is considered to have strong noise interference, and the standard deviation σ of the Gaussian filter kernel is adjusted accordingly. g Adjust to 1.5-2.0 to enhance noise filtering; if the variance σ within the window... 2 If σ ≤ E, then the window noise interference is considered weak, and σ is set to... g Adjusted to 0.8-1.4, while removing noise, the edge details of the lesion are preserved to the greatest extent. After this adaptive Gaussian filtering process, the peak signal-to-noise ratio of the visible light image is improved to ≥38dB, and the PSNR of the polarized light image is improved to ≥35dB.

[0057] The window size is automatically switched between 3×3, 5×5, and 7×7 according to the image noise density. The noise density is determined by calculating the standard deviation of the image gray value. When the standard deviation is >30, a 7×7 window is selected; when 15 < standard deviation ≤30, a 5×5 window is selected; and when the standard deviation ≤15, a 3×3 window is selected.

[0058] The value of A is between 25 and 40, and the optimal value is determined through training with 10,000 clinical dermatology image samples;

[0059] The preprocessed visible light image feature map, polarized light image feature map, and normalized skin temperature and humidity parameters are then fused. First, feature mapping is performed on the temperature and humidity parameters; the specific calculation method is as follows:

[0060]

[0061] Among them, F T Let W be a temperature feature vector, δ represent the ReLU activation function, and W be a temperature feature vector. T Represented as a weighting factor, T skin Represented as a real-time measured skin temperature value, Tavg The average skin temperature, T std Expressed as temperature standard deviation, b T Represented as a bias term;

[0062]

[0063] Among them, F H Let W be a humidity feature vector, where δ represents the ReLU activation function. H Represented as a weighting factor, H skin H represents the real-time measured skin moisture value. avg H represents average skin moisture. std Expressed as the standard deviation of humidity, b H Represented as a bias term;

[0064] F T F H With F V F P The channel attention-weighted feature maps are concatenated to obtain a multimodal fusion feature map. The specific calculation method is as follows:

[0065] F fuse =Concat(F cV F cP F T F H )

[0066] Among them, F fuse This is represented as a multimodal fusion feature map, where Concat(·) represents the concatenation operation, and F cV Represented as a channel-weighted feature map of a visible light image, F cP Represented as a channel-weighted feature map of a polarized light image, F T Represented as a temperature eigenvector, F H Represented as a humidity feature vector;

[0067] In the automatic segmentation of lesion areas, the improved U-Net network is based on the traditional U-Net encoder-decoder symmetrical structure. To address the problems of "blurred boundaries, irregular shapes, and easy confusion with normal skin texture" in skin lesions, structural improvements are made:

[0068] First, an attention mechanism unit is connected in series after each convolutional operation at the encoding end. This unit first performs global average pooling and global max pooling on the feature map output by the encoding end to obtain two channel description vectors. Then, the two vectors are concatenated and input into a weight generation network consisting of two fully connected layers and a sigmoid activation function to generate a channel attention weight vector. This vector is used to weight each channel of the feature map, strengthening the weight of the lesion feature channel and suppressing the weight of the background feature channel.

[0069] Second, in the skip connection path between the encoding and decoding ends, a 1×1 convolution kernel is used to unify the number of channels of the feature map at the encoding end with the number of channels of the corresponding layer feature map at the decoding end. At the same time, a bilinear interpolation algorithm is used to adjust the size of the feature map at the encoding end to ensure that the feature maps are completely matched in spatial dimension during the skip connection, thus avoiding segmentation deviation caused by feature misalignment.

[0070] After the last convolutional operation at the decoding end, the traditional Softmax activation function is replaced with the Dice loss function. This function optimizes the model parameters by calculating the overlap (Dice coefficient) between the model's predicted segmented region and the manually labeled gold standard region. The calculation formula is as follows:

[0071]

[0072] Where A represents the model's predicted region and B represents the manually labeled gold standard region. When the Dice coefficient approaches 1, it indicates that the segmentation result is completely consistent with the gold standard. This loss function solves the problem of model training being biased towards the background region due to the low pixel proportion of the lesion region in the skin lesion sample.

[0073] The model's predicted region A is calculated through four steps: multimodal fusion feature decoding, convolutional feature mapping, activation function transformation, and threshold determination.

[0074] First, improving the U-Net network decoder requires modifying the multimodal fusion feature map F. fuse The system performs progressive upsampling and convolution operations, and finally outputs the original predicted feature map F through the last convolutional layer. pred ;

[0075] During upsampling and convolution iterations at the decoding end, the feature map of the k-th layer at the decoding end... The specific calculation method is as follows:

[0076]

[0077] in, This is represented as the output feature map of the decoder's k-th layer. When k = 0, This is the initial fused feature map; Conv 3×3 This is represented as a 3×3 convolution operation, and UpSample represents bilinear interpolation upsampling. This is represented as the output feature map of the (k-1)th layer of the decoder. This is represented as a skip connection in the feature map. The feature map of the encoder at layer K-k+1, where K represents the total number of layers and k represents the current layer number;

[0078] The final convolutional feature output: The last layer of the decoder uses a 1×1 convolutional layer to compress the number of channels in the feature map to 2, obtaining the original predicted feature map. The specific calculation method is as follows:

[0079]

[0080] Among them, F pred Represented as the original predicted feature map, Represented as the output feature map of the last layer of the decoder; Conv 1×1 This is represented as a 1×1 convolution operation with no activation function, directly outputting the original predicted values ​​of the two types of features;

[0081] F pred (1,x,y) is labeled as the corresponding background class, F pred (2,x,y) is labeled as the corresponding lesion class, and (x,y) represents the pixel coordinates;

[0082] After the last convolutional layer of U-Net, a Sigmoid activation function adapted to the Dice loss function is used to obtain a lesion class probability feature map. The "lesion class" channel in the original predicted feature map is then activated with Sigmoid to obtain the probability that each pixel is a lesion. The specific calculation method is as follows:

[0083]

[0084] Among them, P lesion (x,y) represents the probability that pixel (x,y) belongs to a lesion, σ(·) represents the Sigmoid activation function, and F pred (2,x,y) represents the corresponding lesion class; when the calculated P lesion The closer (x,y) is to 1, the higher the confidence that the pixel is a lesion;

[0085] In determining the probability threshold, the probability threshold is set by combining the annotation logic of the manually annotated gold standard region B, and a binarized prediction region A is generated. The specific calculation method is as follows:

[0086]

[0087] Where A(x,y) represents the binarized prediction region, P lesion (x,y) represents the probability value that pixel (x,y) belongs to the lesion, and τ represents the probability threshold.

[0088] When A(x,y)=1, it means that the pixel belongs to the lesion area predicted by the model;

[0089] When A(x,y)=0, it indicates that it belongs to the background region.

[0090] Deep learning diagnostic module: Based on transfer learning, a two-branch CNN model is constructed. The first branch extracts features from the segmented lesion image, and the second branch normalizes the physiological parameters. The outputs of the two branches are passed through a feature fusion layer to obtain the disease classification result.

[0091] In the deep learning diagnostic module, the dual-branch CNN model uses transfer learning as its core framework. The input to the first branch is the segmented lesion image output by the image preprocessing module mentioned above, and the input to the second branch is the skin temperature and humidity acquired by the multimodal data acquisition module.

[0092] The first branch is the image feature branch, which uses a ResNet-50 network pre-trained on ImageNet as the basic backbone. Since the segmented image corresponding to the predicted region of the previous model retains the detailed features of the lesions (such as the scale texture of psoriasis and the edge of the exudate area of ​​eczema), an attention mechanism consistent with the encoding end of the improved U-Net network is added after the 3rd and 4th convolutional blocks of ResNet-50 to enhance the ability to extract key features of the lesions. The final output is an image feature vector F with a dimension of 2048×1×1. img ;

[0093] The second branch is the physiological parameter branch. To take advantage of the low dimensionality of physiological parameters, a 3-layer fully connected network structure is designed, and the input layer normalizes the physiological parameters to ensure that the parameter distribution is adapted to the image feature vector.

[0094] The physiological parameter normalization process in the second branch introduces an error compensation term into the Dice loss function. This normalization process needs to be optimized in conjunction with error characteristics. The specific formula derivation is as follows:

[0095] Raw parameter preprocessing: First, outlier acquired values ​​are removed to obtain valid parameters; error compensation normalization introduces an acquisition error coefficient to normalize the valid parameters. The specific calculation method is as follows:

[0096]

[0097] Among them, T norm T represents the normalized temperature. valid T represents the currently valid real-time measurement value. avg The average temperature, T max This represents the upper limit of the temperature acquisition range, T. min This represents the lower limit of the temperature acquisition range, and α represents the error coefficient.

[0098]

[0099] Among them, H norm Humidity expressed as normalized value, H validH represents the currently valid real-time measurement value. avg The average humidity, H max H represents the upper limit of the humidity sampling range. min This represents the lower limit of the humidity sampling range;

[0100] Normalized T norm and H norm The range of values ​​for all values ​​is controlled within [-1, 1].

[0101] Normalized T norm and H norm Concatenate into a 2D vector P raw Feature mapping is performed using a 3-layer fully connected network. The specific calculation methods for each layer of the network are as follows:

[0102] First layer fully connected: F p1 =δ(W1×P raw +b1), where F p1 It is represented as a 128-dimensional feature vector output by the first layer, W1 is represented as a weight matrix of dimension 128×2, and b1 is represented as a bias of dimension 128×1;

[0103] Second layer fully connected: F p2 =δ(W2×F p1 +b2), where F p2 W1 represents the 256×128 weight matrix of the second layer output, W2 represents the 256×128 weight matrix, and b2 represents the 256×1 bias.

[0104] Third layer fully connected F phy =W3×F p2 +b3, where F phy The final output is represented by a 2048-dimensional physiological parameter feature vector, where W3 represents a weight matrix of dimension 2048×256, and F... p2 The weight matrix is ​​represented as 256×128, which is the output of the second layer. b3 represents the bias with a dimension of 2048×1. There is no activation function. The final output is combined with the first branch image feature vector F. img A 2048-dimensional physiological parameter feature vector F with consistent dimensions phy ;

[0105] In the feature fusion layer's fusion mechanism, to achieve effective fusion of image features and physiological parameter features, the feature fusion layer adopts a "weighted attention fusion" strategy. Weights are dynamically allocated based on the discriminative power of the dual-branch features. First, F... img With F phy Global average pooling is performed separately to obtain a 1-dimensional feature descriptor F. img-avg With F phy-avgThen, the weights are calculated using a shared attention network. The specific calculation method is as follows:

[0106] W img =σ(W a ×F img-avg +b a )

[0107] Among them, W img W is represented as the attention weight of image features. a Represented as a shared weight matrix of dimension 1×2048, b a Represented as a shared bias of dimension 1×1;

[0108] W phy =σ(W a ×F phy-avg +b a )

[0109] Among them, W phy The attention weights, W, are represented as physiological parameter features. a Represented as a shared weight matrix of dimension 1×2048, b a Represented as a shared bias of dimension 1×1;

[0110] The fused feature vector F is obtained by weighting and summing the two-branch features according to the attention weights. fusion The calculation method is as follows:

[0111] F fusion =W img ×F img +W phy ×F phy

[0112] Among them, F fusion Represented as a fused feature vector, W img The attention weights, F, are represented as image features. img Represented as an image feature vector, W phy The attention weights, F, are represented as physiological parameter features. phy Represented as a feature vector of physiological parameters;

[0113] This fusion method can dynamically highlight more discriminative features (such as increasing the weight of humidity features in physiological parameters when diagnosing eczema; and increasing the weight of color and texture features in image features when diagnosing pigmented nevi), thereby improving the accuracy of subsequent classification.

[0114] After fusing the feature vectors and inputting them into the classification layer, the predicted probabilities of various skin diseases are calculated using the Softmax activation function. Combined with common clinical skin disease types, the disease classification results are output. The specific process is as follows:

[0115] The classification layer uses a single fully connected network to classify the 2048-dimensional F... fusion The mapping is to an N-dimensional original classification vector (where N is the number of supported skin disease categories), and the specific calculation method is as follows:

[0116] S raw =W cls ×F fusion +b cls

[0117] Among them, S raw Represented as the original classification vector, W cls Represented as a classification weight matrix of dimension N×2048, b cls Represented as a classification bias of dimension N×1, F fusion Represented as a fused feature vector;

[0118] Then for S raw Softmax activation is performed to obtain the predicted probabilities for each class. The specific calculation method is as follows:

[0119]

[0120] Among them, P cls (i) represents the predicted probability of the i-th type of skin disease, S raw (i) represents the original predicted score for the i-th type of skin disease, where i represents the target category being traversed, and S raw (j) represents the original prediction score for the j-th type of skin disease;

[0121] S raw (i) i.e., the original classification vector S raw The element corresponding to the i-th type of skin disease is the inactive prediction result of the classification layer for the i-th type of skin disease;

[0122] S raw (j) refers to the original classification vector S raw The element corresponding to the j-th type of skin disease, and S raw (i) has the same definition, only the index is different, j traverses all categories, including i itself;

[0123] The category with the highest predicted probability is selected as the preliminary disease classification result, and the corresponding confidence score is output. The confidence score is calculated as Conf = max(P cls (i) When the maximum confidence level is greater than 85%, the classification result is considered reliable and is taken as the final skin disease. When the maximum confidence level is less than 85%, the classification result is considered unreliable and triggers the model dynamic update mechanism. The classification layer parameters are optimized by supplementing similar case data to improve the accuracy of subsequent diagnosis.

[0124] Clinical decision support module: Stores a clinical knowledge base of dermatology, automatically matches similar cases and treatment plans based on diagnostic results, and generates structured diagnostic reports, supporting doctors to manually adjust parameters to optimize diagnostic results.

[0125] The clinical decision support module first constructs a dermatology clinical knowledge base comprising a case database, a treatment plan database, and a diagnostic criteria database. The case database stores images acquired by the multimodal acquisition module, segmented lesion images output by the image preprocessing module, classification results and confidence levels from the deep learning diagnostic module, and each case is labeled with the doctor's final diagnosis, pathology report, and follow-up data. The treatment plan database stores clinical guideline recommended treatments (such as topical corticosteroids for eczema and phototherapy for psoriasis) and medications according to the N types of dermatology covered by the deep learning diagnostic module. The database includes statistics on drug contraindications and efficacy, and is linked to individual parameters such as patient allergy history. The diagnostic criteria database integrates the correspondence between ICD-11 dermatology codes and multimodal features (such as hemangioma features under 550-650nm polarized light, and the correlation between humidity >60%RH and fungal infection). It also adopts an "automatic incremental update + manual review" mechanism. After new cases are confirmed by doctors, the database is updated by extracting the segmented lesion image features, physiological parameter features, and classification results through a federated learning framework. The latest clinical guidelines are accessed every quarter and updated after expert review to ensure the timeliness and accuracy of the knowledge base.

[0126] In the process of matching similar cases with treatment plans, the module triggers different logic based on the confidence level (Conf) of the deep learning diagnosis results: if Conf ≥ 85%, the initial disease category is used as the screening condition, and the "weighted feature distance" algorithm is used to calculate the similarity; if Conf < 85%, the matching range is expanded to similar disease categories to assist doctors in differential diagnosis.

[0127] When generating a treatment plan, the standard plan for the corresponding disease is first extracted from the treatment plan database. Combined with plans from similar cases, the plan is then adapted and adjusted according to the individual characteristics of the patient to generate 2-3 alternative plans with expected efficacy marked (e.g., "Plan 1: 82% effective rate, onset time 3-5 days; Plan 2: 75% effective rate, low incidence of side effects"). If the patient's physiological parameters are abnormal (e.g., temperature > 38℃ indicates co-infection, humidity < 30% RH indicates skin barrier damage), risk information is extracted from the diagnostic criteria database and a prompt is added to the plan (e.g., "It is recommended to check blood routine to rule out infection before starting phototherapy"), and drug interactions are also marked.

[0128] In the structured diagnostic report generation process, the module integrates end-to-end data from multimodal acquisition, image preprocessing, deep learning diagnosis, and similar case matching. It includes six modules: basic patient information, multimodal examination data, lesion segmentation results, and deep learning diagnostic results. These modules clearly define the disease name, confidence level, and classification probability distribution (e.g., "Eczema: 89%, Contact Dermatitis: 7%, Other: 4%)," similar case references, and recommended treatment plans. Furthermore, the module supports manual adjustments and optimizations by doctors: preliminary disease classification results can be modified (e.g., changing eczema to contact dermatitis), and the module automatically rematches cases and treatment plans after modification; medication types, dosages, or cycles can be adjusted based on recommended plans, with the module providing real-time feedback on differences from guidelines (e.g., "Dosage is lower than recommended, which may affect efficacy") and recording the reasons for adjustment (e.g., "Patient has sensitive skin, reduce medication frequency"); and weighting coefficients for similar case matching can be customized for different skin diseases and saved as personal preference settings.

[0129] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A computer vision-based dermatology assistance system, characterized in that, include: Multimodal data acquisition module: used to simultaneously acquire visible light images, polarized light images, and physiological parameters of skin surface temperature and humidity in the skin lesion area; Image preprocessing module: used to remove image noise through adaptive Gaussian filtering and to automatically segment lesion regions by combining with an improved U-Net network; Deep learning diagnostic module: Based on transfer learning, a two-branch CNN model is constructed. The first branch extracts features from the segmented lesion image, and the second branch normalizes the physiological parameters. The outputs of the two branches are passed through a feature fusion layer to obtain the disease classification result. Clinical decision support module: Stores a clinical knowledge base of dermatology, automatically matches similar cases and treatment plans based on diagnostic results, and generates structured diagnostic reports, supporting doctors to manually adjust parameters to optimize diagnostic results.

2. The computer vision-based dermatology assistance system according to claim 1, characterized in that: The multimodal data acquisition module adopts an integrated hardware architecture, and synchronously acquires visible light images, polarized light images, and physiological parameters of skin surface temperature / humidity through a central control unit. In addition to acquiring visible light images, the polarized light image acquisition module also integrates a polarizer component and a 550-650nm band filter component. The polarizer adopts a rotatable design with a rotation angle range of 0°-180° and an adjustment accuracy of 1°. The central control unit can control the motor to drive the polarizer to rotate, thereby acquiring lesion images at different polarization angles: 0°, 45°, 90°, and 135°, to extract the polarization characteristics of the lesion area.

3. The computer vision-based dermatology assistance system according to claim 1, characterized in that: The skin surface temperature and humidity physiological parameter acquisition unit adopts a contact sensing scheme; the skin surface temperature uses a high-precision thermocouple sensor with a temperature measurement range of 25-45℃ (covering the normal temperature range of human skin and the abnormal temperature range of lesions). The influence of ambient temperature (15-35℃) on the measurement results is corrected by a temperature compensation algorithm. The compensation formula is: T0=T1+k(T2-25), where T0 represents the actual skin surface temperature, T1 represents the temperature directly measured by the sensor, T2 represents the ambient temperature, and k represents the temperature compensation coefficient, with a value range of 0.02-0.

05.

4. The computer vision-based dermatology assistance system according to claim 1, characterized in that: In the image preprocessing module, the adaptive Gaussian filtering unit employs a modal adaptive filtering strategy tailored to the noise characteristics of images of different modalities: For visible light images, it first traverses the image through a sliding window, calculating the mean gray value μ and variance σ of each pixel within the window. 2 If the variance within the window σ 2 If the noise level is greater than the preset threshold E, the window is considered to have strong noise interference, and the standard deviation σ of the Gaussian filter kernel is adjusted accordingly. g Adjust to 1.5-2.0; if the variance σ within the window... 2 If σ ≤ E, then the window noise interference is considered weak, and σ is set to... g The value was adjusted to 0.8-1.4 to remove noise while preserving the details of the lesion edges to the greatest extent possible; then the preprocessed visible light image feature map, polarized light image feature map, and normalized skin temperature and humidity parameters were fused together. F T F H With F V F P The channel attention-weighted feature maps are concatenated to obtain a multimodal fusion feature map. The specific calculation method is as follows: F fuse =Concat(F cV ,F cP ,F T ,F H ) Among them, F fuse This is represented as a multimodal fusion feature map, where Concat(·) represents the concatenation operation, and F cV Represented as a channel-weighted feature map of a visible light image, F cP Represented as a channel-weighted feature map of a polarized light image, F T Represented as a temperature eigenvector, F H This is represented as a humidity feature vector.

5. The computer vision-based dermatology assistance system according to claim 4, characterized in that: In the automatic segmentation of lesion areas, the improved U-Net network is based on the traditional U-Net's encoder-decoder symmetrical structure. The structure is improved to address the problems of "blurred boundaries, irregular shape, and easy confusion with normal skin texture" in skin lesions. First, an attention mechanism unit is connected in series after each convolutional operation at the encoding end. This unit first performs global average pooling and global max pooling on the feature map output by the encoding end to obtain two channel description vectors. Then, the two vectors are concatenated and input into a weight generation network consisting of two fully connected layers and a sigmoid activation function to generate a channel attention weight vector. This vector is used to weight each channel of the feature map, strengthening the weight of the lesion feature channel and suppressing the weight of the background feature channel. Second, in the skip connection path between the encoding and decoding ends, a 1×1 convolution kernel is used to unify the number of channels of the feature map at the encoding end with the number of channels of the corresponding layer feature map at the decoding end. At the same time, a bilinear interpolation algorithm is used to adjust the size of the feature map at the encoding end to ensure that the feature maps are completely matched in spatial dimension during the skip connection, thus avoiding segmentation deviation caused by feature misalignment. After the final convolutional operation at the decoding end, the traditional Softmax activation function is replaced with the Dice loss function. This function optimizes the model parameters by calculating the overlap between the segmented region predicted by the model and the manually labeled gold standard region. The calculation formula is as follows: Where A represents the model-predicted region and B represents the manually labeled gold standard region. When the Dice coefficient approaches 1, it indicates that the segmentation result is completely consistent with the gold standard.

6. The computer vision-based dermatology assistance system according to claim 4, characterized in that: The model's predicted region A is calculated through four steps: multimodal fusion feature decoding, convolutional feature mapping, activation function transformation, and threshold determination. First, improving the U-Net network decoder requires modifying the multimodal fusion feature map F. fuse The system performs progressive upsampling and convolution operations, and finally outputs the original predicted feature map F through the last convolutional layer. pred ; After the last convolutional layer in U-Net, a Sigmoid activation function adapted to the Dice loss function is used to obtain a lesion class probability feature map. The "lesion class" channel in the original predicted feature map is then activated with Sigmoid to obtain the probability that each pixel is a lesion. The specific calculation method is as follows: Among them, P lesion (x,y) represents the probability that pixel (x,y) belongs to a lesion, σ(·) represents the Sigmoid activation function, and F pred (2,x,y) represents the corresponding lesion class; when the calculated P ledion The closer (x,y) is to 1, the higher the confidence that the pixel is a lesion; In determining the probability threshold, the probability threshold is set by combining the annotation logic of the manually annotated gold standard region B, and a binarized prediction region A is generated. The specific calculation method is as follows: Where A(x,y) represents the binarized prediction region, P lesion (x,y) represents the probability value that pixel (x,y) belongs to the lesion, and τ represents the probability threshold. When A(x,y)=1, it means that the pixel belongs to the lesion area predicted by the model; When A(x,y)=0, it indicates that it belongs to the background region.

7. The computer vision-based dermatology assistance system according to claim 1, characterized in that: In the deep learning diagnostic module, the dual-branch CNN model uses transfer learning as its core framework. The input of the first branch is the segmented lesion image output by the image preprocessing module mentioned above, and the input of the second branch is the skin temperature and humidity obtained by the multimodal data acquisition module. The first branch is the image feature branch, which uses a ResNet-50 network pre-trained on ImageNet as the basic backbone. After the 3rd and 4th convolutional layers of ResNet-50, an attention mechanism consistent with the encoding end of the improved U-Net network is added to enhance the extraction capability of key lesion features. The final output is an image feature vector F with a dimension of 2048×1×1. img ; The second branch is the physiological parameter branch. To address the low dimensionality of physiological parameters, a three-layer fully connected network structure is designed, and the input layer normalizes the physiological parameters.

8. The computer vision-based dermatology assistance system according to claim 1, characterized in that: In the feature fusion layer's fusion mechanism, to achieve effective fusion of image features and physiological parameter features, the feature fusion layer adopts a "weighted attention fusion" strategy. Weights are dynamically allocated based on the discriminative power of the dual-branch features. First, F... img With F phy Global average pooling is performed separately to obtain a 1-dimensional feature descriptor F. img-avg With F phy-avg The weights are then calculated using a shared attention network. After fusing the feature vectors and inputting them into the classification layer, the predicted probabilities of various skin diseases are calculated using the Softmax activation function. Combined with common clinical skin disease types, the disease classification results are output. The specific process is as follows: The classification layer uses a single fully connected network to classify the 2048-dimensional F... fusion The mapping to an N-dimensional original classification vector is specifically calculated as follows: S raw =W cls ×F fusion +b cls Among them, S raw Represented as the original classification vector, W cls Represented as a classification weight matrix of dimension N×2048, b cls Represented as a classification bias of dimension N×1, F fusion Represented as a fused feature vector; Then for S raw Softmax activation is performed to obtain the predicted probabilities for each class. The specific calculation method is as follows: Among them, P cls (i) represents the predicted probability of the i-th type of skin disease, S raw (i) represents the original predicted score for the i-th type of skin disease, where i represents the target category being traversed, and S raw (j) represents the original prediction score for the j-th type of skin disease; The category with the highest predicted probability is selected as the preliminary disease classification result, and the corresponding confidence score is output. The confidence score is calculated as Conf = max(P cls (i) When the maximum confidence level is greater than 85%, the classification result is considered reliable and is taken as the final skin disease; when the maximum confidence level is less than 85%, the classification result is considered unreliable and the model dynamic update mechanism is triggered.

9. The computer vision-based dermatology assistance system according to claim 1, characterized in that: The clinical decision support module first constructs a clinical knowledge base for dermatology, comprising a case database, a treatment plan database, and a diagnostic criteria database. The case database stores images acquired by the multimodal acquisition module, segmented lesion images output by the image preprocessing module, classification results and confidence levels from the deep learning diagnostic module, and each case is labeled with the doctor's final diagnosis, pathology report, and follow-up data. The treatment plan database stores clinical guideline-recommended treatments, drug contraindications, and efficacy statistics according to the N types of dermatology covered by the deep learning diagnostic module, and associates these with individual parameters related to the patient's allergy history. The diagnostic criteria database integrates the correspondence between ICD-11 dermatology codes and multimodal features, and employs an "automatic incremental update + manual review" mechanism. New cases, after doctor confirmation, are updated to the case database by extracting segmented lesion image features, physiological parameter features, and classification results through a federated learning framework. The treatment plan database is updated quarterly after being updated by experts based on the latest clinical guidelines.

Images