Wine stain recognition-efficacy evaluation system based on multi-modal features, electronic device

Abstract

This invention relates to a multimodal feature-based system for identifying and evaluating the therapeutic effect of port-wine stains, and an electronic device, comprising: a multimodal data acquisition module for acquiring multimodal facial data; a data preprocessing module for performing quality inspection, noise reduction, and standardization on the multimodal data; an erythema identification module that uses a hybrid model combining pathological guidance, multi-scale vision, and semantically enhanced text to output facial erythema classification results based on the preprocessed multimodal data; a feature linkage module that establishes a mapping relationship between PWS (Prognostic and Surface Therapy) identification features and therapeutic effect evaluation features: if the facial erythema classification result output by the erythema identification module is port-wine stains, then the core features for therapeutic effect evaluation are extracted; and a therapeutic effect evaluation module that, based on the core features for therapeutic effect evaluation and post-treatment multimodal data of the same type, calculates the color fading rate, area reduction rate, and thickness improvement rate of port-wine stains, and outputs a comprehensive therapeutic effect evaluation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a wine stain identification-therapeutic effect evaluation system based on multi-modal features and an electronic device. BACKGROUND

[0002] Port wine stain (PWS) is also known as fresh red macule, which is a congenital capillary malformation. It can be found at birth, and the color of the lesion deepens and the area expands with age. In adult stage, it may develop into hypertrophy or nodular type. The face is a high incidence site, and it is easy to be confused with other red macular diseases such as hemangioma and allergic purpura. Therefore, there is an urgent need for accurate identification and objective evaluation scheme.

[0003] At present, the identification of PWS mostly depends on experience, which is affected by individual experience difference, visual fatigue, and atypical early features of the lesion, resulting in high misjudgment rate. Dermoscopy is non-invasive but relies on professional skills, which is difficult to popularize at the grassroots level. The accuracy of computer-aided identification is low in scenes with hair obstruction, uneven light and shadow, and skin type difference.

[0004] For the therapeutic effect evaluation of PWS, the visual analog scale method uses subjective scoring, which has large scoring difference and cannot distinguish multi-dimensional therapeutic effect. The computer vision scheme is easily affected by the complex background of the face and the shooting conditions.

[0005] In summary, the current wine stain identification-therapeutic effect evaluation scheme has the following shortcomings:

[0006] (1) Lack of multi-modal data fusion capability, single modality cannot completely capture the information required for identification and evaluation.

[0007] (2) The identification and evaluation processes are separated, the data cannot be reused, the efficiency is low, and the false evaluation is easily caused by misjudgment.

[0008] (3) Not suitable for PWS individual differences and complex facial scenes, the evaluation lacks pertinence.

[0009] In summary, the current lack of a wine stain identification-therapeutic effect evaluation system to solve or partially solve the aforementioned problems. SUMMARY

[0010] The purpose of the present application is to overcome the defects of the prior art and provide a wine stain identification-therapeutic effect evaluation system based on multi-modal features and an electronic device to solve or partially solve the problem of easy confusion with other types of red macules in wine stain identification-therapeutic effect evaluation, which easily leads to false evaluation.

[0011] The purpose of the present application can be achieved by the following technical solutions:

[0012] In one aspect of the present application, a wine stain recognition and efficacy evaluation system based on multi-modal features is provided, comprising:

[0013] A multi-modal data acquisition module is configured to acquire multi-modal data of a face, wherein the multi-modal data comprises visible light images of the face, multi-spectral angiography data, optical coherence tomography lesion thickness data, and structured clinical text data.

[0014] A data preprocessing module is configured to perform quality detection, noise elimination, and standardization processing on the multi-modal data.

[0015] A red spot recognition module is configured to output a face red spot classification result based on the preprocessed multi-modal data using a hybrid model combining pathological guidance, multi-scale vision, and semantic enhanced text.

[0016] A feature linkage module is configured to establish a mapping relationship between PWS identification features and efficacy evaluation features: if the face red spot classification result output by the red spot recognition module is a wine stain, then the vascular density, initial a* value (component in the CIELAB color space of the International Commission on Illumination), and initial thickness in the multi-modal data are extracted as core features for efficacy evaluation; if the face red spot classification result output is not a wine stain, then the efficacy evaluation of the wine stain is terminated.

[0017] An efficacy evaluation module is configured to calculate the color fading rate, area reduction rate, and thickness improvement rate of the wine stain based on the core features for efficacy evaluation and post-treatment multi-modal data of the same type, and output a comprehensive efficacy evaluation result.

[0018] As a preferred technical solution, the red spot recognition module comprises:

[0019] A multi-modal data processing unit is configured to perform image cropping processing based on facial anatomy for the visible light images of the face, and perform term extraction and label annotation processing for the structured clinical text data.

[0020] A multi-scale vision feature pyramid unit comprises a dual-branch multi-scale vision encoder, which obtains multi-scale features by aggregation, wherein the main branch is configured to extract features from the cropped face sub-images, and the adaptive branch is configured to dynamically allocate scale feature weights based on anatomical partitioning of the face sub-images.

[0021] A wine stain text semantic enhancement unit is configured to generate text features based on a preconfigured wine stain dictionary.

[0022] The pathological feature embedding cross-modal attention unit is used to calculate the matching degrees of multi-scale features and feature templates and text features and feature templates respectively based on a plurality of preset feature templates, to obtain cross-modal attention weights, to fuse the multi-scale features and the text features based on the cross-modal attention weights, to obtain a fusion feature vector, and to realize pathological guidance.

[0023] The classification unit is used to output a erythema classification result and a confidence based on the fusion feature vector according to a preset rule.

[0024] As a preferred technical solution, the main branch outputs six scale feature maps, wherein the features of the second layer and the third layer are used as bottom layer features to represent the blood vessel texture of small lesions; the features of the fourth layer and the fifth layer are used as middle layer features to represent the color boundary of the lesions and normal skin; and the features of the sixth layer are used as top layer features to represent the overall morphology of large lesions.

[0025] As a preferred technical solution, in the adaptive branch, the weights of the bottom layer features, the middle layer features and the top layer features decrease in turn for small lesion subgraphs, and the weights of the bottom layer features, the middle layer features and the top layer features increase in turn for large lesion subgraphs.

[0026] As a preferred technical solution, the method further comprises:

[0027] The training set acquisition unit is used to acquire a training set;

[0028] The size adaptation training unit is used to divide the training set into a small lesion subset and a large lesion subset according to lesion sizes; to learn the fine reticular texture of small lesions by combining texture loss and cross-entropy loss on the premise of freezing the sixth layer parameters of the main branch based on the small lesion subset; to learn the boundary of large lesions by combining boundary loss and cross-entropy loss on the premise of freezing the second layer and the third layer parameters of the main branch based on the large lesion subset; and to train the adaptive branch based on the training set with the objective of minimizing the mean square error loss of the predicted weights and the real size labels.

[0029] The part pathology training unit is used to divide the training set into an eye part subset, a nose wing subset, a cheek subset and a forehead subset according to facial parts; to fine-tune the parameters of the sixth layer based on the texture loss, the boundary loss and the size weight loss on the premise of unfreezing the second layer and the third layer parameters based on the eye part subset and the nose wing subset; and to fine-tune the parameters of the second layer and the third layer based on the boundary loss, the texture loss and the size weight loss on the premise of unfreezing the sixth layer parameters based on the cheek subset and the forehead subset.

[0030] As a preferred technical solution, the classification unit comprises a main output layer and an auxiliary output layer, the main output layer is used to obtain a probability of erythema classification based on the fusion feature vector, and the auxiliary output layer is used to obtain a pathological feature matching score based on the fusion feature vector.

[0031] As a preferred technical solution, in the classification unit, the rules comprise:

[0032] If the pathological feature matching score and the probability of wine-colored macule in the erythema classification probability are greater than a first threshold configuration, the wine-colored macule is classified.

[0033] If the pathological feature matching score and the probability of wine-colored macule in the erythema classification probability are greater than a second threshold configuration and less than the first threshold configuration, a secondary verification is triggered.

[0034] If the pathological feature matching score and the probability of wine-colored macule in the erythema classification probability are less than the second threshold configuration, the wine-colored macule is classified as non-wine-colored macule.

[0035] As a preferred technical solution, the color fading rate of the wine-colored macule is: (average value of the wine-colored macule region of interest before treatment-average value of the wine-colored macule region of interest after treatment) / average value of the wine-colored macule region of interest before treatment*100%.

[0036] The area reduction rate is: (actual area before treatment-actual area after treatment) / actual area before treatment*100%, wherein the actual area is obtained by converting the pixel area of the PWS lesion in the high-definition visible light image of the face combined with the actual size of the facial anatomical feature points.

[0037] The thickness improvement rate is: (average thickness before treatment-average thickness after treatment) / average thickness before treatment*100%, wherein the thickness is obtained based on the optical coherence tomography lesion thickness data.

[0038] As a preferred technical solution, the structured clinical text data comprises age, skin type, facial part of erythema lesion, and onset time.

[0039] In another aspect of the present application, an electronic device is provided, comprising one or more processors, a memory, and the aforementioned multi-modal feature-based wine-colored macule identification-therapeutic effect evaluation system.

[0040] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0041] (1) Reduce the possibility of misjudgment caused by non-portion pigmentation: This invention sets up an erythema recognition module and adopts a hybrid model that combines pathological guidance, multi-scale vision and semantically enhanced text. Based on the preprocessed multimodal data, it outputs facial erythema classification results. After verifying that the input data is port-wine pigmentation, the feature linkage module performs subsequent efficacy evaluation core feature extraction, reducing the error in evaluation caused by misjudgment.

[0042] (2) High accuracy of wine stain verification: The erythema recognition module of the present invention includes a multi-scale visual feature pyramid unit, including a dual-branch multi-scale visual encoder, which obtains multi-scale features through aggregation. The main branch is used to extract features from the cropped facial sub-image, and the adaptation branch is used to dynamically allocate the feature weights of each scale based on the anatomical partition of the facial sub-image, thus solving the problem of poor adaptation of single scale. The erythema recognition module also includes a wine stain text semantic enhancement unit, which generates text features based on a pre-set wine stain dictionary, thus solving the problem of insufficient generalization of text semantics. The erythema recognition module also includes a cross-modal attention unit with pathological feature embedding, which obtains cross-modal attention weights based on multiple pre-set feature templates, thus solving the problem of feature fusion without pathological guidance. The above design improves the accuracy of wine stain verification.

[0043] (3) High adaptability of multi-scale visual feature pyramid unit: The main branch of the dual-branch multi-scale visual encoder of the present invention includes bottom layer, middle layer and top layer, which respectively represent the vascular texture of small lesions, the color boundary between lesions and normal skin and the overall shape of large lesions. In the training process, a two-stage training method of size adaptation training stage and site pathology training is adopted. First, the dataset is split according to the size of the lesion, and the bottom texture extraction of small lesions and the top boundary extraction of large lesions are strengthened respectively. Then, the facial parts are grouped and the pathological features of port-wine stains are incorporated to achieve synergistic optimization of size, location and pathology, and improve the feature extraction capability of multi-scale visual feature pyramid unit. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the multimodal feature-based wine stain recognition and efficacy assessment system in the embodiment.

[0045] Figure 2 This is a schematic diagram of the erythema recognition module in the embodiment;

[0046] Figure 3 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0048] Example 1

[0049] To address the problems existing in the aforementioned prior art, this embodiment provides a multimodal feature-based system for identifying and evaluating the treatment efficacy of port-wine stains. This system can be deployed on high-performance computers and servers. (See [link to documentation]). Figure 1 The system includes a multimodal data acquisition module, a data preprocessing module, a PWS identification module, a feature linkage module, and a efficacy evaluation module, which are connected in sequence. The functions of each module will be described in detail below:

[0050] (1) Modal data acquisition module.

[0051] This module is used to collect multimodal data related to port-wine stains. In this embodiment, it includes at least high-resolution visible light images of the face, multispectral angiography data, optical coherence tomography (OCT) lesion thickness data, and structured clinical text data of the patient. The structured clinical text data includes at least the patient's age, skin type, specific facial location of the PWS lesion, and onset time. Lesion locations may include the periorbital area, cheek, nasal alae, and forehead.

[0052] It should be noted that the collection of the above data should be done with the patient's prior consent and under legal conditions.

[0053] For high-resolution visible light images of the face, the resolution should be no less than 300 dpi. A ring light source should be used to uniformly illuminate the face from the front, avoiding shadows on the sides of the cheeks and below the nose to reduce light and shadow interference in facial depressions. Images should include those before treatment, after each treatment, and during follow-up, covering the PWS lesion and an area at least 5 mm beyond its edge to avoid missing the boundaries of facial PWS.

[0054] For optical coherence tomography (OCT) data, the aim is to collect thickness data of the lesion area for possible hypertrophic / nodular lesions of facial PWS. The scanning accuracy should be no less than 10 μm, and it should also cover 5 mm outside the edge of the lesion to adapt to the uneven thickness of facial PWS.

[0055] For structured clinical text data of patients, considering the differences in the rate of PWS regression at different ages and the characteristics of skin color quantification benchmark, at least the patient's age, skin type, specific facial location of PWS lesions, onset time, and previous treatment history should be recorded.

[0056] During the acquisition of visible light images and optical coherence tomography (OCT) data, care should be taken to maintain a fixed posture when taking images, aligning with the outer canthi of both eyes and the root of the nose as a reference, to ensure that the facial PWS position does not shift before and after treatment.

[0057] The OCT scan path corresponds to the PWS lesion area in the high-resolution image to avoid image-thickness data misalignment, ensuring that subsequent steps can correlate visual and structural features.

[0058] (2) Data preprocessing module.

[0059] This module is used to perform quality inspection, noise reduction, and standardization processing on the collected multimodal data.

[0060] The module first detects the sharpness, exposure, and color reproduction of high-resolution visible light images, removing blurry / overexposed images to avoid affecting PWS color feature extraction. Then, denoising is performed, using Gaussian filtering to eliminate random noise and median filtering to remove salt-and-pepper noise caused by facial hair, overcoming the problem of PWS boundaries being obscured by hair around the eyes and forehead. Next, contrast enhancement is performed, using histogram equalization to enhance the color contrast between PWS lesions and normal skin, considering the smaller color difference between PWS and normal skin in patients with light skin. Finally, color space conversion is performed, converting the RGB image to the Lab color space.

[0061] Preferably, the process may also include a feature registration process, using fixed anatomical feature points on the face as coordinate references to establish a spatial mapping relationship between high-resolution images and OCT data before treatment, after treatment, and during follow-up, correcting for PWS lesion position shifts caused by changes in the patient's shooting posture, and ensuring that subsequent assessments are conducted on the same lesion and in the same region. The feature points can be points such as the outer canthi of both eyes, the root of the nose, and the base of the nose, which are not affected by changes in facial expressions.

[0062] (3) PWS identification module.

[0063] This module uses a hybrid model that integrates a visual encoder and a language encoder. It takes preprocessed multimodal data as input and outputs disease classification results, namely PWS or other erythematous diseases.

[0064] To address the issue of misclassification caused by confusion between facial rosacea (PWS) and other erythematous diseases, this module employs a hybrid architecture combining a dual-branch visual encoder and a text encoder. The visual encoder, built on EfficientNet-B4, aims to extract color and texture differences between facial PWS and other erythematous lesions, obtaining visual features from high-resolution visible light images. For example, PWS exhibits a reticular vascular texture, while hemangiomas show a clumpy texture. The text encoder, based on a BioBERT pre-trained model, aims to extract clinical textual differences such as onset time and skin type, obtaining semantic features from structured clinical text. For example, PWS is present at birth, while rosacea develops in adulthood. The two types of features are fused through a cross-modal attention mechanism, with the input to a fully connected layer and the output being the disease classification probability.

[0065] For details, see Figure 2 The erythema recognition module mainly includes a multimodal data processing unit, a parallel multi-scale visual feature pyramid unit, a wine stain text semantic enhancement unit, a cross-modal attention unit for embedding pathological features, and a classification unit, all connected in sequence. The functions of each unit will be described in detail below:

[0066] 1) Multimodal data processing unit.

[0067] This unit is used for image cropping based on facial anatomy for visible light images and multispectral vascular imaging data, and for term extraction and labeling for structured clinical text data.

[0068] Specifically, for high-resolution visible light images of the face, morphological opening operations are used to remove facial hair, and sub-images are cropped according to facial anatomical regions, such as the periorbital area, cheeks, nasal alae, and forehead, to prepare for subsequent multi-scale feature extraction. For structured clinical texts of patients, specific terminology is extracted, pathological labels are added, and structured text vectors are formed. Labels may include vascular structures and onset time.

[0069] 2) Multi-scale visual feature pyramid unit.

[0070] This unit includes a dual-branch multi-scale visual encoder, which obtains multi-scale features through aggregation. The main branch is used to extract features from the cropped facial sub-images, and the adaptation branch is used to dynamically allocate feature weights for each scale based on the anatomical partitions of the facial sub-images.

[0071] For a dual-branch multi-scale visual encoder, it mainly includes a main branch based on EfficientNet-B4 and an adaptation branch:

[0072] The main branch is used to extract features from the cropped facial sub-images, outputting feature maps at six scales, corresponding to layers 1 to 6. Layers 2 and 3 serve as the bottom layer features, capturing pixel-level vascular textures in small lesions such as those around the eyes; these features can be used to determine whether the lesion is reticular. Layers 4 and 5 serve as the middle layer features, capturing the color boundaries between lesions and normal skin; this setup takes into account the clear boundaries of PWS and the blurred boundaries of hemangiomas. Layer 6 serves as the top layer features, capturing the overall morphology of large lesions such as those on the cheeks, used to determine whether they are proportionally distributed according to anatomical regions.

[0073] The adaptation branch aims to predict lesion size by dynamically allocating feature weights at each scale based on the anatomical divisions of the facial sub-image. For example, the periorbital sub-image corresponds to small lesions. For small lesion sub-images: bottom layer feature weight > middle layer feature weight > top layer feature weight. For example, the bottom layer feature weight is 60%, the middle layer is 30%, and the top layer is 10%. This setting prioritizes capturing reticular textures. For large lesion sub-images: bottom layer feature weight < middle layer feature weight < top layer feature weight. For example, the top layer feature weight is 50%, the middle layer is 35%, and the bottom layer is 15%. This setting prioritizes capturing boundary morphology.

[0074] By feature aggregation, the weighted multi-scale features are compressed into PWS visual feature vectors, in which the signal intensity of key PWS visual features such as reticular vascular texture and clear boundaries is improved.

[0075] 3) Text semantic enhancement unit for wine stains.

[0076] This unit is used to generate text features based on a pre-built dictionary of wine stains.

[0077] Specifically, this unit first inputs the PWS structured text preprocessed by the multimodal data processing unit into BioBERT. A pre-built PWS-specific dictionary is loaded into the embedding layer to improve the cosine distance between the embedding vectors of terms such as "present at birth" and "color deepens with age" and other erythematous disease terms, achieving dedicated dictionary embedding. Then, the pathological labels annotated by the multimodal data processing unit are used as enhancement factors for text features, amplifying the features of text fragments containing these labels to achieve pathological label association. For example, with the pathological label "vascular structure - reticular," if the text fragment is "reticular blood vessels are visible under dermoscopy," the corresponding feature is amplified. Finally, a fully connected layer reduces the dimensionality of the text features to PWS text feature vectors, improving the discriminative power of PWS-specific text features such as "onset at birth" and "proportional enlargement with body size."

[0078] 4) Cross-modal attention units embedded with pathological features.

[0079] This unit is used to calculate the matching degree between multi-scale features and feature templates, and between text features and feature templates, based on multiple pre-set feature templates, to obtain cross-modal attention weights. Based on the cross-modal attention weights, multi-scale features and text features are fused to obtain a fused feature vector, thereby achieving pathological guidance.

[0080] This unit first loads the pre-built PWS-specific pathological feature library and outputs feature templates for three core pathological labels, including a visual template for "reticular vascular texture" and a text template for "present at birth". Then, it calculates the attention weights. Specifically, it calculates the matching degree between the visual feature vector and the visual pathological template, and the matching degree between the text feature vector and the text pathological template. The two matching degrees are weighted and summed as the cross-modal attention weights. The weights here can be adjusted according to the actual situation.

[0081] Finally, this unit performs weighted fusion of visual and textual features to generate a 3072-dimensional PWS discrimination fusion feature vector, in which the proportion of key pathological features of PWS is significantly improved compared with existing models.

[0082] 5) Classification unit.

[0083] This unit is used to output erythema classification results and confidence scores based on fused feature vectors and according to preset rules.

[0084] The classifier in this unit employs a dual-output fully connected layer, consisting of a main output layer and an auxiliary output layer. The main output layer outputs the probabilities of four categories: PWS, hemangioma, rosacea, and other erythematous lesions. The auxiliary output layer outputs the pathological feature matching scores, such as the overall matching degree for reticular vessels, congenital lesions, and proportionally enlarged lesions.

[0085] After calculating the probabilities of the four categories and the overall matching degree, classification is performed based on classification rules. If the PWS probability is ≥95% and the pathological feature matching score is ≥80, it is directly determined as PWS; if the PWS probability is 85%-94% and the pathological feature matching score is ≥70, secondary verification is triggered, for example, by calling vascular density data from multispectral imaging to assist in the judgment; all other cases are determined as non-PWS. The above threshold configuration can be adjusted according to the actual situation.

[0086] Finally, this unit outputs the disease classification results and confidence levels, while also annotating key identification criteria, such as visual: reticular vessels; text: present at birth.

[0087] Furthermore, the dual-branch multi-scale visual encoder, a key focus of this embodiment, is crucial for achieving effective identification and treatment evaluation. To address this, this embodiment also constructs a training set acquisition unit, a size adaptation training unit, and a site pathology training unit to pre-train the dual-branch multi-scale visual encoder.

[0088] 6) Training set acquisition unit.

[0089] Multiple high-resolution visible light images of facial PWS were acquired, covering different sizes and locations, including small lesions with an area <1cm² located around the eyes and nasal alae. 2 It also includes large lesions located on the cheeks and forehead, with an area >5cm². 2 The areas represented include around the eyes, nose, cheeks, and forehead. Additionally, images with background interference, such as hair occlusion or uneven lighting, are used to simulate realistic scenes.

[0090] Image annotation, in this embodiment, was performed by a senior dermatologist. Annotation dimensions included: size annotation (using rectangles to mark the extent of lesions, categorizing lesions as small or large by area); location annotation (categorizing lesions based on facial anatomical landmarks, such as the periorbital area, nasal alae, cheek, and forehead); pathological texture annotation (using pixel-level masks to annotate vascular texture areas, labeling them with fine or coarse mesh patterns); and background interference annotation (annotating areas obscured by hair and areas with uneven lighting, used for targeted suppression of interfering features during training).

[0091] The dataset is divided into training, validation, and test sets in a 7:2:1 ratio to ensure that the size, location, and disease distribution of each subset are consistent with the total dataset, thus avoiding data bias.

[0092] 7) Size-adaptive training unit.

[0093] The model is initialized before training.

[0094] For the main branch, it is built based on the pre-trained EfficientNet-B4, retaining its 6-layer feature extraction structure.

[0095] For the adaptation branch, two fully connected layers and a Softmax layer are added. The input is the facial region label, and the output is the weight coefficients for small lesions and large lesions. For example, for the facial region label of the periorbital area, the output is a weight of 0.8 for small lesions and a weight of 0.2 for large lesions, and for the facial region label of the cheek, the output is a weight of 0.2 for small lesions and a weight of 0.8 for large lesions. This is used to dynamically adjust the feature weights of the main branch later.

[0096] The pre-trained parameters of the main branch are retained, and the convolution kernel parameters of layers 2-3 and 6 are locally unfrozen to adapt to PWS features. The adapted branch parameters are randomly initialized using a He normal distribution.

[0097] The goal of this unit is to enable different layers of the main branch to adapt to the feature extraction requirements of small and large lesions, and to adapt the branch learning size-weight mapping relationship. The unit first splits the training set into small lesion subsets and large lesion subsets, and trains them separately.

[0098] For training on a subset of small lesions, the aim is to enhance the texture extraction capability of the bottom layer of the main branch. Top-level parameters are frozen, and the input consists of small lesion images and labels. The loss function is primarily texture loss (SSIM), supplemented by cross-entropy loss; specific weights can be set according to actual conditions. By calculating the similarity between the extracted bottom-level texture and the labeled mesh texture mask, the model is forced to learn the fine mesh texture of the small lesions. In this embodiment, the optimizer is AdamW, with a learning rate of 1e-4 and 50 iterations, each iteration verifying the accuracy of small lesion texture extraction.

[0099] For training on a subset of large lesions, the aim is to enhance the boundary extraction capability of the top layer of the main branch while freezing the bottom layer parameters. The input consists of images and labels of large lesions. The loss function primarily uses boundary loss (Dice), supplemented by cross-entropy loss; specific weights can be set according to actual conditions. By calculating the overlap between the extracted top-level boundaries and the labeled lesion bounding boxes, the model is forced to learn the clear boundaries of large lesions. The optimizer in this embodiment also uses AdamW, with a learning rate of 1e-4 and 50 iterations, validating the IOU of the large lesion boundaries in each iteration.

[0100] The adaptation branch is trained synchronously by taking facial part labels as input and outputting size weight coefficients. The training is based on the mean square error loss between the predicted weights and the actual size labels, so that the adaptation branch can automatically output reasonable size weights according to the parts.

[0101] 8) Site-specific pathology training unit.

[0102] This unit aims to optimize the feature fusion logic of the two branches by combining the pathological differences of different facial regions in PWS, and improve the site adaptability of feature extraction. This unit first divides the training set into 4 groups according to facial regions: periorbital group, nasal alar group, cheek group, and forehead group. Each group of samples contains small / large lesions in the corresponding region.

[0103] For the periorbital and nasal alar groups, which are characterized by fine mesh textures, the bottom-level parameters of the main branch are thawed and the top-level parameters are fine-tuned. The input is periorbital / nasal alar images and fine mesh texture pathological labels. The loss function includes texture loss, boundary loss and size weight loss, with a weight of 1.5 times for small lesions to enhance the extraction of fine mesh textures.

[0104] For the cheek / forehead group, which is characterized by coarse mesh texture and clear boundaries, the top-level parameters of the main branch are unfrozen and the bottom-level parameters are fine-tuned. The input is the cheek / forehead image and pathological labels with coarse mesh texture and clear boundaries. The loss function includes boundary loss, texture loss and size weight loss, with a weight of 1.2 times for large lesions to enhance coarse texture and boundary extraction.

[0105] Furthermore, it is also used for cross-branch feature fusion training. In each training group, the size weight coefficient of the adapted branch output is dynamically adjusted to adjust the feature weight of the main branch. For example, if the weight of small lesions in the periorbital group is 0.8, then the bottom-level features of the main branch contribute 80%. The matching loss between the fused features and the pathological labels is calculated to optimize the fusion logic.

[0106] This unit uses the AdamW optimizer with a learning rate of 5e-5 and 80 iterations. In each iteration, the feature extraction accuracy is verified by part.

[0107] The loss functions involved in the size adaptation training unit and the site pathology training unit will be explained below.

[0108] Texture loss is obtained by calculating the structural similarity between the extracted vascular texture features and the labeled mesh texture template.

[0109] Boundary loss (Dice) calculates the Dice coefficient for each lesion boundary pixel individually, where the boundary pixel can be a pixel within a 5-pixel range of the labeled bounding box.

[0110] The size-weighted loss dynamically adjusts the loss weight based on the sample size, with a loss of 1.5 times for small lesion samples and a loss of 1.2 times for large lesion samples, to avoid diluting the number of small lesion samples during training due to their proportion.

[0111] Through joint training by two units, the weights are adjusted every 10 rounds based on the feature extraction accuracy of the validation set until the overall feature extraction accuracy of the validation set reaches the target accuracy, at which point the loss weight optimization stops.

[0112] (4) Feature linkage module.

[0113] This module establishes a mapping relationship between PWS identification features and efficacy evaluation features. If the PWS identification module outputs PWS, it extracts vascular density from multimodal data based on multispectral vascular imaging data, calculates the initial a* value of the lesion in Lab space based on high-resolution visible light facial images, and calculates the initial thickness based on optical coherence tomography as the core feature for efficacy evaluation. If the output is not PWS, the PWS efficacy evaluation process is terminated and the feature matching logic for the corresponding erythema disease is triggered.

[0114] Preferably, if the output is not PWS, the feature matching logic for the corresponding erythema disease is triggered, and the corresponding efficacy evaluation logic is executed. This part is not the focus of this embodiment and is omitted here.

[0115] (5) Therapeutic efficacy assessment module.

[0116] This module is used to quantitatively calculate the color fading rate, area reduction rate, and thickness improvement rate of PWS based on the core features of efficacy assessment and similar multimodal data after treatment, and output the comprehensive efficacy assessment results.

[0117] 1. Calculation of color fading rate.

[0118] Color fading rate is used as the core indicator to reflect the degree of vascular regression in PWS. The core indicator uses the U-Net++ segmentation model to automatically select the region of interest (ROI) of PWS lesions in high-resolution images, avoiding deviations in the extent of facial PWS caused by manual selection.

[0119] Extract the average a* value within the ROI before and after treatment, which is the red component in the Lab space, and calculate the color fading rate using the formula: (a* value before treatment - a* value after treatment) / a* value before treatment × 100%.

[0120] Preferably, weighting is applied based on differences in facial location. If PWS is located in areas where color is more sensitive to therapeutic effects, such as around the eyes, the weight of this indicator is increased. If it is located on the cheeks, where thickness has a greater impact, the weight is decreased.

[0121] 2. Calculation of area reduction rate.

[0122] The area reduction rate reflects the change in the extent of PWS lesions. The pixel area of ​​PWS lesions in high-resolution images before and after treatment was extracted using the U-Net++ segmentation model.

[0123] Based on the actual size of facial anatomical feature points in the data preprocessing module, the pixel area is converted into the actual area, and the area reduction rate is calculated according to the formula: (actual area before treatment - actual area after treatment) / actual area before treatment × 100%.

[0124] 3. Calculation of thickness improvement rate.

[0125] The thickness improvement rate setting is designed for hypertrophic / nodular facial lesions. PWS extracts the average thickness of PWS lesions before and after treatment from OCT data, and calculates the thickness improvement rate according to the formula: (average thickness before treatment - average thickness after treatment) / average thickness before treatment × 100%.

[0126] 4. Overall scoring and grading.

[0127] Color fading rate, area reduction rate, and thickness improvement rate are input into a multilayer perceptron (MLP) or XGBoost model with site-specific weights, outputting a comprehensive efficacy score from 0 to 100. The score is divided into 4 levels:

[0128] Excellent: >75 points, PWS color fades significantly, area is greatly reduced, and thickness is significantly improved.

[0129] Good: 50-75 points, all three indicators show moderate improvement.

[0130] Moderate: 25-50 points, with at least one indicator showing slight improvement.

[0131] Poor: <25 points, with no improvement or deterioration in any of the three indicators.

[0132] 5. Visualized report output.

[0133] Generate a dedicated facial PWS report, containing four core categories for easy and intuitive interpretation:

[0134] Lesion annotation map: The PWS range is marked with a bright border on the high-resolution image to distinguish changes in facial lesions before and after treatment.

[0135] Color fading heatmaps show the changes in a* values ​​in different areas within the lesion, with red, yellow, and green indicating increasing degrees of fading.

[0136] OCT thickness comparison profile, overlaid with lesion thickness curves before and after treatment, visually demonstrates the thickness improvement.

[0137] The comprehensive assessment form lists the values ​​of three indicators, the overall score, and the results of the efficacy grading.

[0138] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0139] Example 2

[0140] Based on Embodiment 1, this embodiment provides an electronic device, such as... Figure 3 As shown, it includes one or more processors, memory, and the multimodal feature-based wine stain recognition-treatment assessment system of Embodiment 1. Preferably, the electronic device also includes a visualization terminal for displaying a visualization report.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal feature-based system for identifying and evaluating the treatment efficacy of wine follicles, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal data of the face, including visible light images of the face, multispectral vascular imaging data, optical coherence tomography lesion thickness data, and structured clinical text data. The data preprocessing module is used to perform quality detection, noise removal, and standardization on the multimodal data; The erythema recognition module employs a hybrid model that combines pathological guidance, multi-scale vision, and semantically enhanced text. Based on preprocessed multimodal data, it outputs facial erythema classification results. The feature linkage module establishes a mapping relationship between PWS identification features and efficacy evaluation features: if the facial erythema classification result output by the erythema recognition module is port-wine stain, then the vascular density, initial a* value of the lesion, and initial thickness in the multimodal data are extracted as core features for efficacy evaluation; if the output facial erythema classification result is not port-wine stain, then the efficacy evaluation of port-wine stain is terminated. The efficacy evaluation module, based on core efficacy evaluation features and post-treatment multimodal data of the same type, calculates the color fading rate, area reduction rate, and thickness improvement rate of port-wine stains, and outputs a comprehensive efficacy evaluation result. The erythema recognition module includes: The multimodal data processing unit is used for image cropping based on facial anatomy for visible light images of the face, and for term extraction and labeling for structured clinical text data. The multi-scale visual feature pyramid unit includes a dual-branch multi-scale visual encoder, which obtains multi-scale features through aggregation. The main branch is used to extract features from the cropped facial sub-image, and the adaptation branch is used to dynamically allocate feature weights at each scale based on the anatomical partitions of the facial sub-image. The wine stain text semantic enhancement unit is used to generate text features based on a pre-built wine stain dictionary; The cross-modal attention unit embedded in pathological features is used to calculate the matching degree between multi-scale features and feature templates, and between text features and feature templates, based on multiple pre-set feature templates, to obtain cross-modal attention weights. Based on the cross-modal attention weights, multi-scale features and text features are fused to obtain a fused feature vector, thereby realizing pathological guidance. The classification unit is used to output erythema classification results and confidence scores based on the fused feature vector and according to preset rules.

2. The multimodal feature-based system for identifying and evaluating the therapeutic effect of wine follicles according to claim 1, characterized in that, The main branch outputs feature maps at six scales. The features of the second and third layers are used as bottom-level features to characterize the vascular texture of small lesions; the features of the fourth and fifth layers are used as middle-level features to characterize the color boundary between the lesion and normal skin; and the features of the sixth layer are used as top-level features to characterize the overall morphology of large lesions.

3. The multimodal feature-based wine stain recognition and efficacy assessment system according to claim 2, characterized in that, In the adaptation branches, for small lesion sub-images, the weights of bottom-level features, middle-level features, and top-level features decrease sequentially; for large lesion sub-images, the weights of bottom-level features, middle-level features, and top-level features increase sequentially.

4. The multimodal feature-based system for identifying and evaluating the therapeutic effect of wine follicles according to claim 2, characterized in that, Also includes: The training set acquisition unit is used to acquire the training set. Size-adaptive training units are used to divide the training set into small lesion subsets and large lesion subsets according to the size of the lesions; Based on the small lesion subset, and with the parameters of the sixth layer of the main branch frozen, the fine mesh texture of the small lesions is learned by combining texture loss and cross-entropy loss. Based on the large lesion subset, and with the parameters of the second and third layers of the main branch frozen, the boundary of the large lesion is learned by combining boundary loss and cross-entropy loss. Based on the training set, the adaptation branch is trained with the goal of minimizing the mean squared error loss between the predicted weights and the true size labels. A site-specific pathology training unit is used to divide the training set into a periorbital subset, a nasal alar subset, a cheek subset, and a forehead subset according to facial sites. Based on the periorbital and nasal alar subsets, and with the parameters of the second and third layers thawed, the parameters of the sixth layer are fine-tuned based on texture loss, boundary loss, and size weight loss. Based on the cheek and forehead subsets, and with the parameters of the sixth layer thawed, the parameters of the second and third layers are fine-tuned based on boundary loss, texture loss, and size weight loss.

5. The multimodal feature-based system for identifying and evaluating the therapeutic effect of wine follicles according to claim 1, characterized in that, The classification unit includes a main output layer and an auxiliary output layer. The main output layer is used to obtain the erythema classification probability based on the fused feature vector, and the auxiliary output layer is used to obtain the pathological feature matching score based on the fused feature vector.

6. The multimodal feature-based system for identifying and evaluating the therapeutic effect of wine follicles according to claim 5, characterized in that, The rules in the classification unit include: If the pathological feature matching score and the probability of port-wine stain in the erythema classification probability are greater than the first threshold configuration, it is classified as port-wine stain; If the pathological feature matching score and the probability of port-wine stain in the erythema classification probability are greater than the second threshold configuration and less than the first threshold configuration, then secondary verification is triggered. If the pathological feature matching score and the probability of wine stain in the erythema classification probability are less than the second threshold configuration, then it is classified as a non-wine stain.

7. The multimodal feature-based system for identifying and evaluating the therapeutic effect of wine follicles according to claim 1, characterized in that, The color fading rate of the port-wine stains is calculated as: (average area of ​​interest before treatment - average area of ​​interest after treatment) / average area of ​​interest before treatment × 100%; The area reduction rate is: (actual area before treatment - actual area after treatment) / actual area before treatment × 100%, where the actual area is calculated by combining the pixel area of ​​PWS lesions in high-definition visible light images of the face with the actual size of facial anatomical feature points. The thickness improvement rate is: (average thickness before treatment - average thickness after treatment) / average thickness before treatment × 100%, where the thickness is obtained based on the optical coherence tomography lesion thickness data.

8. The multimodal feature-based system for identifying and evaluating the therapeutic effect of wine follicles according to claim 1, characterized in that, The structured clinical text data includes age, skin type, facial location of erythematous lesions, and time of onset.

9. An electronic device, characterized in that, It includes one or more processors, memory, and the multimodal feature-based wine stain recognition-treatment assessment system as described in any one of claims 1-8.

Images