Grape wine color spot identification-curative effect evaluation system based on multi-modal characteristics and electronic equipment

Abstract

The invention relates to a wine color spot recognition-curative effect evaluation system based on multi-modal characteristics and electronic equipment, and the system comprises a multi-modal data collection module which is used for collecting multi-modal data of a face; the data preprocessing module is used for carrying out quality detection, noise elimination and standardization processing on the multi-modal data; the erythema recognition module is used for outputting a facial erythema classification result based on the preprocessed multi-modal data by adopting a mixed model combining pathological guidance, multi-scale vision and semantic enhancement text; the feature linkage module is used for establishing a mapping relation between PWS identification features and curative effect evaluation features: if the facial erythema classification result output by the erythema identification module is wine stains, extracting curative effect evaluation core features; and the curative effect evaluation module is used for calculating the color fading rate, the area reduction rate and the thickness improvement rate of the wine color spots based on the curative effect evaluation core features and the same type of multi-modal data after treatment, and outputting a comprehensive curative effect evaluation result.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a multimodal feature-based system for identifying and evaluating the therapeutic effect of wine stains, and an electronic device. Background Technology

[0002] Port wine stain (PWS), also known as hemangioma, is a congenital capillary malformation that can be detected at birth. As the child grows older, the lesion darkens in color and expands in size. In adulthood, it may develop into a hypertrophic or nodular type. The face is the most common site of occurrence, and it is easily confused with hemangiomas, allergic purpura, and other erythematous diseases. Currently, there is an urgent need for accurate identification and objective assessment methods.

[0003] Currently, the identification of PWS largely relies on experience, which is affected by individual differences in experience, visual fatigue, and atypical early lesion features, resulting in a high misjudgment rate. Dermoscopy is non-invasive but requires specialized skills, making it difficult to implement at the grassroots level. Computer-aided identification has low accuracy in scenarios with hair occlusion, uneven lighting, and differences in skin type.

[0004] For the efficacy evaluation of PWS, the visual analog scale (VAS) method uses subjective scoring, lacks a unified standard, and has large differences in scores, making it impossible to distinguish multi-dimensional efficacy. The computer vision solution is easily affected by complex facial backgrounds and shooting conditions.

[0005] In summary, the current wine stain identification and treatment evaluation protocol has the following drawbacks: (1) Lack of multimodal data fusion capability; a single modality cannot fully capture the information required for identification and evaluation.

[0006] (2) The identification and evaluation processes are separated, the data cannot be reused, the efficiency is low and the evaluation is prone to errors due to misjudgment.

[0007] (3) It does not adapt to individual differences in PWS and complex facial scenarios, and the assessment lacks specificity.

[0008] In summary, there is currently a lack of a system for identifying and assessing the treatment efficacy of wine follicles to solve or partially solve the aforementioned problems. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art by providing a multimodal feature-based system and electronic device for identifying and evaluating the therapeutic effect of wine spots, so as to solve or partially solve the problem that wine spots are easily confused with other types of erythema, which can easily lead to incorrect evaluation.

[0010] The objective of this invention can be achieved through the following technical solutions: One aspect of the present invention provides a multimodal feature-based system for identifying and evaluating the treatment efficacy of wine follicles, comprising: A multimodal data acquisition module is used to acquire multimodal data of the face, including visible light images of the face, multispectral vascular imaging data, optical coherence tomography lesion thickness data, and structured clinical text data. The data preprocessing module is used to perform quality detection, noise removal, and standardization on the multimodal data; The erythema recognition module employs a hybrid model that combines pathological guidance, multi-scale vision, and semantically enhanced text. Based on preprocessed multimodal data, it outputs facial erythema classification results. The feature linkage module establishes a mapping relationship between PWS identification features and efficacy evaluation features: if the facial erythema classification result output by the erythema recognition module is port-wine stain, then the vascular density, initial a* value of the lesion (component in the CIELAB color space of the International Commission on Illumination), and initial thickness in the multimodal data are extracted as core features for efficacy evaluation; if the output facial erythema classification result is not port-wine stain, then the efficacy evaluation of port-wine stain is terminated. The efficacy assessment module calculates the color fading rate, area reduction rate, and thickness improvement rate of port-wine stains based on the core features of efficacy assessment and post-treatment multimodal data of the same type, and outputs a comprehensive efficacy assessment result.

[0011] As a preferred technical solution, the erythema recognition module includes: The multimodal data processing unit is used for image cropping based on facial anatomy for visible light images of the face, and for term extraction and labeling for structured clinical text data. The multi-scale visual feature pyramid unit includes a dual-branch multi-scale visual encoder, which obtains multi-scale features through aggregation. The main branch is used to extract features from the cropped facial sub-image, and the adaptation branch is used to dynamically allocate feature weights at each scale based on the anatomical partitions of the facial sub-image. The wine stain text semantic enhancement unit is used to generate text features based on a pre-built wine stain dictionary; The cross-modal attention unit embedded in pathological features is used to calculate the matching degree between multi-scale features and feature templates, and between text features and feature templates, based on multiple pre-set feature templates, to obtain cross-modal attention weights. Based on the cross-modal attention weights, multi-scale features and text features are fused to obtain a fused feature vector, thereby realizing pathological guidance. The classification unit is used to output erythema classification results and confidence scores based on the fused feature vector and according to preset rules.

[0012] As a preferred technical solution, the main branch outputs feature maps at six scales, wherein the features of the second and third layers are used as bottom layer features to characterize the vascular texture of small lesions; the features of the fourth and fifth layers are used as middle layer features to characterize the color boundary between the lesion and normal skin; and the features of the sixth layer are used as top layer features to characterize the overall morphology of large lesions.

[0013] As a preferred technical solution, in the adaptation branch, for small lesion sub-images, the weights of bottom-level features, middle-level features, and top-level features decrease sequentially; for large lesion sub-images, the weights of bottom-level features, middle-level features, and top-level features increase sequentially.

[0014] As a preferred technical solution, the following are also included: The training set acquisition unit is used to acquire the training set. A size adaptation training unit is used to divide the training set into small lesion subsets and large lesion subsets according to the size of the lesions. Based on the small lesion subset, the fine mesh texture of the small lesions is learned by combining texture loss and cross-entropy loss, while freezing the parameters of the sixth layer of the main branch. Based on the large lesion subset, the boundary of the large lesions is learned by combining boundary loss and cross-entropy loss, while freezing the parameters of the second and third layers of the main branch. Based on the training set, the adaptation branch is trained with the goal of minimizing the mean squared error loss between the predicted weights and the true size labels. A site-specific pathology training unit is used to divide the training set into a periorbital subset, a nasal alar subset, a cheek subset, and a forehead subset according to facial sites. Based on the periorbital and nasal alar subsets, and with the parameters of the second and third layers thawed, the parameters of the sixth layer are fine-tuned based on texture loss, boundary loss, and size weight loss. Based on the cheek and forehead subsets, and with the parameters of the sixth layer thawed, the parameters of the second and third layers are fine-tuned based on boundary loss, texture loss, and size weight loss.

[0015] As a preferred technical solution, the classification unit includes a main output layer and an auxiliary output layer. The main output layer is used to obtain the erythema classification probability based on the fused feature vector, and the auxiliary output layer is used to obtain the pathological feature matching score based on the fused feature vector.

[0016] As a preferred technical solution, the rules in the classification unit include: If the pathological feature matching score and the probability of port-wine stain in the erythema classification probability are greater than the first threshold configuration, it is classified as port-wine stain; If the pathological feature matching score and the probability of port-wine stain in the erythema classification probability are greater than the second threshold configuration and less than the first threshold configuration, then secondary verification is triggered. If the pathological feature matching score and the probability of wine stain in the erythema classification probability are less than the second threshold configuration, then it is classified as a non-wine stain.

[0017] As a preferred technical solution, the color fading rate of the port-wine stain is: (average value of the port-wine stain area of ​​interest before treatment - average value of the port-wine stain area of ​​interest after treatment) / average value of the port-wine stain area of ​​interest before treatment × 100%; The area reduction rate is: (actual area before treatment - actual area after treatment) / actual area before treatment × 100%, where the actual area is calculated by combining the pixel area of ​​PWS lesions in high-definition visible light images of the face with the actual size of facial anatomical feature points. The thickness improvement rate is: (average thickness before treatment - average thickness after treatment) / average thickness before treatment × 100%, where the thickness is obtained based on the optical coherence tomography lesion thickness data.

[0018] As a preferred technical solution, the structured clinical text data includes age, skin type, facial location of erythematous lesions, and onset time.

[0019] In another aspect, the present invention provides an electronic device including one or more processors, a memory, and the aforementioned multimodal feature-based wine stain recognition-treatment evaluation system.

[0020] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Reduce the possibility of misjudgment caused by non-portion pigmentation: This invention sets up an erythema recognition module and adopts a hybrid model that combines pathological guidance, multi-scale vision and semantically enhanced text. Based on the preprocessed multimodal data, it outputs facial erythema classification results. After verifying that the input data is port-wine pigmentation, the feature linkage module performs subsequent efficacy evaluation core feature extraction, reducing the error in evaluation caused by misjudgment.

[0021] (2) High accuracy of wine stain verification: The erythema recognition module of the present invention includes a multi-scale visual feature pyramid unit, including a dual-branch multi-scale visual encoder, which obtains multi-scale features through aggregation. The main branch is used to extract features from the cropped facial sub-image, and the adaptation branch is used to dynamically allocate the feature weights of each scale based on the anatomical partition of the facial sub-image, thus solving the problem of poor adaptation of single scale. The erythema recognition module also includes a wine stain text semantic enhancement unit, which generates text features based on a pre-set wine stain dictionary, thus solving the problem of insufficient generalization of text semantics. The erythema recognition module also includes a cross-modal attention unit with pathological feature embedding, which obtains cross-modal attention weights based on multiple pre-set feature templates, thus solving the problem of feature fusion without pathological guidance. The above design improves the accuracy of wine stain verification.

[0022] (3) High adaptability of multi-scale visual feature pyramid unit: The main branch of the dual-branch multi-scale visual encoder of the present invention includes bottom layer, middle layer and top layer, which respectively represent the vascular texture of small lesions, the color boundary between lesions and normal skin and the overall shape of large lesions. In the training process, a two-stage training method of size adaptation training stage and site pathology training is adopted. First, the dataset is split according to the size of the lesion, and the bottom texture extraction of small lesions and the top boundary extraction of large lesions are strengthened respectively. Then, the facial parts are grouped and the pathological features of port-wine stains are incorporated to achieve synergistic optimization of size, location and pathology, and improve the feature extraction capability of multi-scale visual feature pyramid unit. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the multimodal feature-based wine stain recognition and efficacy assessment system in the embodiment. Figure 2 This is a schematic diagram of the erythema recognition module in the embodiment; Figure 3 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1 To address the problems existing in the aforementioned prior art, this embodiment provides a multimodal feature-based system for identifying and evaluating the treatment efficacy of port-wine stains. This system can be deployed on high-performance computers and servers. (See [link to documentation]). Figure 1 The system includes a multimodal data acquisition module, a data preprocessing module, a PWS identification module, a feature linkage module, and a efficacy evaluation module, which are connected in sequence. The functions of each module will be described in detail below: (1) Modal data acquisition module.

[0026] This module is used to collect multimodal data related to port-wine stains. In this embodiment, it includes at least high-resolution visible light images of the face, multispectral angiography data, optical coherence tomography (OCT) lesion thickness data, and structured clinical text data of the patient. The structured clinical text data includes at least the patient's age, skin type, specific facial location of the PWS lesion, and onset time. Lesion locations may include the periorbital area, cheek, nasal alae, and forehead.

[0027] It should be noted that the collection of the above data should be done with the patient's prior consent and under legal conditions.

[0028] For high-resolution visible light images of the face, the resolution should be no less than 300 dpi. A ring light source should be used to uniformly illuminate the face from the front, avoiding shadows on the sides of the cheeks and below the nose to reduce light and shadow interference in facial depressions. Images should include those before treatment, after each treatment, and during follow-up, covering the PWS lesion and an area at least 5 mm beyond its edge to avoid missing the boundaries of facial PWS.

[0029] For optical coherence tomography (OCT) data, the aim is to collect thickness data of the lesion area for possible hypertrophic / nodular lesions of facial PWS. The scanning accuracy should be no less than 10 μm, and it should also cover 5 mm outside the edge of the lesion to adapt to the uneven thickness of facial PWS.

[0030] For structured clinical text data of patients, considering the differences in the rate of PWS regression at different ages and the characteristics of skin color quantification benchmark, at least the patient's age, skin type, specific facial location of PWS lesions, onset time, and previous treatment history should be recorded.

[0031] During the acquisition of visible light images and optical coherence tomography (OCT) data, care should be taken to maintain a fixed posture when taking images, aligning with the outer canthi of both eyes and the root of the nose as a reference, to ensure that the facial PWS position does not shift before and after treatment.

[0032] The OCT scan path corresponds to the PWS lesion area in the high-resolution image to avoid image-thickness data misalignment, ensuring that subsequent steps can correlate visual and structural features.

[0033] (2) Data preprocessing module.

[0034] This module is used to perform quality inspection, noise reduction, and standardization processing on the collected multimodal data.

[0035] The module first detects the sharpness, exposure, and color reproduction of high-resolution visible light images, removing blurry / overexposed images to avoid affecting PWS color feature extraction. Then, denoising is performed, using Gaussian filtering to eliminate random noise and median filtering to remove salt-and-pepper noise caused by facial hair, overcoming the problem of PWS boundaries being obscured by hair around the eyes and forehead. Next, contrast enhancement is performed, using histogram equalization to enhance the color contrast between PWS lesions and normal skin, considering the smaller color difference between PWS and normal skin in patients with light skin. Finally, color space conversion is performed, converting the RGB image to the Lab color space.

[0036] Preferably, the process may also include a feature registration process, using fixed anatomical feature points on the face as coordinate references to establish a spatial mapping relationship between high-resolution images and OCT data before treatment, after treatment, and during follow-up, correcting for PWS lesion position shifts caused by changes in the patient's shooting posture, and ensuring that subsequent assessments are conducted on the same lesion and in the same region. The feature points can be points such as the outer canthi of both eyes, the root of the nose, and the base of the nose, which are not affected by changes in facial expressions.

[0037] (3) PWS identification module.

[0038] This module uses a hybrid model that integrates a visual encoder and a language encoder. It takes preprocessed multimodal data as input and outputs disease classification results, namely PWS or other erythematous diseases.

[0039] To address the issue of misclassification caused by confusion between facial rosacea (PWS) and other erythematous diseases, this module employs a hybrid architecture combining a dual-branch visual encoder and a text encoder. The visual encoder, built on EfficientNet-B4, aims to extract color and texture differences between facial PWS and other erythematous lesions, obtaining visual features from high-resolution visible light images. For example, PWS exhibits a reticular vascular texture, while hemangiomas show a clumpy texture. The text encoder, based on a BioBERT pre-trained model, aims to extract clinical textual differences such as onset time and skin type, obtaining semantic features from structured clinical text. For example, PWS is present at birth, while rosacea develops in adulthood. The two types of features are fused through a cross-modal attention mechanism, with the input to a fully connected layer and the output being the disease classification probability.

[0040] For details, see Figure 2 The erythema recognition module mainly includes a multimodal data processing unit, a parallel multi-scale visual feature pyramid unit, a wine stain text semantic enhancement unit, a cross-modal attention unit for embedding pathological features, and a classification unit, all connected in sequence. The functions of each unit will be described in detail below: 1) Multimodal data processing unit.

[0041] This unit is used for image cropping based on facial anatomy for visible light images and multispectral vascular imaging data, and for term extraction and labeling for structured clinical text data.

[0042] Specifically, for high-resolution visible light images of the face, morphological opening operations are used to remove facial hair, and sub-images are cropped according to facial anatomical regions, such as the periorbital area, cheeks, nasal alae, and forehead, to prepare for subsequent multi-scale feature extraction. For structured clinical texts of patients, specific terminology is extracted, pathological labels are added, and structured text vectors are formed. Labels may include vascular structures and onset time.

[0043] 2) Multi-scale visual feature pyramid unit.

[0044] This unit includes a dual-branch multi-scale visual encoder, which obtains multi-scale features through aggregation. The main branch is used to extract features from the cropped facial sub-images, and the adaptation branch is used to dynamically allocate feature weights for each scale based on the anatomical partitions of the facial sub-images.

[0045] For a dual-branch multi-scale visual encoder, it mainly includes a main branch based on EfficientNet-B4 and an adaptation branch: The main branch is used to extract features from the cropped facial sub-images, outputting feature maps at six scales, corresponding to layers 1 to 6. Layers 2 and 3 serve as the bottom layer features, capturing pixel-level vascular textures in small lesions such as those around the eyes; these features can be used to determine whether the lesion is reticular. Layers 4 and 5 serve as the middle layer features, capturing the color boundaries between lesions and normal skin; this setup takes into account the clear boundaries of PWS and the blurred boundaries of hemangiomas. Layer 6 serves as the top layer features, capturing the overall morphology of large lesions such as those on the cheeks, used to determine whether they are proportionally distributed according to anatomical regions.

[0046] The adaptation branch aims to predict lesion size by dynamically allocating feature weights at each scale based on the anatomical divisions of the facial sub-image. For example, the periorbital sub-image corresponds to small lesions. For small lesion sub-images: bottom layer feature weight > middle layer feature weight > top layer feature weight. For example, the bottom layer feature weight is 60%, the middle layer is 30%, and the top layer is 10%. This setting prioritizes capturing reticular textures. For large lesion sub-images: bottom layer feature weight < middle layer feature weight < top layer feature weight. For example, the top layer feature weight is 50%, the middle layer is 35%, and the bottom layer is 15%. This setting prioritizes capturing boundary morphology.

[0047] By feature aggregation, the weighted multi-scale features are compressed into PWS visual feature vectors, in which the signal intensity of key PWS visual features such as reticular vascular texture and clear boundaries is improved.

[0048] 3) Text semantic enhancement unit for wine stains.

[0049] This unit is used to generate text features based on a pre-built dictionary of wine stains.

[0050] Specifically, this unit first inputs the PWS structured text preprocessed by the multimodal data processing unit into BioBERT. A pre-built PWS-specific dictionary is loaded into the embedding layer to improve the cosine distance between the embedding vectors of terms such as "present at birth" and "color deepens with age" and other erythematous disease terms, achieving dedicated dictionary embedding. Then, the pathological labels annotated by the multimodal data processing unit are used as enhancement factors for text features, amplifying the features of text fragments containing these labels to achieve pathological label association. For example, with the pathological label "vascular structure - reticular," if the text fragment is "reticular blood vessels are visible under dermoscopy," the corresponding feature is amplified. Finally, a fully connected layer reduces the dimensionality of the text features to PWS text feature vectors, improving the discriminative power of PWS-specific text features such as "onset at birth" and "proportional enlargement with body size."

[0051] 4) Cross-modal attention units embedded with pathological features.

[0052] This unit is used to calculate the matching degree between multi-scale features and feature templates, and between text features and feature templates, based on multiple pre-set feature templates, to obtain cross-modal attention weights. Based on the cross-modal attention weights, multi-scale features and text features are fused to obtain a fused feature vector, thereby achieving pathological guidance.

[0053] This unit first loads the pre-built PWS-specific pathological feature library and outputs feature templates for three core pathological labels, including a visual template for "reticular vascular texture" and a text template for "present at birth". Then, it calculates the attention weights. Specifically, it calculates the matching degree between the visual feature vector and the visual pathological template, and the matching degree between the text feature vector and the text pathological template. The two matching degrees are weighted and summed as the cross-modal attention weights. The weights here can be adjusted according to the actual situation.

[0054] Finally, this unit performs weighted fusion of visual and textual features to generate a 3072-dimensional PWS discrimination fusion feature vector, in which the proportion of key pathological features of PWS is significantly improved compared with existing models.

[0055] 5) Classification unit.

[0056] This unit is used to output erythema classification results and confidence scores based on fused feature vectors and according to preset rules.

[0057] The classifier in this unit employs a dual-output fully connected layer, consisting of a main output layer and an auxiliary output layer. The main output layer outputs the probabilities of four categories: PWS, hemangioma, rosacea, and other erythematous lesions. The auxiliary output layer outputs the pathological feature matching scores, such as the overall matching degree for reticular vessels, congenital lesions, and proportionally enlarged lesions.

[0058] After calculating the probabilities of the four categories and the overall matching degree, classification is performed based on classification rules. If the PWS probability is ≥95% and the pathological feature matching score is ≥80, it is directly determined as PWS; if the PWS probability is 85%-94% and the pathological feature matching score is ≥70, secondary verification is triggered, for example, by calling vascular density data from multispectral imaging to assist in the judgment; all other cases are determined as non-PWS. The above threshold configuration can be adjusted according to the actual situation.

[0059] Finally, this unit outputs the disease classification results and confidence levels, while also annotating key identification criteria, such as visual: reticular vessels; text: present at birth.

[0060] Furthermore, the dual-branch multi-scale visual encoder, a key focus of this embodiment, is crucial for achieving effective identification and treatment evaluation. To address this, this embodiment also constructs a training set acquisition unit, a size adaptation training unit, and a site pathology training unit to pre-train the dual-branch multi-scale visual encoder.

[0061] 6) Training set acquisition unit.

[0062] Multiple high-resolution visible light images of facial PWS were acquired, covering different sizes and locations, including small lesions with an area <1cm² located around the eyes and nasal alae. 2 It also includes large lesions located on the cheeks and forehead, with an area >5cm². 2 The areas represented include around the eyes, nose, cheeks, and forehead. Additionally, images with background interference, such as hair occlusion or uneven lighting, are used to simulate realistic scenes.

[0063] Image annotation, in this embodiment, was performed by a senior dermatologist. Annotation dimensions included: size annotation (using rectangles to mark the extent of lesions, categorizing lesions as small or large by area); location annotation (categorizing lesions based on facial anatomical landmarks, such as the periorbital area, nasal alae, cheek, and forehead); pathological texture annotation (using pixel-level masks to annotate vascular texture areas, labeling them with fine or coarse mesh patterns); and background interference annotation (annotating areas obscured by hair and areas with uneven lighting, used for targeted suppression of interfering features during training).

[0064] The dataset is divided into training, validation, and test sets in a 7:2:1 ratio to ensure that the size, location, and disease distribution of each subset are consistent with the total dataset, thus avoiding data bias.

[0065] 7) Size-adaptive training unit.

[0066] The model is initialized before training.

[0067] For the main branch, it is built based on the pre-trained EfficientNet-B4, retaining its 6-layer feature extraction structure.

[0068] For the adaptation branch, two fully connected layers and a Softmax layer are added. The input is the facial region label, and the output is the weight coefficients for small lesions and large lesions. For example, for the facial region label of the periorbital area, the output is a weight of 0.8 for small lesions and a weight of 0.2 for large lesions, and for the facial region label of the cheek, the output is a weight of 0.2 for small lesions and a weight of 0.8 for large lesions. This is used to dynamically adjust the feature weights of the main branch later.

[0069] The pre-trained parameters of the main branch are retained, and the convolution kernel parameters of layers 2-3 and 6 are locally unfrozen to adapt to PWS features. The adapted branch parameters are randomly initialized using a He normal distribution.

[0070] The goal of this unit is to enable different layers of the main branch to adapt to the feature extraction requirements of small and large lesions, and to adapt the branch learning size-weight mapping relationship. The unit first splits the training set into small lesion subsets and large lesion subsets, and trains them separately.

[0071] For training on a subset of small lesions, the aim is to enhance the texture extraction capability of the bottom layer of the main branch. Top-level parameters are frozen, and the input consists of small lesion images and labels. The loss function is primarily texture loss (SSIM), supplemented by cross-entropy loss; specific weights can be set according to actual conditions. By calculating the similarity between the extracted bottom-level texture and the labeled mesh texture mask, the model is forced to learn the fine mesh texture of the small lesions. In this embodiment, the optimizer is AdamW, with a learning rate of 1e-4 and 50 iterations, each iteration verifying the accuracy of small lesion texture extraction.

[0072] For training on a subset of large lesions, the aim is to enhance the boundary extraction capability of the top layer of the main branch while freezing the bottom layer parameters. The input consists of images and labels of large lesions. The loss function primarily uses boundary loss (Dice), supplemented by cross-entropy loss; specific weights can be set according to actual conditions. By calculating the overlap between the extracted top-level boundaries and the labeled lesion bounding boxes, the model is forced to learn the clear boundaries of large lesions. The optimizer in this embodiment also uses AdamW, with a learning rate of 1e-4 and 50 iterations, validating the IOU of the large lesion boundaries in each iteration.

[0073] The adaptation branch is trained synchronously by taking facial part labels as input and outputting size weight coefficients. The training is based on the mean square error loss between the predicted weights and the actual size labels, so that the adaptation branch can automatically output reasonable size weights according to the parts.

[0074] 8) Site-specific pathology training unit.

[0075] This unit aims to optimize the feature fusion logic of the two branches by combining the pathological differences of different facial regions in PWS, and improve the site adaptability of feature extraction. This unit first divides the training set into 4 groups according to facial regions: periorbital group, nasal alar group, cheek group, and forehead group. Each group of samples contains small / large lesions in the corresponding region.

[0076] For the periorbital and nasal alar groups, which are characterized by fine mesh textures, the bottom-level parameters of the main branch are thawed and the top-level parameters are fine-tuned. The input is periorbital / nasal alar images and fine mesh texture pathological labels. The loss function includes texture loss, boundary loss and size weight loss, with a weight of 1.5 times for small lesions to enhance the extraction of fine mesh textures.

[0077] For the cheek / forehead group, which is characterized by coarse mesh texture and clear boundaries, the top-level parameters of the main branch are unfrozen and the bottom-level parameters are fine-tuned. The input is the cheek / forehead image and pathological labels with coarse mesh texture and clear boundaries. The loss function includes boundary loss, texture loss and size weight loss, with a weight of 1.2 times for large lesions to enhance coarse texture and boundary extraction.

[0078] Furthermore, it is also used for cross-branch feature fusion training. In each training group, the size weight coefficient of the adapted branch output is dynamically adjusted to adjust the feature weight of the main branch. For example, if the weight of small lesions in the periorbital group is 0.8, then the bottom-level features of the main branch contribute 80%. The matching loss between the fused features and the pathological labels is calculated to optimize the fusion logic.

[0079] This unit uses the AdamW optimizer with a learning rate of 5e-5 and 80 iterations. In each iteration, the feature extraction accuracy is verified by part.

[0080] The loss functions involved in the size adaptation training unit and the site pathology training unit will be explained below.

[0081] Texture loss is obtained by calculating the structural similarity between the extracted vascular texture features and the labeled mesh texture template.

[0082] Boundary loss (boundary Dice) calculates the Dice coefficient for each lesion boundary pixel individually, where the boundary pixel can be a pixel within a 5-pixel range of the labeled bounding box.

[0083] The size-weighted loss dynamically adjusts the loss weight based on the sample size, with a loss of 1.5 times for small lesion samples and a loss of 1.2 times for large lesion samples, to avoid diluting the number of small lesion samples during training due to their proportion.

[0084] Through joint training by two units, the weights are adjusted every 10 rounds based on the feature extraction accuracy of the validation set until the overall feature extraction accuracy of the validation set reaches the target accuracy, at which point the loss weight optimization stops.

[0085] (4) Feature linkage module.

[0086] This module establishes a mapping relationship between PWS identification features and efficacy evaluation features. If the PWS identification module outputs PWS, it extracts vascular density from multimodal data based on multispectral vascular imaging data, calculates the initial a* value of the lesion in Lab space based on high-resolution visible light facial images, and calculates the initial thickness based on optical coherence tomography as the core feature for efficacy evaluation. If the output is not PWS, the PWS efficacy evaluation process is terminated and the feature matching logic for the corresponding erythema disease is triggered.

[0087] Preferably, if the output is not PWS, the feature matching logic for the corresponding erythema disease is triggered, and the corresponding efficacy evaluation logic is executed. This part is not the focus of this embodiment and is omitted here.

[0088] (5) Therapeutic efficacy assessment module.

[0089] This module is used to quantitatively calculate the color fading rate, area reduction rate, and thickness improvement rate of PWS based on the core features of efficacy assessment and similar multimodal data after treatment, and output the comprehensive efficacy assessment results.

[0090] 1. Calculation of color fading rate.

[0091] Color fading rate is used as the core indicator to reflect the degree of vascular regression in PWS. The core indicator uses the U-Net++ segmentation model to automatically select the region of interest (ROI) of PWS lesions in high-resolution images, avoiding deviations in the extent of facial PWS caused by manual selection.

[0092] Extract the average a* value within the ROI before and after treatment, which is the red component in the Lab space, and calculate the color fading rate using the formula: (a* value before treatment - a* value after treatment) / a* value before treatment × 100%.

[0093] Preferably, weighting is applied based on differences in facial location. If PWS is located in areas where color is more sensitive to therapeutic effects, such as around the eyes, the weight of this indicator is increased. If it is located on the cheeks, where thickness has a greater impact, the weight is decreased.

[0094] 2. Calculation of area reduction rate.

[0095] The area reduction rate reflects the change in the extent of PWS lesions. The pixel area of ​​PWS lesions in high-resolution images before and after treatment was extracted using the U-Net++ segmentation model.

[0096] Based on the actual size of facial anatomical feature points in the data preprocessing module, the pixel area is converted into the actual area, and the area reduction rate is calculated according to the formula: (actual area before treatment - actual area after treatment) / actual area before treatment × 100%.

[0097] 3. Calculation of thickness improvement rate.

[0098] The thickness improvement rate setting is designed for hypertrophic / nodular facial lesions. PWS extracts the average thickness of PWS lesions before and after treatment from OCT data, and calculates the thickness improvement rate according to the formula: (average thickness before treatment - average thickness after treatment) / average thickness before treatment × 100%.

[0099] 4. Overall scoring and grading.

[0100] Color fading rate, area reduction rate, and thickness improvement rate are input into a multilayer perceptron (MLP) or XGBoost model with site-specific weights, outputting a comprehensive efficacy score from 0 to 100. The score is divided into 4 levels: Excellent: >75 points, PWS color fades significantly, area is greatly reduced, and thickness is significantly improved.

[0101] Good: 50-75 points, all three indicators show moderate improvement.

[0102] Moderate: 25-50 points, with at least one indicator showing slight improvement.

[0103] Poor: <25 points, with no improvement or deterioration in any of the three indicators.

[0104] 5. Visualized report output.

[0105] Generate a dedicated facial PWS report, containing four core categories for easy and intuitive interpretation: Lesion annotation map: The PWS range is marked with a bright border on the high-resolution image to distinguish changes in facial lesions before and after treatment.

[0106] Color fading heatmaps show the changes in a* values ​​in different areas within the lesion, with red, yellow, and green indicating increasing degrees of fading.

[0107] OCT thickness comparison profile, overlaid with lesion thickness curves before and after treatment, visually demonstrates the thickness improvement.

[0108] The comprehensive assessment form lists the values ​​of three indicators, the overall score, and the results of the efficacy grading.

[0109] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0110] Example 2 Based on Embodiment 1, this embodiment provides an electronic device, such as... Figure 3 As shown, it includes one or more processors, memory, and the multimodal feature-based wine stain recognition-treatment assessment system of Embodiment 1. Preferably, the electronic device also includes a visualization terminal for displaying a visualization report.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-modal feature based wine color stain identification-efficacy evaluation system, characterized in that, The method comprises the steps of: a multi-modal data acquisition module for acquiring multi-modal data of a face, the multi-modal data comprising facial visible light images, multi-spectral vascular imaging data, optical coherence tomography lesion thickness data, and structured clinical text data; a data preprocessing module for quality detection, noise removal, and standardization processing of the multi-modal data; a erythema recognition module, which adopts a hybrid model combining pathological guidance, multi-scale vision, and semantic enhanced text, and outputs a facial erythema classification result based on the preprocessed multi-modal data; a feature linkage module for establishing a mapping relationship between PWS identification features and efficacy evaluation features: if the facial erythema classification result output by the erythema recognition module is a port-wine stain, then the vascular density, initial a* value, and initial thickness in the multi-modal data are extracted as core features for efficacy evaluation; if the facial erythema classification result output is not a port-wine stain, then the port-wine stain efficacy evaluation is terminated; an efficacy evaluation module for calculating the color fading rate, area reduction rate, and thickness improvement rate of the port-wine stain based on the core features for efficacy evaluation and the same type of multi-modal data after treatment, and outputting a comprehensive efficacy evaluation result.

2. The wine stain identification-efficacy evaluation system based on multi-modal features according to claim 1, characterized in that, The erythema recognition module comprises: a multi-modal data processing unit for image cropping processing based on facial anatomy for facial visible light images, and term extraction and label annotation processing for structured clinical text data; a multi-scale vision feature pyramid unit comprising a double-branch multi-scale vision encoder, which obtains multi-scale features by aggregation, wherein the main branch is used for feature extraction of the cropped facial subgraph, and the adaptive branch is used for dynamically allocating feature weights of each scale based on anatomical partitioning of the facial subgraph; a port-wine stain text semantic enhancement unit for generating text features based on a pre-set port-wine stain dictionary; a pathological feature embedded cross-modal attention unit for calculating the matching degrees of multi-scale features and feature templates, and text features and feature templates based on a plurality of pre-set feature templates, obtaining cross-modal attention weights, fusing multi-scale features and text features based on the cross-modal attention weights, and obtaining a fusion feature vector to realize pathological guidance; a classification unit for outputting an erythema classification result and a confidence level based on the fusion feature vector according to a pre-set rule.

3. The wine stain identification-efficacy evaluation system based on multi-modal features according to claim 2, characterized in that, The main branch outputs six scale feature maps, wherein the features of the second layer and the third layer are used as bottom layer features for representing the vascular texture of small lesions; the features of the fourth layer and the fifth layer are used as middle layer features for representing the color boundary between lesions and normal skin; and the features of the sixth layer are used as top layer features for representing the overall morphology of large lesions.

4. The wine stain identification-efficacy evaluation system based on multi-modal features according to claim 3, characterized in that, In the adaptive branch, the weights of the bottom layer features, the middle layer features, and the top layer features decrease in turn for small lesion subgraphs, and increase in turn for large lesion subgraphs.

5. The multi-modal feature based wine color stain identification-efficacy evaluation system as claimed in claim 3, wherein, Further comprising: a training set acquisition unit for acquiring a training set; a size adaptation training unit for dividing the training set into a small lesion subset and a large lesion subset according to lesion size; based on the small lesion subset, learning the fine reticular texture of small lesions by combining texture loss and cross-entropy loss under the premise of freezing the sixth layer parameters of the main branch; Based on the large lesion subset, under the premise of freezing the second and third layer parameters of the main branch, the boundary of the large lesion is learned in combination with the boundary loss and the cross-entropy loss; Based on the training set, the adaptive branch is trained to minimize the mean square error loss of the predicted weight and the true size label; The part pathology training unit is used to divide the training set into an eye area subset, a nose wing subset, a cheek subset and a forehead subset according to the facial parts; based on the eye area subset and the nose wing subset, under the premise of unfreezing the second and third layer parameters, the parameters of the sixth layer are fine-tuned based on the texture loss, the boundary loss and the size weight loss; based on the cheek subset and the forehead subset, under the premise of unfreezing the sixth layer parameters, the parameters of the second and third layers are fine-tuned based on the boundary loss, the texture loss and the size weight loss.

6. The multi-modal feature based wine color stain identification-efficacy evaluation system as claimed in claim 2, wherein, The classification unit includes a main output layer and an auxiliary output layer, the main output layer is used to obtain a probability of erythema classification based on the fusion feature vector, and the auxiliary output layer is used to obtain a pathology feature matching score based on the fusion feature vector.

7. The multi-modal feature based wine color stain identification-efficacy evaluation system as claimed in claim 6, wherein, In the classification unit, the rules include: If the pathology feature matching score and the probability of wine-colored macule in the erythema classification probability are greater than the first threshold configuration, the wine-colored macule is classified; If the pathology feature matching score and the probability of wine-colored macule in the erythema classification probability are greater than the second threshold configuration and less than the first threshold configuration, secondary verification is triggered; If the pathology feature matching score and the probability of wine-colored macule in the erythema classification probability are less than the second threshold configuration, the non-wine-colored macule is classified.

8. The wine-colored macule identification-efficacy evaluation system based on multi-modal features according to claim 1, wherein the color fading rate of the wine-colored macule is: (average value of the wine-colored macule region of interest before treatment-average value of the wine-colored macule region of interest after treatment) / average value of the wine-colored macule region of interest before treatment*100%; the area reduction rate is: (actual area before treatment-actual area after treatment) / actual area before treatment*100%, wherein the actual area is obtained by converting the pixel area of the PWS lesion in the facial high-definition visible light image combined with the actual size of the facial anatomical feature points; the thickness improvement rate is: (average thickness before treatment-average thickness after treatment) / average thickness before treatment*100%, wherein the thickness is obtained based on the optical coherence tomography lesion thickness data. The structured clinical text data includes age, skin type, erythema lesion facial part and onset time. The wine-colored macule identification-efficacy evaluation system based on multi-modal features includes one or more processors, memories and the wine-colored macule identification-efficacy evaluation system based on multi-modal features according to any one of claims 1-9. ​ 9. The multi-modal feature based wine color stain identification-efficacy evaluation system as claimed in claim 1, wherein, ​ 10. An electronic device, comprising: ​

Images