Skin state evaluation method and device based on multi-modal data

Abstract

The invention belongs to the technical field of skin detection specific computer models, and particularly relates to a skin state evaluation method and device based on multi-modal data, and the method comprises the steps: obtaining facial image data, flora data and host phenotype data which are used for representing a skin state, and carrying out the preprocessing of the data, image features, flora features and host phenotypic features are obtained; constructing a classification model based on an XGBoost algorithm, inputting the image features, the flora features and the host phenotypic features into the classification model for iterative training, and verifying the model effect through five-fold cross validation to obtain a trained target classification model; and predicting samples in the test set by using the target classification model to obtain a positive class probability that each sample belongs to an ideal skin class, constructing an ideal skin index based on the positive class probability, and evaluating a skin health state according to the ideal skin index. According to the invention, the feature rule in the multi-modal data can be effectively captured, and the comprehensive evaluation of the skin health condition is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of specific computer models for skin detection, and in particular relates to a method and device for evaluating skin condition based on multimodal data. Background Art

[0002] As the largest organ in the human body, the skin serves as the primary barrier between the body and the external environment, performing numerous physiological functions, including protection, temperature regulation, sensation, secretion, and excretion. Skin health not only impacts appearance but is also closely linked to overall health. For example, impaired skin barrier function can lead to water loss and the intrusion of external irritants, potentially causing various skin problems such as eczema and dermatitis.

[0003] Recent studies have shown that the skin microbiome plays a vital role in maintaining skin health. A large number of specific microorganisms, including bacteria, fungi, and viruses, live on the skin surface, interacting with skin cells to form a complex ecosystem. A healthy skin microbiome can inhibit the growth of pathogens, maintain the acid-base balance on the skin surface, and participate in skin immune regulation. However, when the balance of the skin microbiome is disrupted, it may lead to the occurrence of various skin diseases such as acne, psoriasis, and atopic dermatitis. For example, the over-proliferation of Propionibacterium acnes is closely related to the occurrence of acne, while the colonization of Staphylococcus aureus may aggravate the symptoms of atopic dermatitis.

[0004] Traditional skin condition assessment methods rely primarily on visual inspection by doctors and subjective descriptions by patients, which are subject to limitations such as high subjectivity and low accuracy. In recent years, the rapid development of multimodal data fusion technology and machine learning algorithms has provided new approaches for objective and accurate assessment of skin condition.

[0005] Currently, image-based skin condition assessment technology utilizes computer vision to analyze skin images, extracting features such as texture, color, and pores for evaluation. For example, some research uses convolutional neural networks (CNNs) to classify skin images and automatically identify skin conditions such as acne and pigmentation.

[0006] For example, Chinese patent document CN116342549A discloses a skin condition detection method and apparatus, comprising: obtaining parallel polarized light brightness maps and cross polarized light brightness maps for multiple regions of a face to be evaluated; calculating a brightness segmentation threshold for each region based on the cross polarized light brightness map of each region; extracting an oiliness map for each region from the parallel polarized light brightness map based on the brightness segmentation threshold, wherein the oiliness map includes: first target pixels whose pixel values are greater than or equal to the brightness segmentation threshold corresponding to each region; calculating an oiliness evaluation parameter for each region based on the number of first target pixels in the oiliness map; calculating a highlight evaluation parameter for each region based on the brightness difference between pixels in the parallel polarized light brightness map of each region; and calculating a reflective state parameter for the face to be evaluated based on the oiliness evaluation parameters and highlight evaluation parameters of the multiple regions. This method can improve the accuracy of skin condition detection.

[0007] Meanwhile, microbiome-based skin condition assessment technologies assess skin health by analyzing the composition and function of the skin's surface microbiome. For example, a study using 16S rRNA sequencing technology to analyze the skin microbiome found an association between specific bacterial communities and skin diseases. Furthermore, phenomics-based skin condition assessment technologies assess skin condition by analyzing the physical, chemical, and biological characteristics of the skin surface. For example, a study used skin moisture testers, sebum testers, and other devices to measure skin moisture, oil content, and other indicators to assess skin barrier function.

[0008] However, existing technologies and research still face numerous challenges. First, the limitations of single-modality data make assessments incomplete and inaccurate. For example, while image analysis-based methods can intuitively reflect surface skin issues, they cannot capture intrinsic issues such as microbiome imbalance or impaired skin barrier function. Similarly, microbiome-based methods can reveal changes in the microbiome but struggle to directly correlate them with phenotypic characteristics of the skin surface.

[0009] Secondly, traditional machine learning models perform poorly when processing high-dimensional, nonlinear, and multimodal data, resulting in limited model prediction accuracy and generalization capabilities. For example, traditional algorithms such as support vector machines (SVMs) and random forests are prone to overfitting or underfitting when processing multimodal data, making it difficult to effectively capture the complex relationships between data. In addition, most existing studies are limited to a single task or a single disease and lack a holistic assessment of skin condition. For example, some studies only focus on specific skin problems such as acne or eczema, while ignoring the multidimensionality and complexity of skin conditions. Finally, existing methods lack personalized assessment capabilities. Because skin condition is affected by multiple factors such as genetics, environment, and lifestyle, skin problems may vary significantly among different individuals, and existing technologies struggle to provide targeted assessment results. Summary of the Invention

[0010] The present invention aims to overcome at least one of the above-mentioned defects of the prior art and provide a skin condition assessment method based on multimodal data.

[0011] The present invention also discloses a device loaded with a skin condition assessment method based on multimodal data.

[0012] The detailed technical solutions of the present invention are as follows: A skin condition assessment method based on multimodal data, the method comprising: S1. Acquire facial image data, bacterial flora data, and host phenotypic data for characterizing skin conditions, and preprocess the facial image data, bacterial flora data, and host phenotypic data to obtain corresponding image features, bacterial flora features, and host phenotypic features; S2. Build a classification model based on the XGBoost algorithm, and input the image features, bacterial flora features, and host phenotypic features as training samples into the classification model for iterative training. Test the model effect through five-fold cross-validation to obtain a trained target classification model. S3. Use the target classification model to predict the samples in the test set to obtain the positive probability that each sample belongs to the ideal skin category, construct an ideal skin index based on the positive probability, and evaluate the skin health status based on the ideal skin index.

[0013] Preferably, in S1, the facial image data is 3648*5472 pixels, and preprocessing the facial image data specifically includes: The facial image data is cropped, scaled to a shorter side of 256 pixels, and a 224*224 pixel area is cropped from the center; Convert the cropped facial image data into a PyTorch tensor and scale the pixel values to [0.0, 1.0]; Normalizing the converted facial image data to conform to the distribution of the ImageNet dataset; Load the weights of the ResNet-18 model pre-trained on the ImageNet dataset, delete its last fully connected layer, and set the ResNet-18 model to evaluation mode to make it a feature extractor; Name the normalized facial image data with a sample ID, and set up a txt file to record the sample ID and its order; The facial image data and its txt file are input into the ResNet-18 model, processed by the convolution layer, pooling layer and residual block, and finally a 224-dimensional feature vector is extracted.

[0014] According to a preferred embodiment of the present invention, in S1, preprocessing the flora data specifically includes: The original sequences of the bacterial community data were denoised and de-chimerized using PMS software to remove low-quality sequences; The retained sequences were aligned with the reference database GreenGenes using the Wearch tool for species annotation, and the alignment results were used to calculate the relative abundance of each taxon; The relative abundance was adjusted by copy number correction to obtain the relative abundance at the microbial genus level and generate a microbiome feature matrix.

[0015] According to a preferred embodiment of the present invention, in S1, preprocessing the host phenotypic data specifically includes: performing grouping and discretization processing on the host phenotypic data to obtain grouped data; The grouped data is converted into a binary feature vector using a one-hot encoding method.

[0016] Preferably, according to the present invention, in S2, the image features, bacterial flora features, and host phenotypic features are input as training samples into the classification model for iterative training, specifically including: The importance index of each flora feature of the flora data is calculated based on the XGBoost algorithm, and the optimal 13 flora feature combinations are screened out using the forward feature selection method; Calculate the importance index of each host phenotypic feature of the host phenotypic data based on the XGBoost algorithm, and screen out 24 phenotypic feature sets that are more relevant to the set target task; The screened bacterial flora feature combination, phenotypic feature set and image features are subjected to multimodal data fusion to form a multi-source feature data set, and the multi-source feature data is used to train the XGBoost algorithm, which outputs the optimal feature combination as the target data set.

[0017] Preferably, according to the present invention, in S2, the model effect is tested by five-fold cross validation, specifically including: using a five-fold cross validation method, dividing the target data set into five mutually exclusive subsets, taking one of the subsets as a test set and the rest as training sets in turn, and repeating five times to verify the model effect of the XGBoost algorithm, while adjusting the parameters of the XGBoost algorithm during the iteration process, and finally using the trained XGBoost algorithm as the target classification model.

[0018] According to a preferred embodiment of the present invention, in S3, constructing an ideal skin index based on the positive class probability specifically includes: Based on the positive class probability of each sample belonging to the ideal skin category output by the target classification model, a probability distribution of the ideal skin group and a probability distribution of the non-ideal skin group are obtained; The midpoint between the 75% quantile of the probability distribution of the ideal skin group and the 25% quantile of the probability distribution of the non-ideal skin group was selected as the demarcation threshold; The positive class probability is linearly transformed, that is, the positive class probability is compared with a cutoff threshold. If it is lower than the cutoff threshold, it is proportionally reduced; if it is higher than the cutoff threshold, it is proportionally amplified to enhance the discrimination and interpretability to form an ideal skin index.

[0019] In another aspect of the present invention, a device for implementing a skin condition assessment method based on multimodal data is provided, the device comprising: a data acquisition module, configured to acquire facial image data, bacterial flora data, and host phenotypic data for characterizing skin conditions, and preprocess the facial image data, bacterial flora data, and host phenotypic data, respectively, to obtain corresponding image features, bacterial flora features, and host phenotypic features; A model building module is used to build a classification model based on the XGBoost algorithm, and input the image features, bacterial flora features and host phenotypic features as training samples into the classification model for iterative training, and test the model effect through five-fold cross validation to obtain a trained target classification model; The skin condition assessment module is used to use the target classification model to predict the samples in the test set, obtain the positive class probability that each sample belongs to the ideal skin category, construct an ideal skin index based on the positive class probability, and assess the skin health status based on the ideal skin index.

[0020] In another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to perform the skin condition assessment method based on multimodal data as described above.

[0021] In another aspect of the present invention, a machine-readable storage medium is provided, which stores executable instructions. When the instructions are executed, the machine performs the skin condition assessment method based on multimodal data as described above.

[0022] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention provides a skin condition assessment method based on multimodal data, which integrates multimodal information such as images, microbial and host phenotypic data, and uses the XGBoost machine learning algorithm to establish a skin condition assessment classification model through supervised learning. The classification model is iteratively optimized and trained using the integrated multimodal data, so that it can effectively capture the characteristic patterns in the multimodal data and achieve a comprehensive assessment of skin health status.

[0023] (2) This invention innovatively integrates multi-source data and achieves accurate quantitative assessment of skin condition through machine learning algorithms. This method has the characteristics of comprehensive assessment dimensions and objective and accurate results, and can provide reliable data support and new research directions for personalized skin care, skin disease assessment and cosmetics development. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 4 is a flowchart of the skin condition assessment method based on multimodal data according to the present invention.

[0025] Figure 2 3 is a receiver operating characteristic (ROC) curve diagram drawn in Example 1 of the present invention.

[0026] Figure 3 This is a distribution diagram of the ideal skin index of the ideal skin group and the non-ideal skin group in Example 1 of the present invention. DETAILED DESCRIPTION

[0027] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0029] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0030] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0031] To address the shortcomings of existing technologies, the present invention provides a skin condition assessment method based on multimodal data. This method comprises three core steps: data preprocessing, model building, and ideal skin index development. In the data preprocessing stage, raw multimodal data is converted into a standardized data matrix through feature extraction and classification. The model building stage employs a machine learning algorithm to construct a skin condition assessment model through supervised learning, effectively capturing the characteristic patterns in multimodal data. Finally, the ideal skin index development stage constructs a comprehensive scoring system by weighted fusion of multimodal data.

[0032] The method of the present invention integrates multimodal information such as images, microbial and host phenotypic data, and uses a machine learning model to comprehensively evaluate and score skin health status, achieving high accuracy and robustness under various skin types and capable of generating personalized scores in real time.

[0033] The skin condition assessment method and device based on multimodal data of the present invention will be further described below with reference to specific embodiments.

[0034] Example 1 Ginseng Figure 1 This embodiment provides a skin condition assessment method based on multimodal data, which includes steps such as data preprocessing, model construction and optimization, and ideal skin index design, aiming to achieve quantitative assessment of skin condition through multimodal data analysis.

[0035] The method specifically includes: S1. Acquire facial image data, bacterial flora data, and host phenotypic data for characterizing skin conditions, and preprocess the facial image data, bacterial flora data, and host phenotypic data, respectively, to obtain corresponding image features, bacterial flora features, and host phenotypic features.

[0036] First, facial image data of a volunteer or user is collected and preprocessed. In this embodiment, facial image data of the volunteer or user can be collected using a dedicated skin detection instrument. The facial image data is 3648*5472 pixels. The preprocessing operation on this facial image data primarily involves feature extraction using the deep learning model Resnet18. The Resnet18 model, through its residual network structure, effectively captures key features such as facial texture, color, and blemishes, converting the original image into a feature vector.

[0037] Specifically, the deep learning model Resnet18 is used to extract features from the facial image data, including: Image cropping: scaling the facial image to 256 pixels on the short side and cropping a 224*224 pixel image from the center; Image conversion, which converts the cropped facial image into a PyTorch tensor and scales the pixel values to [0.0, 1.0]; Image normalization, that is, normalizing the converted facial images to conform to the distribution of the ImageNet dataset; Feature extraction involves first loading the weights of a ResNet-18 model pre-trained on the ImageNet dataset, deleting its last fully connected layer, and setting the ResNet-18 model to evaluation mode to make it a feature extractor. Normalized facial image files are then named with sample IDs, and a text file is prepared to record the sample IDs and their order. Finally, the facial image and its text file are fed into the ResNet-18 model, processed through convolutional layers, pooling layers, and residual blocks, to extract a 224-dimensional feature vector.

[0038] An example of the extracted facial image feature vector is shown in Table 1 below: Table 1

[0039] In Table 1, the first column is the sample name, the first row is the feature name, and the specified position is the feature value of the sample.

[0040] Next, facial samples from volunteers or users are collected for DNA extraction and 16S rRNA amplicon sequencing to obtain microbiome data. This data is then preprocessed to obtain a microbiome feature matrix, specifically including: First, low-quality sequences were removed through quality control. That is, the original bacterial sequences were preprocessed by PMS (Parallel-MetaSuite) software, including denoising using the ASV (Amplicon Sequence Variants) algorithm and removing chimeras using the UCHIME algorithm to reduce sequencing errors and false-positive sequences.

[0041] The sequencing data was then aligned with the reference genome using a sequence alignment tool, and the relative abundance at the microbial genus level was calculated. That is, the sequence was aligned with the reference database GreenGenes using the Wearch tool for species annotation, and the alignment results were used to calculate the relative abundance of each taxonomic unit (such as genus).

[0042]

[0043] Finally, the relative abundance was adjusted by copy number correction to avoid over-representation of high copy number genes, and the relative abundance at the microbial genus level was obtained, and a microbiome feature matrix was generated, as shown in Table 2 below: Table 2

[0044] Table 2 shows an example of a microbiome feature matrix, where the first column is the sample name, the first row is the microbial genus name, and the designated position is the relative abundance of a specific genus in the sample.

[0045] The above preprocessing operations on the microbial community data were all completed in PMS (Parallel-Meta Suite).

[0046] Next, we collect the host phenotypic data of the volunteer or user, including age, BMI (Body Mass Index), skin type, etc., and perform the following preprocessing operations on the data: First, we perform grouping and discretization. For example, we divide age into three intervals: "≤20," "20-30," and ">30." We also divide skin types into four categories: "dry," "ordinary," "oil," and "sensitive." For the continuous numerical indicators obtained by the skin detector, we use the average value of the ideal skin group as the threshold and classify them into "high" and "low," as shown in Table 3. Table 3

[0047] In Table 3, the first column is the sample name, the first row is the feature name, and the designated position is the grouping information of the specific phenotype of the sample.

[0048] One-hot encoding method is then used to convert the categorical data into binary feature vectors.

[0049] Based on the above, the corresponding feature value data such as image features, bacterial community features and host phenotypic features were obtained respectively. These feature value data will be used for training and model testing in subsequent steps.

[0050] S2. Build a classification model based on the XGBoost algorithm, and input the image features, flora features, and host phenotypic features as training samples into the classification model for iterative training. Test the model effect through five-fold cross-validation to obtain a trained target classification model.

[0051] In this embodiment, given that the machine learning XGBoost algorithm can effectively process high-dimensional features through its gradient boosting framework and supports fusion learning of multimodal data, the XGBoost algorithm is preferably used as the classification model and the data features of the three modalities of image features, microbial features, and host phenotypic features are jointly learned. During the feature selection process, all features are systematically evaluated and ranked based on the feature importance scores provided by the XGBoost algorithm, such as gain, coverage, and other indicators, to screen out the top n features that contribute most to the model's predictive ability, thereby improving the efficiency and performance of the model.

[0052] Specifically, this embodiment uses Python language and XGBoost library to build a classification model. In this model, its relevant parameters are set as follows: the n_estimators parameter is set to 50, indicating that the model consists of 50 decision trees; the max_depth parameter is set to 5, which is used to limit the maximum depth of each decision tree to 5 layers to prevent model overfitting; the min_child_weight parameter is set to 4, which is used to control the minimum value of the sum of sample weights in leaf nodes, further adjusting the complexity of the model; the subsample parameter is set to 0.7, which means that 70% of the samples are randomly used during training of each decision tree to enhance the generalization ability of the model; the learning_rate parameter is set to 0.05, which is used to control the contribution of each decision tree to the final model. A smaller learning rate makes the model more stable; the gamma parameter is set to 4, which is used to specify the minimum loss function drop value required for node splitting, thereby controlling the complexity of the model.

[0053] Next, the classification model is iteratively optimized and trained using image features, bacterial flora features, and host phenotypic features as training samples. Specifically: The XGBoost algorithm was trained based on the bacterial community data. The importance index of each bacterial community feature was calculated by the XGBoost algorithm, and the forward feature selection method was used to screen out the optimal combination of 13 bacterial community features.

[0054] For host phenotypic data, the same method is used for feature selection, that is, the importance index of each host phenotypic feature is calculated by the XGBoost algorithm, and a set of 24 phenotypic features that are significantly related to the set target task is screened out.

[0055] Finally, the screened microbial community characteristics, host phenotypic characteristics and image features are subjected to multimodal data fusion to form a multi-source feature dataset, and the XGBoost algorithm is used to jointly model the multi-source feature data. Through iterative training of the optimization algorithm, the optimal feature combination of the three modal data of microbial community, phenotype and image is screened out as the target dataset.

[0056] This embodiment selects the optimal combination by eliminating irrelevant or redundant features, which not only reduces model learning noise, focuses on key factors, and improves generalization ability, but also simplifies model logic, reduces the computing overhead of the device, and accelerates training and prediction.

[0057] To verify model performance, this example uses the target dataset as input and employs a five-fold cross-validation method. The target dataset is divided into five mutually exclusive subsets, one of which is used as the test set, and the remaining subsets as the training set. This process is repeated five times to ensure the stability and reliability of the model evaluation results. Furthermore, hyperparameters such as the learning rate (learning_rate), maximum tree depth (max_depth), and subsampling rate are adjusted to further optimize model performance and enhance generalization. During the validation process, the receiver operating characteristic (ROC) curve is plotted to observe model performance, and the final target classification model is output.

[0058] like Figure 2 As shown, the x-axis is the false positive rate, which indicates the proportion of negative samples that are misclassified; the y-axis is the true positive rate, which indicates the proportion of positive samples that are correctly classified. Figure 2 The figure shows the ROC curves for each fold during the five-fold cross-validation process, as well as the average ROC curve for all folds (the blue curve in the figure). The area under the curve (AUC) is 0.87, indicating that the model has good classification performance. The light blue area in the figure represents one standard deviation of the average ROC curve, demonstrating the stability of the model performance. The curves in other colors in the figure represent the ROC curves for each fold, and their fluctuations are consistent with the average curve, further verifying the reliability of the model. The gray dashed line in the figure is the performance baseline, a key reference line, and the light blue area (±1 Std. Dev. area) represents the standard deviation range.

[0059] S3. Use the target classification model to predict the samples in the test set to obtain the positive probability that each sample belongs to the ideal skin category, construct an ideal skin index based on the positive probability, and evaluate the skin health status based on the ideal skin index.

[0060] That is, use the trained target classification model to predict the samples in the test set, and obtain the probability value of each sample belonging to the ideal skin category, that is, the positive class probability, analyze the distribution characteristics of the two groups of probability data, and select the middle value of 0.2 between the 75% quantile of the positive class probability distribution of the ideal skin group and the 25% quantile of the positive class probability distribution of the non-ideal skin group as the dividing threshold. Then perform a linear transformation. Specifically, the dividing threshold is mapped to the value of 0, and the positive class probability is compared with the dividing threshold. If it is lower than the dividing threshold, it is proportionally reduced; if it is higher than the dividing threshold, it is proportionally amplified, and the probability and distribution of each sample predicted by the target classification model as the positive class (that is, ideal skin state) are mapped to the range of -0.2~0.8 to enhance the discrimination and interpretability to construct the ideal skin index. In this way, an ideal skin index that can be intuitively interpreted is formed, and a box plot of the ideal skin index distribution of the ideal skin group (IS) and the non-ideal skin group (non-IS) is drawn. Figure 3 shown.

[0061] By comparing the Ideal Skin Index with the threshold, the index can be further converted into a classification result, such as an index ≥ 0.3 indicating ideal skin condition. The resulting Ideal Skin Index can intuitively reflect the degree of proximity between an individual's skin condition and the ideal skin standard, providing a quantitative basis for skin health assessment and further expanding its application scenarios.

[0062] Furthermore, in the actual operation of the method of this embodiment, before entering the data processing and modeling process, it is necessary to install and configure the corresponding Python packages, specifically torch 2.5.1, pandas 2.2.3, numpy 2.0.2, scikit-learn 1.5.2, seaborn 0.13.2, matplotlib 3.9.2, torchvision 0.20.1, xgboost 2.1.3, and openpyxl 3.1.5, to facilitate subsequent use needs.

[0063] The following supplements the rating description of the ideal skin group.

[0064] By observing the front-facing VISIA CR photos of the volunteers, pores, acne, spots, and wrinkles were used as evaluation indicators for scoring. The pores, acne, spots, and wrinkles of the volunteers were scored according to the following criteria. Finally, the total score of each volunteer was verified. The scoring standard for volunteers to be rated as ideal skin is: total score ≤ 2, and the score of each item ≤ 1. The specific indicator grading and scoring are as follows: The pore size classification is mainly based on the research published by Dr. Wang Qing, a dermatologist at West China Hospital of Sichuan University. The specific scoring criteria have been slightly modified based on Professor Wang Qing's research. Pore scoring standard: 0 - no visible pores; 1 - visible skin pores 0.1~0.3mm 2 ;2-aperture 0.3~0.6mm 2 ; No blackhead embedding; 3-pore diameter 0.3~0.6mm 2 , containing blackheads.

[0065] The acne scoring standard is customized and the specific standards are as follows: Acne scoring criteria: 0 - no obvious acne; 1 - 1-2 acnes; 2 - less than 10 acnes; 3 - more than 10 acnes with papules and nodules.

[0066] The spot scoring criteria are mainly based on the area and severity index of melasma. The self-scoring criteria are slightly modified. The specific criteria are as follows: Spot scoring criteria: 0 - no visible spots; 1 - spots occupying less than 10% of the total facial area; 2 - spots occupying 10% to 29% of the total facial area; 3 - spots occupying more than 30% of the total facial area.

[0067] The wrinkle scoring standard is mainly based on the Facial Wrinkle Scale (FWS). The specific scoring standards are as follows: Wrinkle scoring criteria: 0 - no visible wrinkles; 1 - fine wrinkles and slight indentations similar to smile lines; 2 - wrinkles are clearly visible, with a depth of 1-2 mm; 3 - wrinkles are obvious in depth and number, with a depth of ≥3 mm.

[0068] When evaluating volunteers with ideal skin aged 25 to 35, the wrinkle scoring criteria were slightly relaxed as follows: Wrinkle relaxation standards: 0 - no visible wrinkles, or restorable dynamic expression lines; 1 - irreversible static expression lines, wrinkle depth 1-2mm; 2 - obvious wrinkle depth and number, wrinkle depth ≥3mm; 3 - long and deep wrinkles, wrinkles are V-shaped after stretching, wrinkle depth ≥3mm.

[0069] In summary, the skin condition assessment method based on multimodal data of the present invention, in the data preprocessing stage, can comprehensively characterize the skin condition by extracting facial image features through Resnet18, overcoming the problem of insufficient feature expression ability of traditional image processing methods; one-hot encoding is used for phenotypic data, which effectively eliminates the numerical bias of categorical data and improves the interpretability of the data; strict quality control and standardization of microbiome data are performed to ensure the reliability and comparability of microbiome data.

[0070] During the model construction and optimization stage, the XGBoost algorithm model can fully explore the correlation between image, phenotypic and microbiome data through multimodal data fusion learning, significantly improving the model's predictive performance; the feature selection method based on feature importance scoring can effectively reduce model complexity and avoid overfitting problems; the introduction of the five-fold cross-validation method ensures the stability and reliability of the model evaluation results, providing a scientific basis for model optimization.

[0071] During the ideal skin index design phase, the ideal skin index, constructed based on model prediction probability, is highly intuitive and interpretable, and can quantitatively assess the degree of closeness between an individual's skin condition and the ideal condition. The index is mapped to the range of -0.2 to 0.8 to facilitate user understanding and comparison. The setting of the demarcation threshold further expands the application range of the index, making it usable not only for condition assessment but also for classification decision-making, providing a scientific basis for the formulation of personalized skin care plans.

[0072] Example 2 This embodiment provides a device for implementing a skin condition assessment method based on multimodal data, the device comprising: a data acquisition module, configured to acquire facial image data, bacterial flora data, and host phenotypic data for characterizing skin conditions, and preprocess the facial image data, bacterial flora data, and host phenotypic data, respectively, to obtain corresponding image features, bacterial flora features, and host phenotypic features; A model building module is used to build a classification model based on the XGBoost algorithm, and input the image features, bacterial flora features and host phenotypic features as training samples into the classification model for iterative training, and test the model effect through five-fold cross-validation to obtain a trained target classification model; The skin condition assessment module is used to use the target classification model to predict the samples in the test set, obtain the positive class probability that each sample belongs to the ideal skin category, construct an ideal skin index based on the positive class probability, and assess the skin health status based on the ideal skin index.

[0073] Example 3 This embodiment further provides an electronic device, including: At least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to perform the skin condition assessment method based on multimodal data as described above.

[0074] In this embodiment, electronic devices may include, but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, and the like.

[0075] Example 4 This embodiment further provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the skin condition assessment method based on multimodal data as described above.

[0076] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0077] In this case, the program code itself read from the machine-readable medium can implement the functions of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0078] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0079] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0083] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A skin condition assessment method based on multimodal data, characterized in that: The method comprises: S1. Acquire facial image data, bacterial flora data, and host phenotypic data for characterizing skin conditions, and preprocess the facial image data, bacterial flora data, and host phenotypic data to obtain corresponding image features, bacterial flora features, and host phenotypic features; S2. Build a classification model based on the XGBoost algorithm, and input the image features, bacterial flora features, and host phenotypic features as training samples into the classification model for iterative training. Test the model effect through five-fold cross-validation to obtain a trained target classification model. S3. Use the target classification model to predict the samples in the test set to obtain the positive probability that each sample belongs to the ideal skin category, construct an ideal skin index based on the positive probability, and evaluate the skin health status based on the ideal skin index.

2. The skin condition assessment method based on multimodal data according to claim 1, characterized in that: In S1, the facial image data is 3648*5472 pixels, and preprocessing the facial image data specifically includes: The facial image data is cropped, scaled to a shorter side of 256 pixels, and a 224*224 pixel area is cropped from the center; Convert the cropped facial image data into a PyTorch tensor and scale the pixel values to [0.0, 1.0]; Normalizing the converted facial image data to conform to the distribution of the ImageNet dataset; Load the weights of the ResNet-18 model pre-trained on the ImageNet dataset, delete its last fully connected layer, and set the ResNet-18 model to evaluation mode to make it a feature extractor; The normalized facial image data is named with a sample ID, and a txt file is set up to record the sample ID and its order; The facial image data and its txt file are input into the ResNet-18 model, processed by the convolution layer, pooling layer and residual block, and finally a 224-dimensional feature vector is extracted.

3. The skin condition assessment method based on multimodal data according to claim 1, characterized in that: In S1, the microbial community data is preprocessed, specifically including: The original sequences of the bacterial community data were denoised and de-chimerized using PMS software to remove low-quality sequences; The retained sequences were aligned with the reference database GreenGenes using the Wearch tool for species annotation, and the alignment results were used to calculate the relative abundance of each taxon; The relative abundance was adjusted by copy number correction to obtain the relative abundance at the microbial genus level and generate a microbiome feature matrix.

4. The skin condition assessment method based on multimodal data according to claim 1, characterized in that: In S1, the host phenotypic data is preprocessed, specifically including: performing grouping and discretization processing on the host phenotypic data to obtain grouped data; The grouped data is converted into a binary feature vector using a one-hot encoding method.

5. The skin condition assessment method based on multimodal data according to claim 1, characterized in that: In S2, the image features, bacterial flora features, and host phenotypic features are input as training samples into the classification model for iterative training, specifically including: The importance index of each flora feature of the flora data is calculated based on the XGBoost algorithm, and the optimal 13 flora feature combinations are screened out using the forward feature selection method; Calculate the importance index of each host phenotypic feature of the host phenotypic data based on the XGBoost algorithm, and screen out 24 phenotypic feature sets related to the set target task; The screened bacterial flora feature combination, phenotypic feature set and image features are subjected to multimodal data fusion to form a multi-source feature data set, and the multi-source feature data is used to train the XGBoost algorithm, which outputs the optimal feature combination as the target data set.

6. The skin condition assessment method based on multimodal data according to claim 5, characterized in that: In S2, the model effect is tested by five-fold cross validation, specifically including: using a five-fold cross validation method, dividing the target data set into five mutually exclusive subsets, taking one of the subsets as a test set and the rest as training sets, and repeating five times to verify the model effect of the XGBoost algorithm, while adjusting the parameters of the XGBoost algorithm during the iterative process, and finally using the trained XGBoost algorithm as the target classification model.

7. The skin condition assessment method based on multimodal data according to claim 1, characterized in that: In S3, constructing an ideal skin index based on the positive class probability specifically includes: Based on the positive class probability of each sample belonging to the ideal skin category output by the target classification model, a probability distribution of the ideal skin group and a probability distribution of the non-ideal skin group are obtained; The midpoint between the 75% quantile of the probability distribution of the ideal skin group and the 25% quantile of the probability distribution of the non-ideal skin group was selected as the demarcation threshold; The positive class probability is linearly transformed, that is, the positive class probability is compared with a cutoff threshold. If it is lower than the cutoff threshold, it is proportionally reduced; if it is higher than the cutoff threshold, it is proportionally amplified to enhance the discrimination and interpretability to form an ideal skin index.

8. A device for implementing a skin condition assessment method based on multimodal data, characterized in that: The device comprises: a data acquisition module, configured to acquire facial image data, bacterial flora data, and host phenotypic data for characterizing skin conditions, and preprocess the facial image data, bacterial flora data, and host phenotypic data, respectively, to obtain corresponding image features, bacterial flora features, and host phenotypic features; A model building module is used to build a classification model based on the XGBoost algorithm, and input the image features, bacterial flora features and host phenotypic features as training samples into the classification model for iterative training, and test the model effect through five-fold cross validation to obtain a trained target classification model; The skin condition assessment module is used to use the target classification model to predict the samples in the test set, obtain the positive class probability that each sample belongs to the ideal skin category, construct an ideal skin index based on the positive class probability, and assess the skin health status based on the ideal skin index.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to perform the skin condition assessment method based on multimodal data according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores executable instructions, which, when executed, enable the machine to perform the skin condition assessment method based on multimodal data according to any one of claims 1 to 7.

Images