Anti-aging cosmetic comprehensive effect characterization method and system based on convolutional neural network

Abstract

The invention discloses an anti-aging cosmetic comprehensive effect characterization method and system based on a convolutional neural network. The characterization method comprises the steps that various data of the same user in the cosmetic use period are collected, and the various data at least comprise skin microscopic images, 3D face topological data and physiological index numerical values; aligning the various data according to timestamps to form a time sequence multi-modal data sequence; the time sequence multi-modal data sequence is processed by using a space-time dual-channel convolutional neural network CNN, a space channel extracts image space features based on ResNet-50, a time sequence channel extracts time sequence change features based on LSTM, and outputs of the two channels are fused to obtain a dynamic improvement score; and outputting the dynamic improvement score and the visual heat map for representing the anti-aging effect of the cosmetics.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and cosmetic efficacy evaluation, specifically to a method and system for quantitatively characterizing the comprehensive effects of anti-aging cosmetics based on convolutional neural networks (CNN). Background Technology

[0002] 1. Existing technological pain points: Subjectivity limitations: The evaluation of the effects of traditional anti-aging cosmetics often relies on expert visual ratings or consumer questionnaires, which are subject to human bias and make it difficult to quantify subtle changes in the skin.

[0003] Limited to single-dimensional detection: Existing instruments (such as skin elastometers and VISIA imaging systems) can only measure single parameters (such as moisture, elasticity, wrinkles, etc.), lacking a global characterization of skin aging.

[0004] Lack of dynamic tracking: Most methods cannot achieve dynamic analysis of the effects during long-term use, making it difficult to link the causal relationship between cosmetic ingredients and efficacy.

[0005] 2. Relevant Technological Advances: Convolutional neural networks (CNNs) have been successfully applied in image recognition (such as medical image analysis), but have not yet been used for multimodal data fusion evaluation of cosmetic efficacy.

[0006] Current research has proposed skin detection based on image analysis, but it has not solved the problems of temporal quantification of anti-aging effects and multi-feature collaborative characterization. Summary of the Invention

[0007] 1. A method for characterizing the comprehensive effects of anti-aging cosmetics based on convolutional neural networks, characterized by comprising: (1) Collect multiple data from the same user during the use of cosmetics, including at least skin microscopic images, 3D facial topology data and physiological index values; (2) Align the various data according to timestamps to form a time-series multimodal data sequence; (3) The temporal multimodal data sequence is processed using a spatiotemporal dual-path convolutional neural network (CNN), wherein the spatial path extracts image spatial features based on ResNet-50, and the temporal path extracts temporal change features based on LSTM. The outputs of the two paths are then fused to obtain a dynamic improvement score. (4) Output the dynamic improvement score and visualization heatmap to characterize the anti-aging effect of cosmetics.

[0008] Furthermore, before CNN processing, skin microscopic images, 3D facial topology data, and spectral information at the same time are stitched together to construct a multi-dimensional input matrix for global characterization of skin aging.

[0009] Furthermore, the spatiotemporal dual-path convolutional neural network (CNN) uses a ResNet-50 backbone network in the spatial path, removes the global average pooling layer and the fully connected classification layer, and retains only the convolutional layer, pooling layer and residual block to output the spatial feature vector.

[0010] Furthermore, the spatiotemporal dual-path CNN architecture includes: Spatial pathways are used to extract local aging features; The time-series pathway is used to extract global time-series change trends to obtain the improvement rate after 28 days of use.

[0011] Furthermore, the temporal path in the spatiotemporal dual-path uses a temporal analysis module, which is based on an LSTM layer, which has one or more layers; optionally, an attention layer is connected to the output of the LSTM layer; then the features are mapped to the final output dimension through a fully connected layer.

[0012] Furthermore, before the dynamic score output, the dataset is divided into a training set, a validation set, and a test set according to the user ID. All data of the same user exists in only one set, and quantitative indicators and Cohen-Kappa coefficient or Pearson correlation coefficient with dermatologist scores are obtained on the test set.

[0013] Furthermore, the physiological indicators include pH, TEWL, oil, and humidity. The 3D facial topology data is obtained through structured light or laser scanning and is used to extract wrinkle depth, density, and length parameters.

[0014] Furthermore, the visualized heatmap is generated via Grad-CAM and used to locate key skin areas where cosmetics are effective.

[0015] Furthermore, the formula for quantifying the effect: ; in, For comprehensive scoring, For the first Class feature output, The rate of change over time. and These are the weights for static features and dynamic rate of change, respectively. For the first Weights of class features , This represents the total number of feature categories.

[0016] A method and system for characterizing the comprehensive effects of anti-aging cosmetics based on convolutional neural networks, comprising the following steps: The data acquisition layer is used to acquire skin microscopic images, 3D facial topology data, and physiological index values. The data processing and storage layer is used to clean, standardize, calibrate, and store the raw data; The algorithm layer includes a feature extraction module and a temporal analysis module, and is used to stitch and align multimodal data. The feature extraction module takes the processed 2D skin image as input and outputs a spatial feature vector. The temporal analysis module takes the sequence formed by stitching the spatial feature vectors of multiple consecutive time points with the corresponding numerical data as input and outputs a dynamic score. The application layer is used to generate personalized anti-aging reports and visual heatmaps.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This method uses deep learning technology to achieve multi-dimensional dynamic monitoring of skin aging characteristics (such as wrinkles, sagging, and age spots) and objective quantitative evaluation of the efficacy of cosmetics.

[0018] Multimodal data fusion: Combining skin microscopic images, 3D topological data, and spectral information (such as infrared and polarized light), a multi-dimensional input matrix of aging characteristics is constructed to globally characterize the skin aging state.

[0019] Dynamic effect representation model: Design a spatiotemporal dual-path CNN architecture to extract local aging features (such as wrinkle depth) and global temporal change trends (such as improvement rate after 28 days of use).

[0020] Interpretable output: Generates visual heatmaps using gradient-weighted class activation mapping (Grad-CAM) to pinpoint key skin areas where cosmetics work. Attached Figure Description

[0021] Figure 1 To visualize the heatmap, the color depth in the image represents the degree of improvement of spots or wrinkles; the darker the color, the higher the improvement rate. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0023] 1. System Architecture 1.1 Data Acquisition Layer: High-resolution skin imaging (UV / white light mode): Captures subcutaneous pigmentation and fluorescent spots (such as bacterial metabolites) in UV light mode; analyzes skin color, texture, pores, and surface spots in white light mode.

[0024] 3D facial scanner: Using structured light or laser scanning, a high-precision 3D facial model is constructed, accurately quantifying geometric parameters such as the density, depth, and length of wrinkles.

[0025] Biosensors: provide objective physiological data, such as pH value (characterizing skin barrier health), transepidermal water loss (TEWL) (characterizing skin barrier integrity), humidity, and oil content.

[0026] 1.2 Data Processing and Storage Layer: Function: Receives raw data from various instruments, performs preliminary cleaning, format standardization (such as image size unification, sensor data unit unification) and calibration, providing high-quality and consistent input for the algorithm layer.

[0027] Storage is divided into two parts: Structured databases (such as SQL): store user information, numerical results of each measurement (pH value, TEWL value, wrinkle index, etc.) and timestamps.

[0028] Unstructured storage (such as object storage): Stores raw, high-resolution images, 3D model files, and other large data objects.

[0029] 1.3 Algorithm Layer: Feature extraction module (ResNet-50): The core idea of ​​ResNet-50 is to introduce "residual blocks" and "shortcut connections" to solve the gradient vanishing and degradation problems in deep neural networks, allowing the network to be built very deep and still effective.

[0030] For a feature extraction module, typically only its backbone network is used, that is, the final global average pooling layer and fully connected classification layer are removed, and only the convolutional layers and pooling layers are retained. The final output feature map can be used as input for subsequent tasks (such as object detection, image segmentation, image retrieval, etc.).

[0031] The specific process is as follows: Input: Processed 2D skin image.

[0032] process: 1. Initial Convolution and Pooling (Stem): This part performs preliminary downsampling and feature extraction on the input image.

[0033] 2. Convolutional Layers (Stages 1-4): This is the core of ResNet-50, consisting of 4 stages, each composed of multiple stacked residual blocks. At the end of each stage, the spatial size of the feature map is halved, while the number of channels is doubled.

[0034] Output: High-dimensional feature vectors used to quantitatively describe skin condition.

[0035] Timing Analysis Module (LSTM): enter: Numerical data: such as [TEWL, wrinkle index, moisture content, pH value, pigmentation index].

[0036] Image feature vector: A vector obtained by flattening or global average pooling the feature map extracted from ResNet-50.

[0037] The input vector for a single time step is the concatenation of these two parts.

[0038] process: LSTM uses its unique gating mechanism (forget gate, input gate, output gate) to determine which information to remember or forget, making it ideal for capturing long-term dependencies. It can capture dynamic changes in skin condition over time, assess the effectiveness of skincare programs, and even predict future skin conditions.

[0039] A complete time series analysis module typically includes the following layers: LSTM layer: The core layer, which can have one or more layers.

[0040] Optional attention mechanism: To better capture the importance of different time steps in the sequence (e.g., the most recent measurement may be more important than a measurement from a year ago), an attention layer can be added to the LSTM output.

[0041] Fully connected layer: Maps the output of the LSTM to the desired final output dimension.

[0042] Output: Trend analysis results: A score or category label indicating the overall degree of improvement (e.g., [significant improvement, slight improvement, stable, deterioration]).

[0043] Rate of improvement: A specific regression value that represents the degree of improvement per unit of time (e.g., 0.5 means that the wrinkle index decreases by 0.5 per week).

[0044] Future predictions: Predict the numerical indicators to be measured in the next one or several times (e.g., [predict_TEWL, predict_wrinkle index]).

[0045] The core of this module is to process a sequence of data, where each time step contains the user's numerical indicators and image features at that measurement, thereby learning dynamic patterns of skin condition changes.

[0046] Multimodal data fusion: By fusing and aligning features from different instruments (images, numerical data, 3D geometry), a comprehensive, multi-dimensional user skin condition profile is constructed, providing a foundation for generating accurate reports.

[0047] 1.4 Application Layer: Personalized anti-aging report generation: Based on all the analysis results, an easy-to-understand personalized report is generated. The report includes: current skin problem diagnosis, comparison with historical data, and improvement rates for various indicators (which can be ranked to allow users to clearly see which aspects have shown the most significant progress).

[0048] Ingredient efficacy correlation analysis: The system correlates the ingredients in the skincare products a user uses (either input by the user or matched against a product database) with improvements in skin metrics. Algorithms (such as association rules or statistical analysis) are used to predict which ingredients or combinations of ingredients are most effective for the user's specific problems.

[0049] Visual dashboard: It provides users and physicians with a graphical interface that displays historical data trend charts, 3D model comparisons, improvement rate radar charts, etc., making the results more intuitive.

[0050] 2. Key Steps 2.1. Data Preprocessing: ① Image data standardization (high-resolution skin imager) Alignment and cropping: Facial images taken at different times are aligned using facial landmark detection (such as Dlib or MTCNN), and fixed regions (such as forehead, corners of the eyes, and cheeks) are cropped to ensure the consistency of the analysis area.

[0051] Uniform resolution: Scale all images to a uniform size (e.g., 224x224, to fit ResNet input).

[0052] Color normalization: Using standardized color correction cards or algorithms to reduce color deviations caused by different lighting and equipment parameters.

[0053] ② 3D Data Meshing and Feature Extraction (3D Facial Scanner) Mesh preprocessing: Denoising, smoothing, and hole filling are performed on the original 3D point cloud or mesh data.

[0054] Feature point annotation: Automatically annotate key anatomical feature points (such as the corners of the eyes and mouth) on the 3D model that correspond to the 2D image.

[0055] Wrinkle parameter extraction: In a specific region (such as the crow's feet area), the 3D model is converted into a quantified wrinkle feature vector (such as average depth, total volume, density) through algorithms such as curvature analysis and contour extraction.

[0056] ③ Numerical data calibration and normalization (biosensors) Equipment calibration: Calibrate the biosensor readings according to the equipment manual or standard reagents to ensure data accuracy. Normalization: Values ​​of different dimensions and ranges, such as pH and TEWL, are normalized to the same scale through Z-score standardization or Min-Max scaling, which facilitates model processing.

[0057] ④ Data alignment Time alignment: Linking multimodal data (images, 3D, numerical data) collected by the same user at the same time point to form a multimodal data snapshot.

[0058] Constructing a time series: Arrange multiple time-point "snapshots" of each user in chronological order to form their time series data sequence.

[0059] 2.2 Model Training: ① Model selection criteria Deep feature extraction capability: ResNet-50 solves the gradient vanishing problem in deep networks through residual connections, making it suitable for capturing multi-level features of skin aging (such as shallow texture and deep structural deformation).

[0060] Advantages of transfer learning: Weights pre-trained on ImageNet can be generalized to skin image analysis, and can be adapted to anti-aging tasks through fine-tuning, reducing data requirements.

[0061] ② Feature fusion strategy: Introducing an attention mechanism Motivation: Simple feature splicing or addition cannot distinguish the importance of different feature channels and spatial locations. Areas with dense wrinkles and pigmentation should contribute more than areas of smooth skin.

[0062] Implementation: After concatenating the two-branch features, a convolutional attention module (CBAM) is introduced.

[0063] Channel attention: Learns the weights of different feature channels, enhances the response of aging-related features (such as wrinkle channels and pigmentation channels), and suppresses irrelevant features.

[0064] Spatial attention: Learns the weights of spatial locations in feature maps, focusing on areas with severe aging problems (such as the corners of the eyes and forehead), to achieve precise attention to key areas.

[0065] ③ Training details Data adaptation: Input size adjustment: Change the original 224×224 resolution to 400×400 to adapt to the output of high-precision skin imaging equipment.

[0066] Normalization parameters: Adjust the Batch Normalization layer based on the skin image dataset (e.g., RGB mean [0.685, 0.523, 0.570]).

[0067] Loss function optimization Composite Loss: Jointly optimizes multiple objectives.

[0068] Focal Loss: Objective: To address the problem of extreme imbalance between positive and negative samples in skin images (e.g., pigmented spots have far fewer pixels than normal skin).

[0069] Function: By adjusting the focusing parameter γ, the model can pay more attention to samples that are difficult to classify (such as color spots with blurred edges), effectively improving the detection rate of subtle aging features.

[0070] SSIM Loss: Objective: In regression tasks (such as image quality reconstruction and feature map generation), constrain the structural similarity between the model output and the ground truth to avoid image blurring caused by relying solely on pixel-level errors.

[0071] Function: To make the features or reconstructed images generated by the model closer to the real skin in terms of texture, contrast, and structure, while preserving structural details.

[0072] 2.3 Effect Verification A strategy combining multiple evaluation methods is employed to comprehensively verify the effectiveness and reliability of the system.

[0073] ① Reserved validation set test Divide the dataset into training, validation, and test sets (e.g., 70% / 15% / 15%) based on user ID to ensure that all data for the same user exists in only one set, preventing data leakage and more realistically reflecting the model's generalization ability.

[0074] Report the final performance on the test set.

[0075] ② Quantitative indicator evaluation For classification tasks (e.g., whether to improve): Accuracy, Precision, Recall, and F1 score.

[0076] For regression tasks (such as predicting index values): mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R-Squared).

[0077] For ranking tasks: NDCG (Normalized Depreciation Cumulative Gain) is used to evaluate the quality of the "Improvement Rate Ranking" list.

[0078] ③ Manual evaluation comparison The model's evaluation results (such as generated reports and improvement rate rankings) were compared with the results of blind reviews by multiple dermatologists.

[0079] Calculate the Coen-Kappa coefficient or Pearson correlation coefficient to quantify the degree of agreement between the model and expert opinions. This is a crucial step in demonstrating its clinical value.

[0080] Table 1. Multidimensional input matrix characterizing aging features Sample ID Image Feature 1 ... Image features N_image 3D Feature 1 ... 3D features N_3d Spectral characteristics 1 ... Spectral characteristics N_spec 1 0.85 ... 12.5 15.2 ... 0.03 1250.5 ... 0.75 2 0.72 ... 9.8 18.7 ... 0.05 1102.3 ... 0.68 ... ... ... ... ... ... ... ...

[0081] Table 1 shows the multidimensional input matrix characterizing aging features, where: Rows: Each sample (i.e., skin measurement data of one subject). Columns: The concatenation of all features [V_image, V_3d, V_spec]. V_image is the dimension of the image features; V_3d is the dimension of the 3D features; V_spec is the dimension of the spectral features. Dimensions: (N_samples, N_total), where N_samples is the number of rows in the matrix, and N_total is the number of columns in the matrix, where N_total = N_image + N_3d + N_spec.

[0082] Table 2 Improvement rate of multidimensional features over time Timestamp Image Feature 1 ... Image features N_image 3D Feature 1 ... 3D features N_3d Spectral characteristics 1 ... Spectral characteristics N_spec w1 1.30% ... 2.5% 2.32% ... 1.70% 2.34% ... 1.86% w2 6.43% ... 9.8% 4.73% ... 5.83% 4.85% ... 7.13% w3 8.37% ... 10.31% 7.47% ... 9.35% 10.17% 9.87% w4 12.65% ... 15.67% 11.63% ... 14.67% 15.35% ... 13.15%

[0083] w1 refers to the initial improvement rate of various skin indicators after one week of using cosmetics; w2, w3, and w4 correspond to the dynamic changes in weeks 2, 3, and 4, respectively. By longitudinally comparing the data fluctuations at different time points, the correlation between feature dimensions and the differences in response time sequence can be clearly identified, thereby assessing the comprehensive impact of the product on skin condition at different times.

[0084] To verify its clinical applicability, taking an anti-wrinkle product as an example, three senior dermatologists were invited to conduct a blind review of before-and-after comparison data from 50 randomly sampled test set users. The model output was then compared with the aggregated expert scores. The Pearson correlation coefficient between the model-predicted improvement score and the mean expert score was as high as 0.87 (p < 0.001).

[0085] On the binary classification task of "whether there is a significant improvement", the Cohen-Kappa coefficient between the model and the expert consensus is 0.82, which shows that the model and the expert opinion have a high degree of consistency.

[0086] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for characterizing the comprehensive effects of anti-aging cosmetics based on convolutional neural networks, characterized in that, include: (1) Collect multiple data from the same user during the use of cosmetics, including at least skin microscopic images, 3D facial topology data and physiological index values; (2) Align the various data according to timestamps to form a time-series multimodal data sequence; (3) The temporal multimodal data sequence is processed using a spatiotemporal dual-path convolutional neural network (CNN), wherein the spatial path extracts image spatial features based on ResNet-50, and the temporal path extracts temporal change features based on LSTM. The outputs of the two paths are then fused to obtain a dynamic improvement score. (4) Output the dynamic improvement score and visualization heatmap to characterize the anti-aging effect of cosmetics.

2. The method according to claim 1, characterized in that, Before CNN processing, skin microscopic images, 3D facial topology data, and spectral information at the same time are stitched together to construct a multi-dimensional input matrix for global characterization of skin aging.

3. The method according to claim 1, characterized in that, The spatiotemporal dual-path convolutional neural network (CNN) uses a ResNet-50 backbone network in the spatial path, removes the global average pooling layer and fully connected classification layer, and retains only the convolutional layer, pooling layer and residual block to output spatial feature vectors.

4. The method according to claim 3, characterized in that, The spatiotemporal dual-path CNN architecture includes: Spatial pathways are used to extract local aging features; The time-series pathway is used to extract global time-series change trends to obtain the improvement rate after 28 days of use.

5. The method according to claim 3, characterized in that, The temporal path in the spatiotemporal dual-path system employs a temporal analysis module, which is based on an LSTM layer. The LSTM layer has one or more layers. Optionally, an attention layer is connected to the output of the LSTM layer. Subsequently, the features are mapped to the final output dimension through a fully connected layer.

6. The method according to claim 1, characterized in that, Before the dynamic score output, the dataset is divided into a training set, a validation set, and a test set according to the user ID. All data of the same user exists in only one set. Quantitative indicators and the Cohen-Kappa coefficient or Pearson correlation coefficient with the dermatologist's score are obtained on the test set.

7. The method according to claim 1, characterized in that, The physiological indicators include pH, TEWL, oil, and humidity. The 3D facial topology data is obtained through structured light or laser scanning and is used to extract wrinkle depth, density, and length parameters.

8. The method according to claim 1, characterized in that, The aforementioned visualization heatmap is generated using Grad-CAM and is used to locate key skin areas where cosmetics work.

9. The method according to claim 1, characterized in that, Formula for quantifying effects: ； in, For comprehensive scoring, For the first Class feature output, The rate of change over time. and These are the weights for static features and dynamic rate of change, respectively. For the first Weights of class features , This represents the total number of feature categories.

10. A method and system for characterizing the comprehensive effects of anti-aging cosmetics based on convolutional neural networks, used to perform the method described in any one of claims 1-9, characterized in that, include: The data acquisition layer is used to acquire skin microscopic images, 3D facial topology data, and physiological index values. The data processing and storage layer is used to clean, standardize, calibrate, and store the raw data; The algorithm layer includes a feature extraction module and a temporal analysis module, and is used to stitch and align multimodal data. The feature extraction module takes the processed 2D skin image as input and outputs a spatial feature vector. The temporal analysis module takes the sequence formed by stitching the spatial feature vectors of multiple consecutive time points with the corresponding numerical data as input and outputs a dynamic score. The application layer is used to generate personalized anti-aging reports and visual heatmaps.

Images