A multi-modal data processing method, apparatus, electronic device, medium and product

Abstract

The application provides a multi-modal data processing method, a multi-modal data processing device, an electronic device, a computer readable storage medium and a computer program product. The method comprises: obtaining a skin physiological parameter of a target user, an image feature of a face image of the target user, a first evaluation index of the target user on the face image, a second evaluation index of an expert on the face image, and a physiological feature of a reference user when watching the face image; generating a multi-modal feature vector based on the skin physiological parameter, the image feature, the first evaluation index, the second evaluation index, and the physiological feature; inputting the multi-modal feature vector into a prediction model to obtain a complexion index score and a complexion classification result of the target user output by the prediction model.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a multimodal data processing method, a multimodal data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Methods for assessing complexion typically rely on a single data source, such as skin detection parameters or facial image analysis. Some solutions attempt to combine facial images with user self-assessment information for a comprehensive judgment. However, because these technologies fail to effectively integrate multi-dimensional data for quantitative complexion assessment, they suffer from strong subjectivity and difficulty in achieving objective quantification. Summary of the Invention

[0003] This application provides a multimodal data processing method, a multimodal data processing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a multimodal data processing method, the method comprising: Acquire the target user's skin physiological parameters, the image features of the target user's facial image, the target user's first evaluation index of the facial image, the expert's second evaluation index of the facial image, and the physiological characteristics of the reference user when viewing the facial image; Based on the skin physiological parameters, the image features, the first evaluation index, the second evaluation index, and the physiological features, a multimodal feature vector is generated. The multimodal feature vector is input into the prediction model to obtain the target user's complexion index score and complexion grading result output by the prediction model.

[0005] This application provides a multimodal data processing apparatus, including: The acquisition unit is used to acquire the skin physiological parameters of the target user, the image features of the target user's facial image, the first evaluation index of the target user on the facial image, the second evaluation index of the expert on the facial image, and the physiological characteristics of the reference user when viewing the facial image. The processing unit is configured to generate a multimodal feature vector based on the skin physiological parameters, the image features, the first evaluation index, the second evaluation index, and the physiological features. The processing unit is used to input the multimodal feature vector into the prediction model to obtain the target user's complexion index score and complexion grading result output by the prediction model.

[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 6.

[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the multimodal data processing method provided in this application when executed by a processor.

[0008] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the multimodal data processing method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: By comprehensively collecting the target user's skin physiological parameters, facial image features, subjective scores, expert scores, and EEG neural features, and uniformly converting them into multimodal feature vectors before inputting them into the prediction model, it can effectively integrate multiple heterogeneous data sources, improving the scientific rigor and objectivity of complexion assessment. Compared to related technologies that rely on a single parameter or subjective judgment, this application introduces multidimensional information combining subjective and objective factors, enhancing the stability and credibility of the assessment results. Attached Figure Description

[0010] Figure 1 This is a first flowchart illustrating the multimodal data processing method provided in this application embodiment; Figure 2 This is a schematic diagram of the second process of the multimodal data processing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the third process of the multimodal data processing method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the fourth process of the multimodal data processing method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the fifth process of the multimodal data processing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the multimodal data processing device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0011] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0016] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0017] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0018] Figure 1 This is a schematic diagram of the first process of the multimodal data processing method provided in the embodiments of this application. The following will be combined with... Figure 1 The steps shown will be explained. It should be noted that... Figure 1 The method described uses an electronic device as the execution subject as an example. Figure 1As shown, the method includes the following steps 101 to 103. Step 101: Obtain the target user's skin physiological parameters, the target user's facial image features, the target user's first evaluation index of the facial image, the expert's second evaluation index of the facial image, and the reference user's physiological characteristics when viewing the facial image.

[0019] Among them, the target user's skin physiological parameters refer to various quantitative indicators reflecting skin condition collected by a skin analyzer, such as stratum corneum moisture content, transepidermal water loss rate, skin radiance, erythema index (EI value), elasticity (R2 value), firmness (F4 value), and color (ITA value, L). value, a value, b The parameters include skin tone evenness, wrinkle area, and crow's feet volume. These parameters objectively describe the health and appearance of the skin, providing basic data for subsequent complexion assessment.

[0020] In this context, facial image features refer to the visual features extracted from frontal facial images under standard lighting conditions, including but not limited to skin color distribution, texture information, and contour structure. Facial image features reflect the overall appearance and details of an individual's face and are an important component in constructing multimodal feature vectors.

[0021] The primary evaluation indicator refers to the subjective rating given by the target user based on facial images. This can be achieved using a nine-point Likert scale or other rating methods, and can be based on multiple dimensions such as skin tone evenness, radiance, rosy complexion, and mental state. The primary evaluation indicator reflects the target user's own perception and satisfaction, and serves as an effective supplement to objective data.

[0022] The second evaluation metric consists of expert ratings of the same facial image. While typically based on the same scoring dimensions, the second metric is determined by assessors with specialized knowledge. Expert ratings possess greater authority and consistency, more accurately reflecting the true complexion and health of the face in the image.

[0023] The physiological characteristics of the reference user when viewing facial images include electroencephalogram (EEG) signals. An EEG acquisition device records the brain activity of the reference user while observing the same facial image and making a subjective evaluation. After preprocessing, neurophysiological features related to facial color perception are extracted from the EEG signals, such as energy changes in specific frequency bands and event-related potentials (ERPs). These extracted neurophysiological features further enrich the input data of the evaluation model.

[0024] In practice, by integrating various measurement tools and scoring methods, not only can comprehensive skin physiological and visual characteristics be obtained, but subjective and objective evaluations and neurophysiological feedback can also be introduced, thereby constructing a more scientific and systematic multimodal dataset. Multi-source data fusion helps improve the accuracy of complexion assessment and provides a reliable basis for personalized beauty guidance.

[0025] Step 102: Generate a multimodal feature vector based on skin physiological parameters, image features, a first evaluation index, a second evaluation index, and physiological features.

[0026] In this embodiment, the multimodal feature vector refers to the unified feature representation formed after standardizing, reducing the dimensionality, and fusing the above-mentioned heterogeneous data.

[0027] Electronic devices stitch these processed data together to form a multimodal feature vector. This multimodal feature vector integrates information from various aspects, including skin physiology, visual appearance, subjective and objective evaluations, and neurophysiology, and can comprehensively depict the target user's complexion and condition.

[0028] In practice, standardizing, reducing dimensionality, and fusing features on the original data can effectively reduce noise interference and improve the model's generalization ability. Simultaneously, the construction of multimodal feature vectors provides rich input information for subsequent machine learning models, helping to improve the stability and reliability of prediction results.

[0029] Step 103: Input the multimodal feature vector into the prediction model to obtain the target user's complexion index score and complexion grading result output by the prediction model.

[0030] In this embodiment, the prediction model is a machine learning model. Its input is a multimodal feature vector, and its output includes a complexion index and a corresponding complexion level. The complexion index is a continuous value, typically ranging from 0 to 10, used to quantify the target user's complexion state. The complexion level is a classification of the complexion index based on a preset score range. For example, the classification criteria include: poor (n<4), quite poor (4≤n≤5), moderate (5≤n≤6), good (6≤n≤7), and excellent (n>7), where n represents the complexion index. This classification method makes the evaluation results more intuitive and easier for the target user to understand.

[0031] In this embodiment, the training process of the prediction model includes feature selection, model parameter tuning, and cross-validation. Ultimately, by optimizing the structure and parameters of the prediction model, it achieves high prediction accuracy and generalization ability on test data. During the model application phase, simply inputting the multimodal feature vectors of new user (i.e., new target user) samples into the trained prediction model quickly outputs the complexion index and grading results for the new user samples, eliminating the need for manual intervention and significantly improving evaluation efficiency.

[0032] In practice, the application of predictive models not only improves the automation level of complexion assessment but also significantly shortens the time from data collection to result output, making the entire complexion assessment process more efficient. Furthermore, by continuously optimizing the performance of predictive models, the accuracy and stability of complexion assessment can be further improved, providing strong technical support for fields such as cosmetic efficacy verification and personalized beauty recommendations.

[0033] In summary, the multimodal data processing method provided in this application integrates multiple data sources, including skin physiological parameters, facial image features, subjective and objective evaluations, and electroencephalogram (EEG) signals, to construct a scientific and systematic complexion assessment system. This multimodal data processing method not only overcomes the limitations of traditional assessment methods but also achieves quantitative expression and grading of complexion status, providing a new technical path for research and practice in the cosmetics industry and related fields.

[0034] In some embodiments, obtaining the target user's skin physiological parameters includes: collecting multidimensional physiological parameters of the target user's face through a multi-parameter skin analyzer, including but not limited to indicators such as stratum corneum moisture content, skin color, luster, elasticity, moisture, transepidermal water loss, smoothness, skin blood flow, and collagen concentration, to obtain the target user's skin physiological parameters.

[0035] The multi-parameter skin analyzer includes a facial image analyzer, which is a non-invasive skin detection device based on multispectral imaging technology. It acquires standardized facial images through a combination of high-resolution cameras and multiple light sources, and performs multi-dimensional quantitative analysis of the skin.

[0036] The following is an example illustrating the acquisition of multidimensional physiological parameters: The moisture content of the stratum corneum can be measured using a professional skin stratum corneum moisture testing instrument based on the capacitance method. Measurement principle: Utilizing the positive correlation between stratum corneum moisture content and skin capacitance, a capacitance sensor at a specific frequency detects changes in local skin capacitance, which are then converted into a moisture content value. This measurement requires multiple tests (e.g., 6 times) and the average value taken.

[0037] Transepidermal water loss (TEWL) and evaporative heat loss (HL) can be measured using a multi-parameter skin physiological function testing device. Measurement principle: Based on the "open chamber method," a high-density sensor array is used to detect the diffusion gradient of water vapor and heat on the skin surface in real time, thereby quantifying TEWL and HL. The measurement requires multiple tests (e.g., 3 times) and the average value taken.

[0038] Skin glossiness can be measured using a high-precision instrument specifically designed to assess the optical properties of the skin surface. Measurement principle: Based on the reflection and scattering characteristics of light on the skin surface, skin glossiness is assessed by quantitatively analyzing the intensity of reflected light. The testing requirements are as follows: Multiple tests (e.g., 3 times) are performed, and the average value is taken.

[0039] The Erythema Index (EI) can be measured using a professional skin melanin and hemoglobin testing probe. Measurement principle: An algorithm converts the light absorption at a wavelength of 660nm into an Erythema Index (EI), typically ranging from 0 to 999, in arbitrary ink units. A higher EI value indicates a higher hemoglobin content in the skin, corresponding to more pronounced redness, inflammation, or telangiectasia. This measurement requires multiple tests (e.g., 3 times) and the average value taken.

[0040] The skin elasticity R2 value (i.e., skin resilience) can be measured using a skin elasticity tester. Measurement principle: It measures the skin's elasticity through suction and stretching principles. The core indicator, R2 value, is a key parameter for assessing skin firmness and elastic recovery ability. During the test, the instrument applies a controlled negative pressure (e.g., 200–500 mbar, 45 kPa) to the skin surface, drawing a localized area of ​​skin into a circular test probe with a diameter of 2–8 mm. The depth of skin absorption depends on its elasticity and firmness: better elasticity results in stronger resistance to deformation and a shallower absorption depth; conversely, weaker elasticity leads to a shallower absorption depth. After maintaining the negative pressure for a period, it is suddenly released, and the skin returns to its original shape under its own elastic recoil force. The instrument simultaneously records the displacement changes throughout the process. This measurement requires multiple tests (e.g., 3 times) and the average value taken.

[0041] Skin tone, or Individual Typology Angle (ITA), can be measured using image analysis software. The acquired images are multi-angle images: three sets of images are captured from the front (0°), left (37°), and right (37°) to ensure complete facial contours and lighting distribution, reducing misjudgments of skin tone due to angular deviations. Measurement principle: Based on multispectral imaging and RGB color space conversion, standardized image acquisition and quantitative algorithms calculate the brightness and yellow-red tint of skin tone, achieving objective quantification of skin tone.

[0042] Skin tone L The value (luminance component in the CIE Lab color space as defined by the International Commission on Illumination) can be measured using a facial image analyzer. Three sets of images are acquired: one from the front (0°), one from the left (37°), and one from the right (37°), ensuring that all facial areas (such as the cheeks, forehead, and jaw) are imaged under uniform lighting and geometric conditions, reducing the impact of shadows and angular deviations on luminance judgment. Measurement principle: Based on multispectral imaging technology and CIE Lab color space conversion.

[0043] Skin tone a The value (which is the red-green axis component in the CIE Lab color space defined by the International Commission on Illumination) can be measured using a facial image analyzer. Three sets of images are acquired: one from the front (0°), one from the left (37°), and one from the right (37°) to ensure complete facial contours and light and shadow distribution, avoiding misjudgment of local red areas due to angular deviations. Measurement principle: Based on multispectral imaging technology and CIE Lab color space conversion.

[0044] Skin tone b The value (the yellow and blue axis components in the CIE Lab color space as defined by the International Commission on Illumination) can be measured using a facial image analyzer. Three sets of images are acquired: one from the front (0°), one from the left (37°), and one from the right (37°), comprehensively covering the facial contour and avoiding missed detections of localized macular areas due to angular deviations. Measurement principle: Based on multispectral imaging technology and CIE Lab color space conversion.

[0045] The area of ​​the red zone on the skin can be measured using a facial image analyzer. Three sets of images are acquired: one from the front (0°), one from the left (37°), and one from the right (37°), ensuring consistent lighting, angle, and distance to improve repeatability and lateral contrast. Measurement principle: The facial image analyzer images the skin using three light sources: white light, polarized light, and red / brown / exogenous (RBX). RBX technology effectively separates and enhances red signals, making vascular red areas more clearly visible. By setting a specific threshold range for the red channel, the red zone pixels in the image are segmented and identified from the background area. The total area of ​​the red zone pixels is calculated and converted into actual skin area units (mm²).

[0046] Skin tone uniformity can be measured using a facial image analyzer. Three sets of images are captured under natural lighting conditions: a 0° frontal view, a 37° left side view, and a 37° right side view. These images accurately reflect the overall color and brightness distribution of the skin, forming the basis for assessing skin tone uniformity. Measurement principle: During image acquisition, multiple light sources and standard shooting angles are used to comprehensively record the color information of the skin surface and deeper layers. The image analysis stage completes the quantitative analysis of skin tone uniformity through the following steps: Image calibration and region segmentation: The image undergoes color standardization processing to eliminate color casts caused by equipment or the environment; key facial areas (such as cheeks, forehead, and nose) are identified, excluding non-skin interference such as hair and shadows; Color space conversion and parameter extraction: The image is converted from RGB to the Lab or HSV space, which is more suitable for skin tone analysis; Region comparison and heatmap generation: The system automatically compares the color difference (ΔE) between different regions, generating a pseudo-color heatmap to visually display areas of skin tone fluctuation—the darker the color (e.g., red / purple), the greater the deviation in skin tone; Output quantitative score: The final result is output in the form of "skin tone uniformity index" or "color difference standard deviation," supporting horizontal comparison and long-term tracking.

[0047] Wrinkle area can be measured using a facial image analyzer; three-angle imaging (0° front, 37° left, and 37° right) ensures data consistency. Measurement principle: Skin is imaged using three light sources: white light, polarized light, and RBX (red / brown / exogenous). RBX technology effectively separates and enhances wrinkle-related texture information. After image acquisition, the software extracts the wrinkle area from the background and calculates its area through image segmentation, threshold setting, and pixel analysis.

[0048] Skin roughness parameter (SEr) can be measured using a skin surface texture analysis system. Measurement principle: Based on high-resolution ultraviolet imaging combined with grayscale analysis technology, the fine structure of the skin surface is quantitatively evaluated.

[0049] The area of ​​wrinkles around the eyes can be measured using a rapid optical imaging system for the skin. Measurement principle: Based on high-precision three-dimensional optical imaging and quantitative analysis technology, it enables a non-contact, objective assessment of wrinkles in the corner of the eyes area.

[0050] The volume and average depth of crow's feet wrinkles can both be measured using a rapid optical imaging system for the skin. Measurement principle: Based on three-dimensional optical reconstruction technology, a high-precision digital model of the skin's microstructure is created, and algorithms automatically calculate quantitative parameters such as volume and depth.

[0051] In some embodiments, obtaining image features of a target user's facial image includes: acquiring a standardized frontal facial image of the target user using a facial image analyzer, and extracting facial region information under a black background, frontal view, and standard lighting conditions using an image analysis system algorithm to obtain image features of the target user's facial image.

[0052] Here, the target user's facial image can be one or more of the following processes performed on the acquired image: background unification, size and resolution adjustment. For example, the background can be unified to a monochrome, the face can be directly facing the imaging device, obvious jewelry can be removed, the head and neck areas can be preserved, and the image can be adjusted to a standard size (840×1080 pixels / inch).

[0053] In some embodiments, experts may score facial images based on multiple aspects, including: skin tone evenness, translucency and radiance, fair and rosy complexion, skin without dullness or sallowness, skin that is moisturized and not dry, skin firmness and lift, severity of eye wrinkles, severity of nasolabial folds, plumpness and fullness of skin, skin elasticity, skin smoothness, fineness of pores, brightness of eyes, elegance of eyebrows and eyelashes, overall overall appearance, overall vitality, mental state, overall complexion, and facial complexion; to generate a second evaluation index.

[0054] In some embodiments, the primary evaluation metric for the target user's assessment of the facial image is the result of the target user's self-subjective evaluation. The evaluation dimensions include the expert rating dimensions mentioned above, as well as multiple dimensions such as skin barrier fragility, skin tolerance, and skin's ability to resist external stimuli. For example, the electronic device guides the target user in conducting a self-subjective evaluation, using a nine-point Likert scale and other grading methods to record subjective rating information from multiple dimensions such as skin tone evenness, translucency, rosiness, skin brightness, vitality, and mental state, thereby obtaining a subjective evaluation score.

[0055] In some embodiments, the physiological characteristics of a reference user viewing facial images include electroencephalogram (EEG) signals. EEG signals are electrical signals generated by the activity of neurons in the cerebral cortex, recorded by a multi-channel EEG acquisition device. The EEG acquisition device records the brain activity of the reference user while observing facial images and making subjective evaluations. For example, the EEG signal acquisition device is a 32-channel Brain Products actiCHamp Plus EEG system; all electrodes are arranged according to the international 10-20 system, with a sampling rate of 500Hz, impedance controlled below 5kΩ, and a 0.1-30Hz bandpass filter and a 50Hz notch filter used to remove power frequency noise during acquisition.

[0056] In some embodiments, the prediction model includes a support vector regression model.

[0057] In this embodiment, Support Vector Regression (SVR) is a machine learning method based on statistical learning theory used for regression tasks. SVR constructs an optimal hyperplane to minimize error while maintaining the principle of minimizing structural risk, thereby achieving prediction of continuous target variables. Unlike traditional linear regression, SVR introduces a kernel function mechanism, enabling it to handle nonlinear relationships and allowing a certain degree of error tolerance through a soft-margin optimization strategy. The Support Vector Regression model is suitable for high-dimensional data spaces and is well-suited to the multimodal feature fusion scenario in this application.

[0058] In this embodiment, the training data for the support vector regression model consists of a large number of samples, each containing a normalized multimodal feature vector and a corresponding expert complexion rating label. By training the model using a combination of the multimodal feature vectors and expert complexion rating labels, the support vector regression model can learn the complex mapping relationship between the multimodal feature vectors and complexion, and ultimately output a complexion index and a corresponding complexion level determination.

[0059] In practical applications, multimodal feature vectors include information such as skin parameters, facial image features, subjective ratings, expert ratings, and electroencephalogram (EEG) signals. Electronic devices standardize, reduce dimensionality, and fuse these features before inputting the processed features into a support vector regression (SVR) model. The SVR model uses its internal algorithm to calculate a complexion index and categorizes target users into different complexion levels based on preset interval standards. For example, these preset interval standards include: poor (n<4), quite poor (4≤n≤5), moderate (5≤n≤6), good (6≤n≤7), and excellent (n>7), where n represents the complexion index.

[0060] In this embodiment, by employing a support vector regression model as the prediction model, the nonlinear relationships that may exist in multimodal data can be effectively addressed, improving the model's generalization ability and prediction accuracy. Furthermore, the support vector regression model possesses strong robustness, maintaining high stability even under significant noise interference, thereby enhancing the overall reliability of the evaluation system.

[0061] In practical applications, after multimodal data collection is completed for new users, the electronic device automatically extracts the user's features and generates feature vectors. These feature vectors are then input into a pre-trained support vector regression model to quickly obtain the user's complexion index and complexion level. This process not only improves the efficiency of cosmetic efficacy evaluation but also ensures the scientific rigor and objectivity of the evaluation results, providing reliable technical support for cosmetic efficacy verification and personalized beauty recommendations.

[0062] In some embodiments, step 102 generates a multimodal feature vector based on skin physiological parameters, image features, a first evaluation index, a second evaluation index, and physiological features. This can be achieved through methods such as... Figure 2 The steps shown are to be completed. Figure 2 This is a schematic diagram of the second process of the multimodal data processing method provided in the embodiments of this application; Step 201: Standardize the skin physiological parameters, image features, first evaluation index, second evaluation index, and physiological features respectively to obtain standardized skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features.

[0063] In this embodiment, standardization refers to the process of transforming data from different sources and with different dimensions to the same scale to facilitate subsequent model training and feature fusion. This application can employ Z-score standardization (i.e., subtracting the mean and dividing by the standard deviation) to ensure the data has zero mean and unit variance. For example, in this application, skin physiological parameters may be distributed across different numerical ranges. By using standardization, dimensional differences can be eliminated, thereby improving the model's generalization ability and stability.

[0064] In practical applications, the Z-score standardized calculation formula is as follows: ; in, The standardized feature values ​​are calculated using the Z-score standardization method. This represents the raw feature values, i.e., the feature data that has not undergone standardization. This represents the mean of the eigenvalues, used for Z-score standardization. The standard deviation of the eigenvalues ​​is used for Z-score standardization.

[0065] In practical applications, standardization is applied to skin physiological parameters, image features, the first evaluation metric, the second evaluation metric, and physiological features respectively. This effectively prevents features with large values ​​from dominating the model learning process and causing model bias. Standardization of these features results in a more balanced weight distribution among them in the model, thereby improving prediction accuracy and enhancing the reliability of evaluation results.

[0066] Step 202: Dimensionality reduction is performed on the standardized skin physiological parameters to obtain the dimensionality-reduced skin physiological parameters.

[0067] In this embodiment, dimensionality reduction is a data compression technique used to reduce feature dimensions while retaining key information. In this application, principal component analysis (PCA) can be used for dimensionality reduction. The principle of PCA is to project high-dimensional data into a low-dimensional space through linear transformation, while ensuring that as much variance information of the original data as possible is preserved.

[0068] In practical applications, several principal components are calculated based on PCA. These principal components are linear combinations of the original features and are orthogonal to each other. When the cumulative variance explained is greater than 80%, this application considers that most of the important information has been retained. Dimensionality reduction not only reduces model complexity but also removes redundant features, thereby improving training efficiency and predictive performance. Furthermore, dimensionality reduction can alleviate overfitting, making the model more adaptable to new samples.

[0069] In practical applications, PCA is used to reduce the dimensionality of the normalized skin parameters. The principal component retention criterion is a cumulative variance explanation rate greater than 80%. Variables that significantly contribute to changes in complexion are extracted from the PCA principal component loadings. The formula for calculating the cumulative variance explanation rate is as follows: ; in, The cumulative variance explained represents the percentage of variance explained. The proportion of total variance explained by each principal component In principal component analysis, the first... Eigenvalues ​​of each principal component; This represents the total number of principal components, i.e., the original feature dimension.

[0070] The principal components extracted by PCA are concatenated with the scoring data and EEG features to serve as the multimodal feature vector input to the complexion assessment model.

[0071] Step 203: The dimensionality-reduced skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features are concatenated to generate a multimodal feature vector.

[0072] In this embodiment, the concatenation operation refers to sequentially joining multiple preprocessed feature vectors into a single complete vector, which serves as the input to the machine learning model. In this application, the concatenated multimodal feature vector integrates various information such as skin physiological data, facial image features, subjective and expert ratings, and EEG neural signals, thus achieving cross-modal information fusion. This cross-modal information fusion method not only preserves the independent characteristics of each modality but also enhances the overall feature expressiveness.

[0073] In some embodiments, the above steps perform dimensionality reduction processing on the standardized skin physiological parameters to obtain dimensionality-reduced skin physiological parameters, which can be achieved through methods such as... Figure 3 The steps shown are to be completed. Figure 3 This is a schematic diagram of the third process of the multimodal data processing method provided in the embodiments of this application; Step 301: Use principal component analysis algorithm to reduce the dimensionality of standardized skin physiological parameters to obtain the feature values ​​of multiple principal components.

[0074] In this embodiment, Principal Component Analysis (PCA) is a multivariate statistical analysis method that projects high-dimensional data into a low-dimensional space through linear transformation while preserving as much information as possible from the original data. PCA effectively removes redundant information from the data and extracts the main directions of change, i.e., the principal components. In this embodiment, PCA is used to reduce the dimensionality of standardized skin physiological parameters to reduce the complexity of subsequent model training and improve the model's generalization ability.

[0075] In principal component analysis, eigenvalues ​​refer to the variance of each principal component. Eigenvalues ​​reflect the contribution of each principal component to the variability of the data. Larger eigenvalues ​​indicate that the principal component contains more information and has a stronger explanatory power for the data.

[0076] In practical applications, principal component analysis (PCA) can transform the complex set of skin physiological parameters into a small number of representative principal components, thereby reducing computational burden and retaining most of the key information. Dimensionality reduction using PCA improves the efficiency and stability of complexion assessment models, preventing overfitting caused by too many parameters when processing data.

[0077] Step 302: Calculate the cumulative variance explained rate based on the eigenvalues ​​of multiple principal components.

[0078] In this embodiment, the cumulative variance explained rate refers to the proportion of the sum of the eigenvalues ​​of the first k principal components to the total sum of all eigenvalues. This proportion is used to measure the degree to which the first k principal components cover the variability of the original data. For example, when the cumulative variance explained rate reaches 80%, it indicates that the first k principal components have contained 80% of the effective information of the original data.

[0079] In this embodiment, the cumulative variance explained ratio is used to determine the number of principal components to be retained. This application selects a cumulative variance explained ratio greater than 80% as the retention criterion to ensure that the dimensionality-reduced data can still reflect the structure and characteristics of the original data well. By calculating the cumulative variance explained ratio, the influence of each principal component on the complexion assessment results can be scientifically determined, thereby rationally selecting the number of principal components to retain and improving the accuracy and robustness of the model.

[0080] Step 303: Based on the cumulative variance explained rate, extract dimensionality-reduced skin physiological parameters from the eigenvalues ​​of multiple principal components.

[0081] In this embodiment, based on the cumulative variance explained rate, principal components whose cumulative variance explained rate exceeds a preset threshold (e.g., 80%) can be selected, and these principal components are output as the final dimensionality reduction result. The principal components whose cumulative variance explained rate exceeds the preset threshold constitute a new skin physiological parameter vector, which is used for subsequent feature fusion and model training.

[0082] By using the dimensionality reduction methods described above, the dimensionality of the input data can be significantly reduced while maintaining the main features of the data; it can also improve the generalization ability of the model, thereby helping the model to more accurately predict the complexion index of the target user.

[0083] In this embodiment, principal component analysis (PCA) is used to reduce the dimensionality of standardized skin physiological parameters, and representative principal components are selected based on the cumulative variance explained rate, thereby generating dimensionality-reduced skin physiological parameters. The dimensionality reduction method employed by PCA effectively reduces data redundancy and improves the operating efficiency of the complexion assessment model. The dimensionality reduction method used by PCA helps to better support the training and prediction tasks of the complexion assessment model. Furthermore, the dimensionality reduction method employed by PCA enables more accurate and faster complexion assessment for Chinese women.

[0084] In some embodiments, obtaining the physiological characteristics of the reference user when viewing the facial image in step 101 can be achieved through methods such as... Figure 4 The steps shown are to be completed. Figure 4 This is a schematic diagram of the fourth process of the multimodal data processing method provided in the embodiments of this application; Step 401: Obtain the electroencephalogram (EEG) signals of the reference user while viewing the facial image.

[0085] In this embodiment, the reference users include multiple recruited users. Electroencephalogram (EEG) signals refer to the electrical signals generated by the activity of neurons in the cerebral cortex, recorded by a multi-channel EEG acquisition device. EEG signals can reflect the cognitive processing state and emotional response of the brain during image viewing, and are an important neurophysiological indicator for assessing the reference users' perception of color and atmosphere. EEG signals are typically measured in millivolts (mV) and digitally recorded using a sampling rate (e.g., 500 Hz).

[0086] In practical applications, the international 10-20 electrode system is used during the acquisition process to ensure coverage of key areas such as the frontal, parietal, and occipital lobes, thereby comprehensively capturing neural activity related to gas color assessment. For example, in this application, a 32-channel Brain Products actiCHamp Plus EEG system is used for synchronous recording, ensuring signal quality and spatial resolution.

[0087] Step 402: Perform data preprocessing on the EEG signal to obtain the preprocessed signal; the preprocessed signal has a higher signal-to-noise ratio than the EEG signal.

[0088] In this embodiment, data preprocessing involves artifact removal, filtering, and rereference operations on the raw EEG signal to remove interference from eye movement, electromyography (EMG), and power line noise, thereby improving signal quality. Exemplary preprocessing methods include spherical interpolation with bad derivatives, full mean-referenced rereference, and Independent Component Analysis (ICA) to automatically remove eye movement and EMG artifacts. Furthermore, a 0.1-30Hz bandpass filter and a 50Hz notch filter are used to further remove irrelevant frequency components. The preprocessed signal has a higher signal-to-noise ratio (SNR), meaning a larger proportion of useful signal to background noise, thus more accurately reflecting the brain's true activity. For example, after ICA processing, eye movement artifacts in the EEG signal are effectively separated and removed, eliminating them from the signal. This processing method allows subsequent analysis to focus more on neural responses related to complexion assessment.

[0089] Step 403: Extract time-frequency features from the preprocessed signal to obtain event-related spectrum perturbation features.

[0090] In this embodiment, Event-Related Spectral Perturbation (ERSP) analysis is a time-frequency analysis technique that can display the power changes of a certain frequency component before and after stimulation. The EEG signal processing method selects a time window from -500ms before stimulation to 2000ms after stimulation. This application uses Short-Time Fourier Transform (STFT) to achieve time-frequency decomposition of the EEG signal and uses the 500ms before stimulation as a baseline for spectral correction. This application can quantify the power changes of the signal using the ERSP calculation formula, which is: ; in, Frequency, measured in Hertz (Hz), represents the frequency components of an electroencephalogram (EEG) signal. The time unit is milliseconds (ms), representing the time point of the EEG signal; This represents the power spectral density in the activated state, that is, the power value at a certain frequency and time point within a period of time after stimulation. This represents the power spectral density at baseline, which is the average power value at a certain frequency during the 500ms period before stimulation.

[0091] All trial data were grouped according to subjective complexion assessment results (good complexion / poor complexion), and the test statistic between the two groups was calculated at each electrode, time, and frequency point. (Used to measure whether there is a significant difference between two groups at a certain time frequency point), its calculation formula is: ; in, Channel (full name: Channel, can also be abbreviated as...) ), indicating the electrode channel number in the EEG acquisition system; The time unit is milliseconds (ms), representing the time point of the EEG signal; Frequency, measured in Hertz (Hz), represents the frequency components of an electroencephalogram (EEG) signal. , The values ​​are the means of the two sets of data, representing the ERSP values ​​of the "good complexion" and "poor complexion" groups at a certain time frequency point. , The variances of the two sets of data are respectively, representing the variances of the "good complexion" and "poor complexion" groups at a certain time frequency point; , The numbers represent the number of samples in the two groups, and the number of trials for the "good complexion" and "poor complexion" groups, respectively.

[0092] Adjacency spatial-temporal-frequency points were divided into functional clusters, and based on all significant... Absolute values ​​and the definition of cluster statistics (representing all significant) (sum of absolute values) ; in, This indicates a significant clustering region, representing a spatiotemporal frequency region with statistical significance identified through cluster analysis.

[0093] Using a permutation test, the labels were shuffled 1000 times to construct the null distribution of the maximum cluster statistic. The value (used to determine statistical significance) is calculated using the following formula: ; in, This represents the maximum clustering statistic generated in the permutation test; This represents the maximum cluster statistic obtained from the actual observed data; This indicates the number of permutations, for example, 1000.

[0094] Step 404: Determine physiological characteristics based on event-related spectrum perturbation features.

[0095] In this embodiment, physiological characteristics refer to quantitative indicators extracted from electroencephalogram (EEG) signal analysis that reflect the reference user's cognitive and emotional state regarding the complexion image. In this embodiment, physiological characteristics mainly include... wave and The mean ERSP value of the wave band, among which wave and The mean ERSP value of the frequency band can reflect the difference in neural response of reference users when distinguishing between faces with good and bad complexion. In practical applications, a permutation test is used to identify significant clustering regions, and these regions are then extracted separately. wave and The mean ERSP value of the frequency band is used as the input feature.

[0096] As demonstrated by the steps described above, this application extracts neurophysiological features related to the process of cognizing complexion by acquiring electroencephalogram (EEG) signals, performing preprocessing and time-frequency analysis. This allows for a more accurate reflection of the user's subjective feelings and judgments about complexion, thereby improving the scientific rigor and objectivity of complexion assessment. Furthermore, it provides reliable data support for verifying the efficacy of cosmetics and providing personalized beauty recommendations.

[0097] In some embodiments, step 404 determines physiological characteristics based on event-related spectrum perturbation features, which can be achieved through methods such as... Figure 5 The steps shown are to be completed. Figure 5This is a schematic diagram of the fifth process of the multimodal data processing method provided in the embodiments of this application; Step 501: Based on the event-related spectrum perturbation characteristics, determine significant spatiotemporal regions.

[0098] In this embodiment, a significant spatiotemporal region refers to a region in the time-frequency domain of an electroencephalogram (EEG) signal that exhibits significant differences, identified through statistical testing methods. Significant spatiotemporal regions typically reflect the brain's cognitive processing of specific stimuli (such as facial complexion). For example, in this application, a permutation test was used to identify that when a reference user viewed images of faces with good and bad complexions, the frontal lobe region... Wave (4–8Hz) and The region exhibiting significant energy changes within the 8–13 Hz frequency band. This region is considered closely related to the reference user's perception of complexion and emotional state.

[0099] In this embodiment, the selection of significant spatiotemporal regions is based on the clustering analysis results of EEG data. By comparing the differences in ERSP (Event Related Spectrum Perturbation) values ​​in the time-frequency space between different groups (e.g., the good complexion group and the poor complexion group), statistically significant regions are selected. This process of comparing the differences in ERSP values ​​between different groups in the time-frequency space and selecting statistically significant regions ensures that the selected regions can represent neural activity related to complexion evaluation, rather than random fluctuations or interference signals.

[0100] Step 502: Extract the mean values ​​of event-related spectrum perturbations in the first and second frequency bands within a significant spatiotemporal region as physiological features.

[0101] In this embodiment, the first frequency band and the second frequency band respectively refer to Wave (4–8Hz) and wave (8–13Hz), wave and Waves have clear neurological functional significance in electroencephalogram (EEG) analysis. Waves primarily reflect higher cognitive processes in the brain, such as emotion regulation and attention control. Waves are related to the brain's basic sensory processing and activation states. In this application, the electronic device... wave and The energy changes of the wave are averaged to quantify the brain activity characteristics of a reference user when viewing facial images and making subjective evaluations.

[0102] In this embodiment, the mean event-related spectrum perturbation is a value obtained by averaging the ERSP values ​​within a significant spatiotemporal region, used to represent the overall energy change level within that region. For example, within a certain significant spatiotemporal region, if... A higher mean ERSP value in a band may indicate that the reference user had a strong attentional or emotional response to the presented complexion image; conversely, if... A lower mean ERSP value for a band may indicate that the reference user is in a more relaxed or automated processing state.

[0103] By wave and Using the mean of the event-related spectrum perturbation of the wave as a physiological feature input model can more accurately capture the reference user's neurocognitive representation of their complexion, thereby improving the accuracy and generalization ability of the prediction model.

[0104] In practical applications, extract from the salient spatiotemporal regions identified by the cluster permutation test, respectively Wave (4–8Hz) and The mean ERSP value in the 8–13 Hz frequency band is used as the neurophysiological input feature of the complexion index prediction model of this invention. Its calculation formula is as follows: ; ; in, , They represent Wave (4–8Hz) and The mean ERSP value in the 8–13 Hz band is used as an input feature for the color index prediction model; This represents the significant clustering regions identified by the permutation test, used to extract the ERSP mean. This represents a combination of spatiotemporal points located within a significant clustering region, that is, all spatiotemporal coordinate points within the significant clustering region that simultaneously satisfy the time point and motor channel conditions.

[0105] In summary, in this embodiment of the application, significant spatiotemporal regions are determined based on event-related spectrum perturbation features, and extraction is performed within these significant spatiotemporal regions. wave and The mean of the event-related spectrum perturbation of the wave is used as a physiological characteristic. Using the above method, we can effectively focus on EEG activity closely related to complexion perception, thereby improving the model's accuracy in recognizing the user's complexion state and achieving a more scientific and objective assessment of the complexion of Chinese women.

[0106] This application also provides a multimodal data processing device for use in electronic devices, such as... Figure 6 As shown, the multimodal data processing device includes: The acquisition unit 601 is used to acquire the target user's skin physiological parameters, the image features of the target user's facial image, the target user's first evaluation index of the facial image, the expert's second evaluation index of the facial image, and the physiological characteristics of the reference user when viewing the facial image. The processing unit 602 is used to generate a multimodal feature vector based on skin physiological parameters, image features, a first evaluation index, a second evaluation index, and physiological features. The processing unit 602 is used to input the multimodal feature vector into the prediction model to obtain the target user's complexion index score and complexion grading result output by the prediction model.

[0107] In some embodiments, the processing unit 602 is configured to perform standardization processing on skin physiological parameters, image features, a first evaluation index, a second evaluation index, and physiological features respectively to obtain standardized skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features; perform dimensionality reduction processing on the standardized skin physiological parameters to obtain dimensionality-reduced skin physiological parameters; and concatenate the dimensionality-reduced skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features to generate a multimodal feature vector.

[0108] In some embodiments, the processing unit 602 is used to perform dimensionality reduction processing on standardized skin physiological parameters using a principal component analysis algorithm to obtain the feature values ​​of multiple principal components; calculate the cumulative variance explained rate based on the feature values ​​of multiple principal components; and extract the dimensionality-reduced skin physiological parameters from the feature values ​​of multiple principal components based on the cumulative variance explained rate.

[0109] In some embodiments, the acquisition unit 601 is used to acquire the electroencephalogram (EEG) signal of a reference user when viewing a facial image; the processing unit 602 is used to perform data preprocessing on the EEG signal to obtain a preprocessed signal; the preprocessed signal has a higher signal-to-noise ratio than the EEG signal; time-frequency features are extracted from the preprocessed signal to obtain event-related spectrum perturbation features; and physiological features are determined based on the event-related spectrum perturbation features.

[0110] In some embodiments, the processing unit 602 is used to determine a significant spatiotemporal region based on the event-related spectrum perturbation features; and to extract the mean values ​​of the event-related spectrum perturbations of the first frequency band and the second frequency band within the significant spatiotemporal region as physiological features.

[0111] In some embodiments, the prediction model includes a support vector regression model.

[0112] This application also provides an electronic device, such as... Figure 7 As shown, electronic device 700 includes: Memory 701 is used to store computer-executable instructions or computer programs; When processor 702 executes computer-executable instructions or computer programs stored in memory 701, it performs the following steps: Acquire the target user's skin physiological parameters, the target user's facial image features, the target user's first evaluation index of the facial image, the expert's second evaluation index of the facial image, and the physiological characteristics of the reference user when viewing the facial image. Based on skin physiological parameters, image features, first evaluation index, second evaluation index, and physiological features, a multimodal feature vector is generated. By inputting the multimodal feature vector into the prediction model, the target user's complexion index score and complexion grading result are obtained from the prediction model output.

[0113] In some embodiments, the processor 702 is configured to perform standardization processing on skin physiological parameters, image features, a first evaluation index, a second evaluation index, and physiological features respectively to obtain standardized skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features; perform dimensionality reduction processing on the standardized skin physiological parameters to obtain dimensionality-reduced skin physiological parameters; and concatenate the dimensionality-reduced skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features to generate a multimodal feature vector.

[0114] In some embodiments, the processor 702 is configured to perform dimensionality reduction processing on standardized skin physiological parameters using a principal component analysis algorithm to obtain eigenvalues ​​of multiple principal components; calculate the cumulative variance explained rate based on the eigenvalues ​​of the multiple principal components; and extract the dimensionality-reduced skin physiological parameters from the eigenvalues ​​of the multiple principal components based on the cumulative variance explained rate.

[0115] In some embodiments, the processor 702 is configured to acquire electroencephalogram (EEG) signals when a reference user views a facial image; perform data preprocessing on the EEG signals to obtain a preprocessed signal; the preprocessed signal has a higher signal-to-noise ratio than the EEG signals; extract time-frequency features from the preprocessed signal to obtain event-related spectrum perturbation features; and determine physiological characteristics based on the event-related spectrum perturbation features.

[0116] In some embodiments, the processor 702 is configured to determine a significant spatiotemporal region based on event-related spectrum perturbation features; and extract the mean values ​​of event-related spectrum perturbations of a first frequency band and a second frequency band within the significant spatiotemporal region as physiological features.

[0117] In some embodiments, the prediction model includes a support vector regression model.

[0118] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimodal data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0119] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the multimodal data processing method described in this application.

[0120] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the multimodal data processing method provided in this application. For example, ... Figure 1 The multimodal data processing method is shown.

[0121] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0122] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0123] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0124] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0125] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A multimodal data processing method, characterized in that, The method includes: Acquire the target user's skin physiological parameters, the image features of the target user's facial image, the target user's first evaluation index of the facial image, the expert's second evaluation index of the facial image, and the physiological characteristics of the reference user when viewing the facial image; Based on the skin physiological parameters, the image features, the first evaluation index, the second evaluation index, and the physiological features, a multimodal feature vector is generated. The multimodal feature vector is input into the prediction model to obtain the target user's complexion index score and complexion grading result output by the prediction model.

2. The method according to claim 1, characterized in that, The step of generating a multimodal feature vector based on the skin physiological parameters, the image features, the first evaluation index, the second evaluation index, and the physiological features includes: The skin physiological parameters, the image features, the first evaluation index, the second evaluation index, and the physiological features are respectively standardized to obtain standardized skin physiological parameters, standardized image features, standardized first evaluation index, standardized second evaluation index, and standardized physiological features. The standardized skin physiological parameters are subjected to dimensionality reduction processing to obtain dimensionality-reduced skin physiological parameters; The reduced skin physiological parameters, the standardized image features, the standardized first evaluation index, the standardized second evaluation index, and the standardized physiological features are concatenated to generate the multimodal feature vector.

3. The method according to claim 2, characterized in that, The dimensionality reduction processing of the standardized skin physiological parameters to obtain dimensionality-reduced skin physiological parameters includes: Principal component analysis algorithm was used to reduce the dimensionality of the standardized skin physiological parameters to obtain the feature values ​​of multiple principal components. Based on the eigenvalues ​​of the principal components, the cumulative variance explained rate is calculated; Based on the cumulative variance explained rate, the dimensionality-reduced skin physiological parameters are extracted from the eigenvalues ​​of the multiple principal components.

4. The method according to claim 1, characterized in that, Obtain the physiological characteristics of the reference user when viewing facial images, including: Acquire the electroencephalogram (EEG) signals of the reference user while viewing the facial image; The EEG signal is preprocessed to obtain a preprocessed signal; the preprocessed signal has a higher signal-to-noise ratio than the EEG signal. Time-frequency features are extracted from the preprocessed signal to obtain event-related spectrum perturbation features; The physiological characteristics are determined based on the event-related spectrum perturbation features.

5. The method according to claim 4, characterized in that, The determination of the physiological characteristics based on the event-related spectral perturbation features includes: Based on the event-related spectral perturbation characteristics, significant spatiotemporal regions are identified; The mean values ​​of event-related spectral perturbations in the first and second frequency bands are extracted within the significant spatiotemporal region and used as the physiological characteristics.

6. The method according to any one of claims 1 to 5, characterized in that, The prediction model includes: support vector regression model.

7. A multimodal data processing device, characterized in that, The device includes: The acquisition unit is used to acquire the skin physiological parameters of the target user, the image features of the target user's facial image, the first evaluation index of the target user on the facial image, the second evaluation index of the expert on the facial image, and the physiological characteristics of the reference user when viewing the facial image. The processing unit is configured to generate a multimodal feature vector based on the skin physiological parameters, the image features, the first evaluation index, the second evaluation index, and the physiological features. The processing unit is used to input the multimodal feature vector into the prediction model to obtain the target user's complexion index score and complexion grading result output by the prediction model.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images