A skin care auxiliary method, device, equipment and medium based on AR glasses

Abstract

This application provides a skin care assistance method, device, equipment, and medium based on AR glasses, relating to the technical field of augmented reality. The method includes: obtaining a facial image and sensor data of a target to be detected; extracting a first feature based on the facial image and a second feature based on the sensor data; fusing the historical skin care behavior data of the target to be detected with the first and second features to obtain fused data; and generating corresponding skin care recommendations based on analysis of the fused data. This application enables users to obtain skin care recommendations more conveniently.

Description

Technical Field

[0001] The present application relates to the technical field of augmented reality, and in particular to a skin care auxiliary method, device, equipment and medium based on AR glasses. Background Art

[0002] Currently, when users seek skincare advice, they typically seek out a professional skincare professional, such as a beautician or dermatologist. While this traditional method can provide a certain degree of personalized guidance, it has several significant limitations. First, users need to spend time and energy to make appointments and personally visit a skincare professional, which is often impractical in our fast-paced lives. Seeking professional skincare advice is particularly difficult for users who live in remote areas or are time-strapped. Second, the uneven number and distribution of professional skincare professionals further limits the ease with which users can obtain skincare advice. In some areas, professional skincare resources may be relatively scarce, making it difficult for users to find a suitable skincare consultant. Therefore, users seeking skincare advice using this method suffers from poor convenience. Therefore, there is an urgent need for a method that allows users to obtain skincare advice more conveniently. Summary of the Invention

[0003] The present application provides a skin care assistance method, device, equipment and medium based on AR glasses, which can enable users to obtain skin care advice more conveniently.

[0004] In a first aspect of the present application, a skin care assistance method based on AR glasses is provided, the method comprising:

[0005] Obtain facial images and sensor data for the target to be detected;

[0006] extracting a first feature based on the facial image and extracting a second feature based on the sensor data;

[0007] According to the acquired historical skin care behavior data of the target to be detected, the historical skin care behavior data is fused with the first feature and the second feature to obtain fused data;

[0008] Based on the analysis of the fused data, corresponding skin care recommendations are generated.

[0009] By employing these technologies, users can capture facial images and sensor data in real time, analyze it in combination with historical skincare behavior data, and generate personalized skincare recommendations. Leveraging the instant feedback and visual prompts provided by AR glasses, users can receive precise skincare guidance without having to schedule an appointment with a skincare professional, significantly improving the convenience and timeliness of obtaining skincare recommendations.

[0010] Optionally, before generating corresponding skin care recommendations based on the analysis of the fused data, the method further includes:

[0011] ;

[0012] Where: Fusion(Q,K,V) is the fusion data, W O is the output weight matrix, head i is the i-th data, i∈(1, 2,…, n);

[0013] ;

[0014] ;

[0015] Among them, Q is the first feature, K is the second feature, V is the historical skin care behavior data, d k is the dimension of the key vector, W i Q is the weight matrix corresponding to the first feature, W i K is the weight matrix corresponding to the second feature, W i V is the weight matrix of the historical skin care behavior data.

[0016] By adopting the above technical solution, the first feature (facial image feature), the second feature (sensor data feature) and the historical skin care behavior data are integrated through the multi-head attention mechanism to generate a comprehensive feature representation. The specific implementation method is to use multiple attention heads to perform weighted calculations on the input features respectively, and then concatenate the outputs of multiple attention heads and then output the weight matrix W. O This mechanism can capture the complex relationships between different features, improve the effectiveness and accuracy of feature fusion, and thus provide a more accurate and comprehensive information basis for generating personalized skin care recommendations.

[0017] Optionally, before generating corresponding skin care recommendations based on the analysis of the fused data, the method further includes:

[0018] Acquire skin care operation images and skin care product images of the target to be detected;

[0019] Identifying the operation steps of the target to be detected based on the skin care operation step image;

[0020] Based on the fused data, the operation step score of the operation step is calculated. The specific calculation formula is as follows:

[0021] ;

[0022] Where S(t) is the score of the operation steps, σ is the activation function, and W s is the weight matrix of step classification, LSTM is the long short-term memory network, and Fusion(Q, K, V) is the fusion data;

[0023] extracting skin care product features based on the skin care product image;

[0024] Determining, based on the operating steps, recommended skin care product features of a recommended skin care product, the recommended skin care product being the skin care product corresponding to the operating steps;

[0025] Calculate the product similarity between the skin care product features and the recommended skin care product features:

[0026] ;

[0027] Wherein, Similarity(p,q) is the product similarity, GCN(p,q) represents the feature similarity calculation based on graph convolutional network, p is the feature of the skin care product, and q is the feature of the recommended skin care product.

[0028] By employing this technical solution, we capture and analyze user skincare application and product images, then use a long short-term memory (LSTM) network and a multi-head attention mechanism (Fusion(Q,K,V)) to score each application step. A graph convolutional network (GCN) then calculates product feature similarity to generate personalized skincare recommendations. This approach accurately identifies user application steps, assesses their effectiveness, and recommends the most appropriate products, achieving targeted optimization and improvement, providing users with more professional and personalized skincare guidance.

[0029] Optionally, generating corresponding skin care recommendations based on the analysis of the fused data specifically includes:

[0030] Calculate the operation step score based on the fusion data, the operation step score, and the product similarity:

[0031] ;

[0032] Among them, Score(u) is the operation step score, S(t) is the operation step score, Similarity(p,q) is the product similarity, Fusion(Q,K,V) is the fusion data, α is the weight parameter corresponding to the operation step score, β is the weight parameter corresponding to the product similarity, γ is the weight parameter corresponding to the fusion data, λ is the regularization coefficient, ||θ|| 2Represents the L2 norm of the model parameters;

[0033] If it is determined that the score of the operation step is less than a preset threshold, prompt information is generated based on the operation step.

[0034] By employing the above technical solution, a step score is generated by comprehensively calculating the step score, product similarity, and fused data, and a regularization term is incorporated to prevent overfitting. Specifically, the step score calculation formula can comprehensively evaluate the effectiveness of each skincare step. If the step score falls below a preset threshold, the system generates improvement suggestions, prompting the user to optimize the skincare steps or choose more appropriate skincare products. This approach ensures that users receive personalized, data-driven skincare guidance, improving skincare results and user satisfaction.

[0035] Optionally, if it is determined that the score of the operation step is less than a preset threshold, generating prompt information based on the operation step specifically includes:

[0036] Retrieve multiple preset prompt information from the preset database;

[0037] Determine the preset prompt information corresponding to the operation steps of the multiple preset prompt information, and display the preset prompt information corresponding to the operation steps on the AR glasses.

[0038] By employing this technical solution, when an operation step's score falls below a preset threshold, multiple preset prompts are retrieved from a preset database, and the prompt corresponding to the current operation step is determined and displayed in real time on the AR glasses. This ensures that users receive timely and targeted improvement suggestions, helping them adjust their skincare procedures to improve effectiveness and efficiency, while also enhancing the convenience and personalization of the user experience.

[0039] Optionally, the step of fusing the historical skin care behavior data of the target to be detected with the first feature and the second feature to obtain fused data specifically includes:

[0040] determining a first acquisition time of the first feature and a second acquisition time of the second feature;

[0041] Aligning the first feature and the second feature according to the first acquisition time and the second acquisition time to obtain a feature group;

[0042] The historical skin care behavior data corresponding to the feature group is determined, and the feature group is fused with the historical skin care behavior data to obtain the fused data.

[0043] By employing this technical solution, the first feature (facial image feature) and the second feature (sensor data feature) are aligned based on their respective acquisition times to form feature groups. These feature groups are then fused with the user's historical skincare behavior data to generate comprehensive fused data. This method provides a more comprehensive and accurate description of the user's skin condition. Combined with historical data for in-depth analysis, it generates personalized skincare recommendations, improving targeted skincare results and user satisfaction.

[0044] Optionally, the extracting the first feature based on the facial image and the extracting the second feature based on the sensor data further includes:

[0045] Extracting appearance features from the facial image to obtain the first features;

[0046] The sensor data is converted into a data matrix to obtain the second feature, wherein the sensor data includes skin temperature data and humidity data.

[0047] By employing this technical solution, facial images are extracted for appearance features to obtain the first feature, while sensor data (including skin temperature and humidity) is converted into a data matrix to obtain the second feature. This approach comprehensively captures the user's skin condition from multiple dimensions. Leveraging the combined information of image features and environmental parameters, it provides precise data support for subsequent analysis and personalized skincare recommendations, thereby enhancing the scientific nature and effectiveness of skincare regimens.

[0048] In a second aspect of the present application, a skin care auxiliary device based on AR glasses is provided, the device comprising an acquisition module, an extraction module, a processing module, and an output module, wherein:

[0049] The acquisition module is used to acquire facial images and sensor data of the target to be detected;

[0050] The extraction module is configured to extract a first feature based on the facial image and a second feature based on the sensor data;

[0051] The processing module is configured to fuse the acquired historical skin care behavior data of the target to be detected with the first feature and the second feature to obtain fused data;

[0052] The output module is used to generate corresponding skin care suggestions based on the analysis of the fused data.

[0053] Optionally, the processing module is configured to, before generating corresponding skin care recommendations based on the analysis of the fused data, further comprise:

[0054] ;

[0055] Where: Fusion(Q,K,V) is the fusion data, W O is the output weight matrix, head i is the i-th data, i∈(1, 2,…, n);

[0056] ;

[0057] ;

[0058] Among them, Q is the first feature, K is the second feature, V is the historical skin care behavior data, d k is the dimension of the key vector, W i Q is the weight matrix corresponding to the first feature, W i K is the weight matrix corresponding to the second feature, W i V is the weight matrix of the historical skin care behavior data.

[0059] Optionally, the acquisition module is used to acquire skin care operation images and skin care product images of the target to be detected;

[0060] The extraction module is used to identify the operation steps of the target to be detected based on the skin care operation step image;

[0061] The processing module is used to calculate the operation step score of the operation step based on the fusion data. The specific calculation formula is as follows:

[0062] ;

[0063] Where S(t) is the score of the operation steps, σ is the activation function, and W s is the weight matrix of step classification, LSTM is the long short-term memory network, and Fusion(Q, K, V) is the fusion data;

[0064] The extraction module is used to extract skin care product features based on the skin care product image;

[0065] The processing module is configured to determine, based on the operation steps, recommended skin care product characteristics, wherein the recommended skin care product is the skin care product corresponding to the operation steps;

[0066] The processing module is used to calculate the product similarity between the skin care product characteristics and the recommended skin care product characteristics:

[0067] ;

[0068] Wherein, Similarity(p,q) is the product similarity, GCN(p,q) represents the feature similarity calculation based on graph convolutional network, p is the feature of the skin care product, and q is the feature of the recommended skin care product.

[0069] Optionally, the processing module is configured to calculate the operation step score based on the fusion data, the operation step score, and the product similarity:

[0070] ;

[0071] Among them, Score(u) is the operation step score, S(t) is the operation step score, Similarity(p,q) is the product similarity, Fusion(Q,K,V) is the fusion data, α is the weight parameter corresponding to the operation step score, β is the weight parameter corresponding to the product similarity, γ is the weight parameter corresponding to the fusion data, λ is the regularization coefficient, ||θ|| 2 Represents the L2 norm of the model parameters;

[0072] The output module is configured to generate prompt information based on the operation step if it is determined that the operation step score is less than a preset threshold.

[0073] Optionally, the acquisition module is used to retrieve multiple preset prompt information from a preset database;

[0074] The processing module is used to determine the preset prompt information corresponding to the operation steps of the multiple preset prompt information, and display the preset prompt information corresponding to the operation steps on the AR glasses.

[0075] Optionally, the processing module is configured to determine a first acquisition time of the first feature and a second acquisition time of the second feature;

[0076] The processing module is configured to align the first feature and the second feature according to the first acquisition time and the second acquisition time to obtain a feature group;

[0077] The processing module is used to determine the historical skin care behavior data corresponding to the feature group, and fuse the feature group with the historical skin care behavior data to obtain the fused data.

[0078] Optionally, the processing module is configured to extract appearance features from the facial image to obtain the first feature;

[0079] The processing module is used to convert the sensor data into a data matrix to obtain the second feature, wherein the sensor data includes skin temperature data and humidity data.

[0080] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs any of the methods described above.

[0081] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions. When the instructions are executed, any one of the methods described above is executed.

[0082] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0083] Real-time facial imaging and sensor data are collected and combined with historical skincare behavior data for comprehensive analysis to generate personalized skincare recommendations. Leveraging the instant feedback and visual prompts provided by AR glasses, users can receive precise skincare guidance without having to schedule an appointment with a skincare professional, significantly improving the convenience and timeliness of obtaining skincare recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 This is a flow chart of a skin care auxiliary method based on AR glasses disclosed in an embodiment of the present application;

[0085] Figure 2 This is a module diagram of a skin care auxiliary device based on AR glasses disclosed in an embodiment of the present application;

[0086] Figure 3 This is a structural diagram of an electronic device disclosed in an embodiment of the present application.

[0087] Explanation of the reference numerals: 201, acquisition module; 202, extraction module; 203, processing module; 204, output module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0088] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0089] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0090] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0091] As living standards improve, skincare has become an essential part of daily life. However, traditional methods of seeking professional skincare advice are inconvenient. Users need to spend considerable time and effort to schedule appointments and visit skincare professionals, a process made even more difficult by the fast pace of life and the lack of professional resources in remote areas. Therefore, a more convenient way to easily access skincare advice is urgently needed.

[0092] This embodiment discloses a skin care auxiliary method based on AR glasses, referring to Figure 1 , including the following steps S110-S140:

[0093] S110: Acquire facial images and sensor data of a target to be detected.

[0094] The embodiments of this application disclose an augmented reality (AR) glasses-based skin care assistance method, which is applied to AR glasses. AR glasses are equipped with a high-resolution camera for capturing facial images and multiple sensors (such as temperature, humidity, and light sensors) for acquiring facial sensor data. A module is developed in advance to capture facial images using the AR glasses' camera, invoke a camera interface to acquire facial images in real time, and then invoke the camera interface to acquire facial images in real time. Simultaneously, another module is developed to collect sensor data on the AR glasses, invoke a sensor interface to acquire data such as temperature, humidity, and light intensity, and then process and correct the sensor data in real time to ensure data accuracy.

[0095] S120, extracting a first feature based on the facial image, and extracting a second feature based on the sensor data.

[0096] The AR glasses' high-definition camera captures facial images of the user (the target to be detected) in real time, ensuring high image quality and clear details. Median or Gaussian filtering can also be applied to remove noise and improve image quality. Histogram equalization or CLAHE (contrast-limited adaptive histogram equalization) can be used to enhance image contrast. A facial landmark detection algorithm (such as the 68-point model in the Dlib library or the Facemark model in OpenCV) is used to locate facial landmarks such as the eyes, nose, and mouth.

[0097] Then, skin color feature extraction is performed, and the image is converted from RGB color space to HSV or Lab color space to better separate color information. The color histogram of the facial skin area is calculated, the color distribution features are extracted, and K-means clustering is used to classify the skin color and identify the main color types.

[0098] Skin texture extraction is also performed, calculating the gray-level co-occurrence matrix and extracting texture features such as contrast, energy, entropy, and uniformity. Local texture features are extracted using the LBP method, generating a texture histogram. Gabor filters are applied to extract texture features at different orientations and scales.

[0099] For the acquired skin surface temperature and humidity data, a moving average filter or Kalman filter is used to remove noise from the sensor data, ensuring data stability and accuracy. The sensor data is also normalized so that it can be analyzed on a unified scale. Z-score or min-max normalization can be used. Temperature and humidity sensor data are collected simultaneously with facial image acquisition. Each sampling moment is timestamped to facilitate subsequent data alignment. Based on the timestamps, the temperature and humidity sensor data are aligned with the corresponding facial image keyframes to ensure data consistency and synchronization. Finally, the collected temperature data is converted into a matrix format, with each data point corresponding to the temperature value of a facial region. The collected humidity data is also converted into a matrix format, with each data point corresponding to the humidity value of a facial region.

[0100] S130 , based on the acquired historical skin care behavior data of the target to be detected, the historical skin care behavior data is fused with the first feature and the second feature to obtain fused data.

[0101] In a possible implementation, based on the acquired historical skin care behavior data of the target to be detected, the historical skin care behavior data is fused with the first feature and the second feature to obtain fused data, specifically including: determining the first acquisition time of the first feature and the second acquisition time of the second feature; aligning the first feature and the second feature according to the first acquisition time and the second acquisition time to obtain a feature group; determining the historical skin care behavior data corresponding to the feature group, and fusing the feature group with the historical skin care behavior data to obtain fused data.

[0102] Each time a facial image is captured, the capture time is recorded and marked as the first capture time. Each time sensor data is captured, the capture time is recorded and marked as the second capture time. Based on the first and second capture times, the first and second features are aligned. Specifically, for each facial image (first feature), the sensor data with the closest timestamp (second feature) is found. If there is a discrepancy between the first and second capture times, interpolation or nearest neighbor matching is used to align the first and second features. The aligned first and second features form a feature group, each of which contains a facial image feature and the corresponding sensor data feature.

[0103] The user's historical skincare behavior data is obtained from the user database. This data may include skincare products used, skincare steps, skincare frequency, and skin changes. The time of each skincare behavior is recorded, referred to as the skincare behavior time. Based on the collection time of the feature group, the corresponding historical skincare behavior data is found. Assuming that the collection time of each feature group is close to the skincare behavior time, the skincare behavior data corresponding to that feature group is considered relevant. A time window approach can be used, for example, skincare behavior data within a period of time before and after the feature group collection time is considered relevant data. Each feature group is fused with its corresponding historical skincare behavior data to form fused data. The feature group (including facial image features and sensor data features) and the historical skincare behavior data can be concatenated to form a high-dimensional feature vector. For example, different types of features can be fused using feature concatenation or feature weighting.

[0104] S140: Generate corresponding skin care recommendations based on the analysis of the fused data.

[0105] Before generating corresponding skin care recommendations based on the analysis of the fused data, the method further includes:

[0106] ;

[0107] Among them: Fusion (Q, K, V) is the fusion data, W O is the output weight matrix, head i is the i-th data, i∈(1, 2,…, n);

[0108] ;

[0109] ;

[0110] Among them, Q is the first feature, K is the second feature, V is the historical skin care behavior data, d k is the dimension of the key vector, W i Q is the weight matrix corresponding to the first feature, W i K is the weight matrix corresponding to the second feature, W i V is the weight matrix of historical skin care behavior data.

[0111] Specifically, the above formula describes a Transformer-based multi-head attention mechanism for analyzing fused data to generate skincare recommendations. The fused data is calculated by concatenating the outputs of multiple attention heads and applying a linear transformation. Concat is used to concatenate the outputs of multiple attention heads, and the output weight matrix is ​​used to linearly transform the concatenated multi-head attention output.

[0112] Attention(Q,K,V) is the result calculated by the attention mechanism, W i Q 、W i K and W i V are the weight matrices corresponding to the first feature, the second feature, and the historical skin care behavior data, respectively, and are used to map these inputs to query, key, and value. In the attention mechanism formula, d k is the dimension of the key vector, used to scale the dot product to prevent it from being too large. Softmax is a normalization function used to calculate the attention weights so that they sum to 1.

[0113] Multiply the attention weight and the value vector to get the weighted sum, which is the output of the attention head. Repeat the above calculation for each attention head to get the output of multiple attention heads. Connect the outputs of multiple attention heads and output the weight matrix W. O Perform linear transformation to obtain the final output of the fused data.

[0114] In the Transformer model's multi-head attention mechanism, the first feature, second feature, and historical skincare behavior data correspond to three different weight matrices, which are used to map these inputs to query (Q), key (K), and value (V) vectors, respectively. These weight matrices are learned through gradient descent during training so that the model can capture the relationship between the input data. The first feature is the facial image feature. The corresponding weight matrix is ​​used to map the facial image feature to the query (Q) vector. W i Q The weight matrix used to map the first feature (facial image feature) to the query vector.

[0115] Assume that the dimension of the first feature is d model , then W i Q The dimension is d model ×d k , where dk is the dimension of the query vector, then Q = first feature × W i Q Similarly, the second feature is the sensor data feature. The corresponding weight matrix is ​​used to map the sensor data feature to the key (K) vector, W i K The weight matrix used to map the second feature (sensor data feature) to the key vector, assuming the dimension of the second feature is d model , then W i K The dimension is d model ×d k , then K = second feature × W i K . Similarly W i V The weight matrix used to map historical skin care behavior data to a value vector, assuming that the dimension of the historical skin care behavior data is d model , then W i V The dimension is d model ×d V , where d V is the dimension of the value vector, then V = historical skin care behavior data × W i V .

[0116] In the multi-head attention mechanism, each attention head has its own set of weight matrices W i Q 、W i K and W i V , these matrices are learned and optimized by gradient descent during training. Assuming there are h attention heads, the overall weight matrix is:

[0117] Query weight matrix: W Q =[W1 Q , W2 Q ,…,W h Q ];

[0118] Bond weight matrix: W K =[W1 K , W2 K ,…,W h K ];

[0119] Value weight matrix: W V =[W1 V , W2 V ,…,W h V ].

[0120] Furthermore, before generating corresponding skin care recommendations based on the analysis of the fused data, the method further includes: acquiring skin care operation images and skin care product images of the target to be detected; identifying the operation steps of the target to be detected based on the skin care operation step images; and calculating the operation step scores of the operation steps based on the fused data. The specific calculation formula is as follows:

[0121] ;

[0122] Among them, S(t) is the operation step score, σ is the activation function, and W s is the weight matrix of step classification, LSTM is the long short-term memory network, and Fusion (Q, K, V) is the fusion data;

[0123] Specifically, use AR glasses or other cameras to capture real-time images of users performing skincare procedures, ensuring clear images and accurate identification of user steps. Use cameras to capture images of skincare products used by users, ensuring clear images and identification of product packaging, brand, and type. Use action recognition algorithms (such as convolutional neural networks and long short-term memory networks, CNN+LSTM) to identify specific skincare steps, such as cleansing, applying serum, massaging, and applying a mask.

[0124] Facial image features (first features), sensor data features (second features), and historical skincare behavior data (fused data) are input into a multi-head attention mechanism to calculate Fusion(Q, K, V). A long short-term memory network (LSTM) is used to process the output of Fusion(Q, K, V) to capture the temporal characteristics of the operation steps.

[0125] The purpose of calculating the step score is to evaluate the effectiveness of each step based on the user's skincare steps and fused data. The step score represents the effectiveness rating of a skincare step at a specific moment. In the calculation formula, the activation function introduces nonlinear transformations. Common activation functions include ReLU and sigmoid. The step classification weight matrix is ​​a parameter that requires training and learning. The Long Short-Term Memory (LSTM) network processes sequence data and captures temporal features. The fused data, obtained through a multi-head attention mechanism, contains information from the primary feature (facial image features), the secondary feature (sensor data features), and historical skincare behavior data. The step classification bias is a parameter that requires training and learning. The LSTM processes the input sequence data (Fusion(Q,K,V)) and outputs a temporal feature representation of the sequence data. This representation captures dependencies within the sequence and helps evaluate the effectiveness of each step.

[0126] The Transformer decoder generates skin condition predictions, including skin health index, moisture content, oiliness, pore status, and more. Based on these skin condition results, the skin health score is calculated. The Transformer and classifier identify specific skin concerns (such as acne, dark spots, wrinkles, and dryness) and generate personalized skin care recommendations. Users can also provide feedback through the user interface, including feedback on usage results, allergic reactions, and satisfaction. Based on this feedback, the model is fine-tuned and the rules are optimized to improve recommendation accuracy and user satisfaction. Recommendation strategies are personalized based on individual user skin characteristics and usage habits, ensuring each user receives the most appropriate skin care regimen. The system dynamically adjusts recommendation rules and model parameters based on long-term user usage data and feedback. Online learning allows the model to be updated to adapt to changes in the user's skin condition. Furthermore, cameras located at different angles within the AR glasses monitor the use and steps of skin care products. Incorrect steps or unrecommended product types are displayed in the AR glasses. The process primarily involves capturing real-time images of the user's face and hands, detecting head movement and posture using sensors such as a gyroscope, processing time series data using an LSTM time series model, analyzing the action sequence, and then inputting this data into a Transformer, combining visual and sensor data for more complex action and step recognition. The user's current actions are then compared against a predefined sequence of skincare steps (such as cleansing, toning, and lotion) to detect correct execution. If errors are detected, natural language prompts are generated, and the AR glasses' display provides real-time notifications, such as text or icons displayed in the field of view, to help the user correct their steps or proceed to the next step.

[0127] Furthermore, skin care product features are extracted based on the skin care product images; based on the operation steps, recommended skin care product features are determined, where the recommended skin care product is the skin care product corresponding to the operation steps; and product similarity between the skin care product features and the recommended skin care product features is calculated:

[0128] ;

[0129] Among them, Similarity(p,q) is the product similarity, GCN(p,q) represents the feature similarity calculation based on graph convolutional network, p is the skin care product feature, and q is the recommended skin care product feature.

[0130] Specifically, the corresponding skincare product type is determined based on the identified action steps. For example, the cleansing step corresponds to a cleanser, and the serum application step corresponds to a serum. Based on the skincare product database, features of the recommended skincare products are extracted, including brand, ingredients, and applicable skin types. Product similarity is then calculated using a graph convolutional network-based similarity calculation. A skincare product feature graph is constructed, with each skincare product feature represented as a node in the graph and similarities between features represented as edges. A graph convolutional network (GCN) is trained on the skincare product feature graph to learn feature representations. The similarity between the skincare product features and the recommended skincare product features is calculated to obtain product similarity. GCNs are designed to process graph-structured data and, through graph convolution operations, learn the relationships between nodes (features). In similarity calculation, GCNs can capture the complex relationships between skincare product features, providing more accurate similarity assessments. A comprehensive analysis is performed by combining the action step score and product similarity to assess the effectiveness of the user's current skincare action. If the action step score is low or the product similarity is low, recommendations are given to the user to improve their skincare steps or use more suitable skincare products.

[0131] Based on the analysis of the fused data, corresponding skin care recommendations are generated, including:

[0132] Calculate the operation step score based on the fusion data, operation step score, and product similarity:

[0133] ;

[0134] Among them, Score(u) is the operation step score, S(t) is the operation step score, Similarity(p,q) is the product similarity, Fusion(Q,K,V) is the fusion data, α is the weight parameter corresponding to the operation step score, β is the weight parameter corresponding to the product similarity, γ is the weight parameter corresponding to the fusion data, λ is the regularization coefficient, ||θ|| 2 Represents the L2 norm of the model parameters; if it is determined that the operation step score is less than the preset threshold, a prompt message is generated based on the operation step.

[0135] The operation step score comprehensively evaluates the effectiveness of the operation step, including the product usage and operation step correctness scores. The weight parameter adjusts the importance of the operation step score, product similarity, and fused data in the final score. Regularization prevents model overfitting by incorporating the L2 norm of model parameters into the loss function to control model complexity. The regularization coefficient determines the strength of the regularization term. The L2 norm of model parameters is the sum of the squares of all model parameters.

[0136] By comprehensively calculating the step score S(t), product similarity (Similarity(p,q), and fusion data (Fusion(Q,K,V)), and introducing a regularization term to control model complexity, the formula ultimately generates the step score Score(u). If the score falls below a preset threshold, it indicates a problem with the current skincare procedure and requires improvement suggestions. This method combines technologies such as multi-head attention, LSTM, and GCN to ensure accurate and personalized evaluation, providing users with scientific and effective skincare guidance.

[0137] Furthermore, a threshold value of the operation step score is set to determine whether the effect of the operation step needs to be improved. For example, assuming the threshold value is 0.6, if the calculated operation step score Score(u) is lower than the preset threshold value, the generation of a prompt message is triggered.

[0138] In order to generate skin care recommendations based on the operation step scores, it is necessary to retrieve multiple preset prompts from the preset database, determine the preset prompts that are suitable for the user's current operation steps, and then display the specific prompts on the AR glasses. Establish a database containing preset prompts, each prompt associated with a specific skin care operation step. Operation step identifier (for example, cleansing, applying essence, massage, applying a mask, etc.). Preset prompt information (for example, the correct cleansing method, essence usage skills, massage techniques, etc.). Based on the calculated operation step scores, determine the operation steps that need to be improved. Use the operation step identifier to retrieve the corresponding preset prompt information from the database. Based on the preset prompt information corresponding to the determined operation step, design the display logic and display the prompt information to the user on the AR glasses.

[0139] By adopting the technical solution of this application, users' facial images and sensor data are acquired in real time, combined with historical skincare behavior data for comprehensive analysis, to generate personalized skincare recommendations. With the instant feedback and visual prompts provided by AR glasses, users can receive precise skincare guidance without having to schedule an appointment with a professional skincare professional, greatly improving the convenience and timeliness of obtaining skincare recommendations.

[0140] This embodiment also discloses a skin care auxiliary device based on AR glasses, referring to Figure 2The device includes an acquisition module 201, an extraction module 202, a processing module 203 and an output module 204, wherein:

[0141] The acquisition module 201 is used to acquire facial images and sensor data of a target to be detected.

[0142] The extraction module 202 is configured to extract a first feature based on the facial image and a second feature based on the sensor data.

[0143] The processing module 203 is configured to fuse the historical skin care behavior data of the target to be detected with the first feature and the second feature to obtain fused data.

[0144] The output module 204 is used to generate corresponding skin care recommendations based on the analysis of the fused data.

[0145] In a possible implementation, the processing module 203 is configured to, before generating corresponding skin care recommendations based on the analysis of the fused data, further include:

[0146] ;

[0147] Among them: Fusion (Q, K, V) is the fusion data, W O is the output weight matrix, head i is the i-th data, i∈(1, 2,…, n).

[0148] ;

[0149] ;

[0150] Among them, Q is the first feature, K is the second feature, V is the historical skin care behavior data, d k is the dimension of the key vector, W i Q is the weight matrix corresponding to the first feature, W i K is the weight matrix corresponding to the second feature, W i V is the weight matrix of historical skin care behavior data.

[0151] In a possible implementation, the acquisition module 201 is configured to acquire skin care operation images and skin care product images of the target to be detected.

[0152] The extraction module 202 is used to identify the operation steps of the target to be detected based on the skin care operation step image.

[0153] The processing module 203 is used to calculate the operation step score of the operation step based on the fusion data. The specific calculation formula is as follows:

[0154] ;

[0155] Among them, S(t) is the operation step score, σ is the activation function, and W s is the weight matrix of step classification, LSTM is the long short-term memory network, and Fusion (Q, K, V) is the fusion data.

[0156] The extraction module 202 is configured to extract skin care product features based on the skin care product image.

[0157] The processing module 203 is configured to determine the recommended skin care product characteristics of the recommended skin care product according to the operation steps, where the recommended skin care product is the skin care product corresponding to the operation steps.

[0158] Processing module 203 is used to calculate the product similarity between the skin care product characteristics and the recommended skin care product characteristics:

[0159] ;

[0160] Among them, Similarity(p,q) is the product similarity, GCN(p,q) represents the feature similarity calculation based on graph convolutional network, p is the skin care product feature, and q is the recommended skin care product feature.

[0161] In a possible implementation, the processing module 203 is configured to calculate an operation step score based on the fused data, the operation step score, and the product similarity:

[0162] ;

[0163] Among them, Score(u) is the operation step score, S(t) is the operation step score, Similarity(p,q) is the product similarity, Fusion(Q,K,V) is the fusion data, α is the weight parameter corresponding to the operation step score, β is the weight parameter corresponding to the product similarity, γ is the weight parameter corresponding to the fusion data, λ is the regularization coefficient, ||θ|| 2 Represents the L2 norm of the model parameters.

[0164] The output module 204 is configured to generate prompt information based on the operation step if it is determined that the operation step score is less than a preset threshold.

[0165] In a possible implementation, the acquisition module 201 is configured to retrieve a plurality of preset prompt information from a preset database.

[0166] The processing module 203 is used to determine preset prompt information corresponding to multiple preset prompt information operation steps, and display the preset prompt information corresponding to the operation steps on the AR glasses.

[0167] In a possible implementation, the processing module 203 is configured to determine a first collection time of the first feature and a second collection time of the second feature.

[0168] The processing module 203 is configured to align the first feature and the second feature according to the first acquisition time and the second acquisition time to obtain a feature group.

[0169] The processing module 203 is used to determine the historical skin care behavior data corresponding to the feature group, and fuse the feature group with the historical skin care behavior data to obtain fused data.

[0170] In a possible implementation, the processing module 203 is configured to extract appearance features from the facial image to obtain a first feature.

[0171] The processing module 203 is configured to convert the sensor data into a data matrix to obtain a second feature, wherein the sensor data includes skin temperature data and humidity data.

[0172] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0173] This embodiment also discloses an electronic device, referring to Figure 3 The electronic device may include: at least one processor 301 , at least one communication bus 302 , a user interface 303 , a network interface 304 , and at least one memory 305 .

[0174] The communication bus 302 is used to implement the connection and communication between these components.

[0175] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0176] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0177] The processor 301 may include one or more processing cores. Using various interfaces and circuits, the processor 301 connects to various components within the server. It executes instructions, programs, code sets, or instruction sets stored in the memory 305, as well as accesses data stored in the memory 305, to perform various server functions and process data. Optionally, the processor 301 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 301 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display screen; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 301 but implemented as a separate chip.

[0178] Memory 305 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory may include non-transitory computer-readable storage medium. Memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. Memory 305 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch control, sound playback, image playback, etc.), and instructions for implementing the aforementioned method embodiments. The data storage area may store data related to the aforementioned method embodiments. Memory 305 may also optionally be at least one storage device located remotely from the processor 301. Memory 305, as a computer storage medium, may include an operating system, a network communication module, a user interface 303 module, and an application for a skin care assistance method based on AR glasses.

[0179] exist Figure 3In the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 301 can be used to call an application stored in the memory 305 for a skin care auxiliary method based on AR glasses. When executed by one or more processors 301, the electronic device executes one or more methods such as those in the above embodiments.

[0180] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0181] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0183] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0184] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 305 and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application. The aforementioned memory 305 includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0186] The present application also discloses a computer-readable storage medium storing instructions, which, when executed by one or more processors 301, enable an electronic device to execute one or more of the methods described in the above embodiments.

[0187] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variation, use or adaptive change of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A skin care auxiliary method based on AR glasses, characterized in that: The method comprises: Obtain facial images and sensor data for the target to be detected; extracting a first feature based on the facial image and extracting a second feature based on the sensor data; According to the acquired historical skin care behavior data of the target to be detected, the historical skin care behavior data is fused with the first feature and the second feature to obtain fused data; generating corresponding skin care recommendations based on analysis of the fused data; Before generating corresponding skin care recommendations based on the analysis of the fused data, the method further includes: Acquire skin care operation images and skin care product images of the target to be detected; Identifying the operation steps of the target to be detected based on the skin care operation image; Based on the fused data, the operation step score of the operation step is calculated. The specific calculation formula is as follows: ； Where S(t) is the score of the operation steps, σ is the activation function, and W s is the weight matrix of the step score, LSTM is the long short-term memory network, Fusion (Q, K, V) is the fusion data, b s is the bias term of the step score, Q is the first feature, K is the second feature, and V is the historical skin care behavior data; extracting skin care product features based on the skin care product image; Determining, based on the operating steps, recommended skin care product features of a recommended skin care product, the recommended skin care product being the skin care product corresponding to the operating steps; Calculate the product similarity between the skin care product features and the recommended skin care product features: ； Where Similarity(p,q) is the product similarity, GCN(p,q) represents feature similarity calculation based on graph convolutional network, p is the feature of the skin care product, and q is the feature of the recommended skin care product; The generating of corresponding skin care suggestions based on the analysis of the fused data specifically includes: Calculate the operation step score based on the fusion data, the operation step score, and the product similarity: ； Among them, Score(u) is the operation step score, S(t) is the operation step score, Similarity(p,q) is the product similarity, Fusion(Q,K,V) is the fusion data, α is the weight parameter corresponding to the operation step score, β is the weight parameter corresponding to the product similarity, γ is the weight parameter corresponding to the fusion data, λ is the regularization coefficient, ||θ|| 2 Represents the L2 norm of the model parameters; If it is determined that the score of the operation step is less than a preset threshold, generating prompt information based on the operation step; If it is determined that the score of the operation step is less than a preset threshold, generating prompt information based on the operation step specifically includes: Retrieve multiple preset prompt information from the preset database; Determine the preset prompt information corresponding to the operation steps of the multiple preset prompt information, and display the preset prompt information corresponding to the operation steps on the AR glasses.

2. The skin care auxiliary method based on AR glasses according to claim 1, characterized in that: Before generating corresponding skin care recommendations based on the analysis of the fused data, the method further includes: ； Where: Fusion(Q,K,V) is the fusion data, W O is the output weight matrix, head i is the i-th data, i∈(1, 2,…, n); ； ； Among them, Q is the first feature, K is the second feature, V is the historical skin care behavior data, d k is the dimension of the key vector, W i Q is the weight matrix corresponding to the first feature, W i K is the weight matrix corresponding to the second feature, W i V is the weight matrix of the historical skin care behavior data.

3. The skin care auxiliary method based on AR glasses according to claim 1, characterized in that: The step of fusing the historical skin care behavior data of the target to be detected with the first feature and the second feature to obtain fused data specifically includes: determining a first acquisition time of the first feature and a second acquisition time of the second feature; Aligning the first feature and the second feature according to the first acquisition time and the second acquisition time to obtain a feature group; The historical skin care behavior data corresponding to the feature group is determined, and the feature group is fused with the historical skin care behavior data to obtain the fused data.

4. The skin care auxiliary method based on AR glasses according to claim 1, characterized in that: The extracting of the first feature based on the facial image and the extracting of the second feature based on the sensor data specifically further includes: Extracting appearance features from the facial image to obtain the first features; The sensor data is converted into a data matrix to obtain the second feature, wherein the sensor data includes skin temperature data and humidity data.

5. A skin care auxiliary device based on AR glasses, characterized in that: The device is used to execute the method according to any one of claims 1 to 4, and comprises an acquisition module (201), an extraction module (202), a processing module (203), and an output module (204), wherein: The acquisition module (201) is used to acquire facial images and sensor data of a target to be detected; The extraction module (202) is configured to extract a first feature based on the facial image and a second feature based on the sensor data; The processing module (203) is used to fuse the historical skin care behavior data of the target to be detected with the first feature and the second feature to obtain fused data; The output module (204) is used to generate corresponding skin care suggestions based on the analysis of the fused data.

6. An electronic device, characterized in that: The electronic device comprises a processor (301), a memory (305), a user interface (303), a network interface (304) and a memory (305), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are both used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 4 is executed.

Images