Skin care product recommendation method and device

Abstract

The invention provides a skin care product recommendation method and device. A target user is guided to make a face posture according to a posture guide instruction; collecting a posture image of each face posture and a close-up image of a specified posture key area so as to determine a key area image; generating a skin analysis result of the key area image through three sub-models in the skin analysis model to determine a target skin care product; and questions and answers can be interacted with the target user. According to the mode, key area images are determined based on an original image containing a posture image and a close-up image, the key area images are input into a skin analysis model, the skin color, the skin type and the skin problem of a target user are subjected to collaborative analysis through a first sub-model, a second sub-model and a third sub-model, and a skin analysis result of each key area image is generated. The method can recommend a proper target skin care product for the user, can also solve the questions of the target user, and can provide a one-stop closed-loop service of skin detection-skin analysis-skin care product recommendation in various life scenes.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and apparatus for recommending skincare products. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, skin condition assessment through image analysis has become an important direction in the beauty and skincare industry. Currently, existing technological solutions for achieving similar functions mainly fall into two categories: one is based on smartphone skin detection applications. Users take photos of their faces using their phone's front or rear camera, and the application uses cloud-based or locally deployed algorithm models to analyze the photos, assessing the user's skin tone, skin texture, acne, wrinkles, dark circles, and other conditions, providing online suggestions. The other solution is based on dedicated desktop skin analyzers. Professional beauty salons or dermatology clinics typically use large, fixed desktop skin analyzers, which provide more accurate results. However, these technologies lack a comprehensive, one-stop "skin detection - skin analysis - skincare product recommendation" closed-loop service solution for various life scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a skincare product recommendation method and device, so as to provide a one-stop closed-loop service solution of "skin detection - skin analysis - skincare product recommendation" in various life scenarios.

[0004] This invention provides a skincare product recommendation method applied to a robot. The method includes: receiving skin analysis instructions from a target user; outputting posture guidance instructions based on the skin analysis instructions to guide the target user to perform various facial postures; during the target user's performance of various facial postures, acquiring posture images corresponding to each facial posture, as well as close-up images of key areas corresponding to specified postures; determining key area images for skin analysis corresponding to the target user based on the original images; wherein the original images include: posture images corresponding to each facial posture and close-up images corresponding to each key area; for each key area image, inputting the key area image into a pre-trained skin analysis model, outputting skin color determination results through a first sub-model in the skin analysis model; outputting skin texture classification results through a second sub-model, and outputting skin problem detection results through a third sub-model; generating skin analysis results corresponding to the key area image based on the skin color determination results, skin texture classification results, and skin problem detection results; recommending target skincare products to the target user based on the skin analysis results of each key area image; if a question about the target skincare product is received from the target user, extracting the target answer corresponding to the question from a preset information knowledge base and feeding it back to the target user.

[0005] Furthermore, the steps of receiving skin analysis instructions from the target user and outputting posture guidance instructions to guide the target user to perform various facial postures include: receiving skin analysis instructions from the target user and capturing a first image containing the target user through a preset camera; calculating the distance between the target user and the robot, and the angle of the target user relative to the robot, based on the first image; determining whether the target user is within the robot's preset detection range based on the distance and angle; if it is confirmed that the target user is not within the preset detection range, outputting a movement instruction to instruct the target user to move to the preset detection range according to the movement instruction; if it is confirmed that the target user is within the preset detection range, outputting posture guidance instructions to guide the target user to perform various facial postures according to the posture guidance instructions.

[0006] Furthermore, the steps of acquiring posture images corresponding to each facial posture and close-up images of key areas corresponding to specified postures during the process of the target user making multiple facial postures include: during the process of the target user making multiple facial postures, for each facial posture, if the facial posture meets the corresponding preset standard posture, acquiring the posture image corresponding to the facial posture; if the facial posture belongs to the preset specified posture, obtaining the key area corresponding to the facial posture, and performing zoom processing on the preset camera to acquire close-up images corresponding to the key area.

[0007] Furthermore, the original image also includes: a specified image; the specified image includes at least one of the following: a sebum image, a hemoglobin distribution map; the specified image is acquired by: switching the light source of the preset multispectral acquisition hardware module or enabling a camera with a specified spectrum of the multispectral acquisition hardware module to acquire the specified image.

[0008] Furthermore, the step of determining the key region image for skin analysis corresponding to the target user based on the original image includes: preprocessing each original image according to a preset preprocessing method to obtain the preprocessing result corresponding to each original image; and cropping the key region image for skin analysis corresponding to the target user based on the preprocessing result corresponding to each original image.

[0009] Furthermore, the steps for recommending target skincare products to target users based on the skin analysis results of each key region image include: obtaining the target user's historical profile data; fusing the skin analysis results corresponding to each key region image with the historical profile data to obtain a fusion result; determining the target user's core skincare needs based on the fusion result; and matching target skincare products that match the core skincare needs from a preset skincare product knowledge base based on the core skincare needs.

[0010] Furthermore, the skin analysis results corresponding to each key area image include scores for multiple preset skin detection dimensions; based on the fusion results, the steps to determine the core skin care needs of the target user include: determining the core skin care needs of the target user based on each score in the fusion results and historical skin problem records in the historical archive data.

[0011] Furthermore, the method also includes: generating a skincare routine plan based on the target skincare product; displaying the skincare routine plan on a screen, and / or playing the skincare routine plan via voice; displaying the results on a screen, and / or playing the results via voice; wherein the results include: skin analysis results, as well as images, names, core ingredients, and reasons for recommendation of the target skincare product.

[0012] Furthermore, the key region images include at least one of the following: a forehead image, a left cheek image, a right cheek image, a nose wing image, and a chin image.

[0013] This invention provides a skincare product recommendation device, which is installed on a robot. The device includes: a guidance module for receiving skin analysis instructions from a target user and outputting posture guidance instructions to guide the target user to perform various facial postures; an acquisition module for acquiring posture images corresponding to each facial posture and close-up images of key areas corresponding to specified postures during the process of the target user performing various facial postures; a determination module for determining key area images for skin analysis corresponding to the target user based on the original images; wherein the original images include: posture images corresponding to each facial posture and close-up images corresponding to each key area; and an analysis module for analyzing each image. The key region image is input into a pre-trained skin analysis model. The first sub-model outputs skin color determination; the second sub-model outputs skin texture classification; and the third sub-model outputs skin problem detection. Based on the skin color determination, skin texture classification, and skin problem detection results, a skin analysis result corresponding to the key region image is generated. A recommendation module recommends target skincare products to the target user based on the skin analysis results of each key region image. An extraction module, if receiving a question from the target user regarding a target skincare product, extracts the corresponding target answer from a pre-set knowledge base and returns it to the target user.

[0014] The skincare product recommendation method and apparatus provided by this invention receive skin analysis instructions from a target user, output posture guidance instructions based on the skin analysis instructions, and guide the target user to perform various facial postures according to the posture guidance instructions; during the process of the target user performing various facial postures, posture images corresponding to each facial posture and close-up images of key areas corresponding to specified postures are acquired; based on the original images, key area images for skin analysis corresponding to the target user are determined; wherein, the original images include: posture images corresponding to each facial posture and close-up images corresponding to each key area; for each key area image, the key area image is input into a pre-trained skin analysis model, and the skin analysis model outputs skin color determination results through the first sub-model; outputs skin texture classification results through the second sub-model; and outputs skin problem detection results through the third sub-model; based on the skin color determination results, skin texture classification results, and skin problem detection results, the skin analysis results corresponding to the key area image are generated; based on the skin analysis results of each key area image, target skincare products are recommended to the target user; if a question about the target skincare product is received from the target user, the target answer corresponding to the question is extracted from a preset information knowledge base and fed back to the target user. This method identifies key region images based on raw images containing pose and close-up images, and inputs them into a skin analysis model. Through the first, second, and third sub-models, it performs collaborative analysis on the target user's skin tone, skin texture, and skin problems, generating skin analysis results for each key region image. This allows the model to recommend suitable target skincare products to the user and answer their questions, providing a one-stop closed-loop service of "skin detection - skin analysis - skincare product recommendation" in various life scenarios. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart of a skincare product recommendation method provided in an embodiment of the present invention; Figure 2 A schematic diagram showing the distribution of sensing devices for a humanoid robot according to an embodiment of the present invention; Figure 3 A flowchart illustrating the implementation of a YOLOv8 face landmark detection algorithm provided in this embodiment of the invention; Figure 4A flowchart illustrating another skincare product recommendation method provided in this embodiment of the invention; Figure 5 This is a schematic diagram of a skincare product recommendation device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] With the rapid development of artificial intelligence and computer vision technologies, skin condition assessment through image analysis has become an important direction in the beauty and skincare industry. Currently, existing technological solutions for achieving similar functions can be mainly divided into the following categories: 1. Smartphone-based skin detection application (APP): This is currently the most common solution. Users take photos of their faces using their phone's front or rear camera. The app uses algorithms deployed in the cloud or locally to analyze the images, assessing the user's skin tone, skin texture, acne, wrinkles, dark circles, and other conditions, and recommends skincare products accordingly.

[0019] This method has the following main drawbacks: (1) User experience is cumbersome and lacks standardization: Users need to hold their mobile phones and take pictures themselves. The shooting distance, angle and lighting environment are entirely controlled by the user, making it difficult to ensure the consistency of the detection conditions each time. The intensity of light and slight changes in color temperature can seriously affect the judgment of skin color, resulting in large fluctuations in the detection results and insufficient reliability.

[0020] (2) Lack of professional-grade interactive guidance: The mobile app cannot provide proactive and personalized interaction like a professional beauty consultant. For example, it cannot guide users to turn their heads to a specific angle to detect the skin on the side profile, nor can it provide voice reminders when it detects that the user's eyes are closed. The entire process is passive and unguided, affecting the integrity of the detection area and the quality of the data.

[0021] (3) Limited service scenarios, unable to form a service loop: This solution is limited to providing online advice. When users are in cosmetics stores, beauty salons, or large shopping malls, the mobile app cannot seamlessly connect with offline physical products and services. It is just a virtual tool and cannot complete the full service chain from "testing" to "experience" to "purchase".

[0022] 2. Equipment based on a dedicated desktop skin analyzer: In professional beauty salons or dermatology clinics, large, stationary benchtop skin analyzers (such as those based on Wood's lamp, polarized light, or high-resolution cameras) are typically used. These devices provide more accurate test results.

[0023] This method has the following main drawbacks: (1) Fixed equipment, high cost and lack of accessibility: These devices are usually bulky and expensive, and require users to fix their chin and forehead to a specific bracket, resulting in a cold, mechanical experience that lacks accessibility. They are limited to specific professional venues and cannot be popularized in a wider range of consumer scenarios (such as homes, shopping mall atriums, etc.).

[0024] (2) Single function and lack of comprehensive service capabilities: The core function of this type of equipment is "detection", and its original design purpose is not for "recommendation" and "interaction". It does not have the humanoid interaction, movement and knowledge question answering capabilities of humanoid robots, and cannot conduct in-depth communication with users after detection, explain the detection results, and vividly recommend related products.

[0025] In summary, existing technologies share a common bottleneck: a disconnect exists between the detection tools and the user. They either rely on user operation, introducing uncertainty, or sacrifice accessibility and interactive experience due to the fixed and specialized nature of the equipment. None of them provide a closed-loop service solution that proactively approaches the user, provides personalized interactive guidance, and offers a one-stop "skin detection-skin analysis-skincare product recommendation" solution across various life scenarios. Based on this, this invention provides a skincare product recommendation method and apparatus, which can be applied to scenarios requiring automated skin detection and skincare product recommendation.

[0026] To facilitate understanding of this embodiment, a skincare product recommendation method disclosed in this embodiment will first be introduced, such as... Figure 1 As shown, the method is applied to robots; in practical applications, the robot may specifically be a humanoid robot, etc.; the method includes the following steps: Step S102: Receive skin analysis instructions from the target user, and output posture guidance instructions according to the skin analysis instructions to guide the target user to make various facial postures according to the posture guidance instructions; In practice, when a target user confirms the need for skin analysis and requests the robot to recommend suitable skincare products, they can send a skin analysis command to the robot. Upon receiving the command, the robot can output posture guidance instructions. The target user can then follow these instructions to perform various facial postures. For example, the robot's display shows a frontal face icon with a voice prompt: "Please face the screen directly." Next, it displays a face icon turning left, prompting: "Please slowly turn your head to your left." Finally, it displays an upward-looking face icon, prompting: "Please gently raise your head so I can see your jawline," and so on.

[0027] Step S104: During the process of the target user making multiple facial poses, capture the pose image corresponding to each facial pose, as well as the close-up image of the key area corresponding to the specified pose. The specified pose can be any one or more of various facial poses, which can be preset according to actual needs; the key area can be understood as a high-incidence area for skin problems, such as the T-zone (including the forehead and nose); in actual implementation, when the target user makes various facial poses according to the pose guidance instructions, the robot can collect one or more pose images corresponding to each facial pose. For a specified pose, such as a frontal face, a close-up image of the key area can be collected based on the preset key area of ​​the specified pose.

[0028] Step S106: Based on the original image, determine the key region image for skin analysis corresponding to the target user; wherein, the original image includes: pose image corresponding to each facial pose and close-up image corresponding to each key region; The aforementioned key region images typically include at least one of the following: forehead image, left cheek image, right cheek image, nose wing image, and chin image. In practice, after acquiring one or more pose images corresponding to each of the above facial poses, as well as close-up images of each key region, the key region images used for skin analysis can be further determined based on these acquired images.

[0029] Step S108: For each key region image, the key region image is input into a pre-trained skin analysis model. The first sub-model of the skin analysis model outputs the skin color determination result; the second sub-model outputs the skin texture classification result; and the third sub-model outputs the skin problem detection result. Based on the skin color determination result, skin texture classification result, and skin problem detection result, the skin analysis result corresponding to the key region image is generated. The first, second, and third sub-models mentioned above can all be implemented using appropriate model structures based on actual needs. For example, the first sub-model can be implemented using a model trained on the LAB color space, the second sub-model can be implemented using a texture analysis model based on CNN (Convolutional Neural Network), and the third sub-model can be implemented using an object detection model based on YOLOv8 and a segmentation model based on U-Net++. Table 1 shows a schematic table of component information, which displays the information table of the third sub-model. This table shows the component version, training dataset, and data source used by the third sub-model. Among them, the COCO (Common Objects in Context) dataset is a large and rich dataset for image recognition, object detection, segmentation, and image captioning; and the ISIC (International Skin Imaging Collaboration) dataset is a medical dataset focusing on dermoscopy images.

[0030] Table 1

[0031] In this embodiment, the skin analysis model is not a single model, but a collaborative analysis system composed of multiple sub-models, each performing different functions. Specifically, for each key region image, the key region image is input into a pre-trained skin analysis model. Each sub-model in the skin analysis model can process the key region image separately and output its corresponding result. Specifically, the first sub-model can analyze the key region image to determine the target user's skin tone and output a skin tone determination result for that target user, such as warm white, cool white, or healthy tan. The second sub-model can analyze the key region image to detect the target user's skin texture, identify the skin surface texture and gloss, and output... Based on the target user's skin type classification results, such as dry skin (rough texture, lack of luster), oily skin (large pores, reflective skin), combination skin (oily T-zone, dry U-zone), and sensitive skin (accompanied by capillary redness), the third sub-model can analyze and process the key area image to detect skin problems present in the target user and output skin problem detection results. This third sub-model can accurately identify and locate specific facial skin problems, including inflammatory acne (red papules with bounding boxes and severity ratings), blackheads / whiteheads (mainly located on the nose and chin), wrinkles (detecting fine and deep lines around the eyes and forehead through texture analysis), and pigmentation (identifying areas of hyperpigmentation caused by UV exposure). The skin analysis model can integrate the results from each sub-model to generate the skin analysis results corresponding to the key area image. This skin analysis result can be a structured skin analysis report, which can include quantitative indicators from multiple skin detection dimensions and a map locating the areas with skin problems.

[0032] Step S110: Based on the skin analysis results of each key region image, recommend target skincare products to the target user; After acquiring the skin analysis results of each key area image, the robot can recommend suitable skin care products to the target user based on these results.

[0033] Step S112: If a question about the target skincare product is received from the target user, the target answer corresponding to the question is extracted from the preset information knowledge base and fed back to the target user.

[0034] The aforementioned pre-set information knowledge base typically stores answers to multiple frequently asked questions. In actual implementation, after the robot recommends a target skincare product to the target user, the target user can ask the robot related questions. For example, the target user can ask questions via voice, such as "Can this serum be used on sensitive skin?" The robot, with its natural language processing capabilities, extracts the corresponding target answers from the pre-set information knowledge base and provides responses, achieving a truly interactive consultant experience.

[0035] The aforementioned skincare product recommendation method receives skin analysis instructions from a target user, outputs posture guidance instructions based on these instructions, and guides the target user to perform various facial postures. During the process of the target user performing these facial postures, posture images corresponding to each posture and close-up images of key areas corresponding to specific postures are captured. Based on the original images, key area images for skin analysis corresponding to the target user are determined. The original images include: posture images corresponding to each facial posture and close-up images corresponding to each key area. For each key area image, the key area image is input into a pre-trained skin analysis model. The first sub-model of the skin analysis model outputs a skin tone determination result; the second sub-model outputs a skin texture classification result; and the third sub-model outputs a skin problem detection result. Based on the skin tone determination result, skin texture classification result, and skin problem detection result, a skin analysis result corresponding to the key area image is generated. Based on the skin analysis result of each key area image, target skincare products are recommended to the target user. If a question about the target skincare product is received from the target user, the target answer corresponding to the question is extracted from a pre-set information knowledge base and fed back to the target user. This method identifies key region images based on raw images containing pose and close-up images, and inputs them into a skin analysis model. Through the first, second, and third sub-models, it performs collaborative analysis on the target user's skin tone, skin texture, and skin problems, generating skin analysis results for each key region image. This allows the model to recommend suitable target skincare products to the user and answer their questions, providing a one-stop closed-loop service of "skin detection - skin analysis - skincare product recommendation" in various life scenarios.

[0036] This invention also provides another method for recommending skincare products, which is implemented based on the method in the above embodiments. This embodiment uses a humanoid robot as an example for illustration. A humanoid robot is a comprehensive platform with mobility, multimodal perception, and anthropomorphic interaction capabilities, and can provide a new technical path to overcome existing technological bottlenecks; see also... Figure 2The diagram shows the distribution of sensing devices for a humanoid robot. These devices typically include: one LiDAR sensor, one fisheye camera, two RGB-D cameras (RGB representing red, green, and blue color channels for capturing color images, and D representing depth for acquiring scene depth information), one interactive screen (corresponding to the aforementioned display screen), one microphone, and a pair of dexterous hands. The method includes the following steps: Step 1: Receive skin analysis instructions from the target user and capture the first image containing the target user through a preset camera; After detecting a potential user ahead using its built-in sensing devices, the humanoid robot actively initiates interaction via its voice output module and display screen, for example, outputting the voice message: "Hello, I am your intelligent skincare consultant. Can I perform a skin analysis for you now?" Upon receiving a positive response from the user, it can be considered that the skin analysis command has been received, and the humanoid robot can then enter the active guidance phase. The core objective of this phase is to ensure the standardization of subsequent image acquisition. The humanoid robot utilizes its visual feedback, through voice and screen animation, to guide the user to adjust to the optimal detection position and posture. Specifically, the user's position needs to be confirmed first, and an RGBD camera can be used to acquire the first image; multiple first images can be acquired in real time. This RGBD camera not only provides color images but also depth information, making scene understanding more comprehensive and accurate.

[0037] Step 2: Calculate the distance between the target user and the robot, as well as the angle of the target user relative to the robot, based on the first image; The distance between the target user and the humanoid robot can be calculated in real time based on the first image mentioned above, and the angle between the target user and the humanoid robot can also be calculated in real time.

[0038] Step 3: Determine whether the target user is within the robot's preset detection range based on distance and angle; The aforementioned preset detection range can be understood as the optimal detection range for the humanoid robot. This preset detection range can be set according to actual needs. For example, the target user may be located directly in front of the humanoid robot, and the distance between the target user and the humanoid robot may be between 1 meter and 1.5 meters. When setting the optimal detection range, it is important to ensure that the distance between the target user and the humanoid robot is not too close, because if the distance is too close, the humanoid robot may touch the person when it makes a move, causing danger. The distance between the target user and the humanoid robot should also not be too far, because if the distance is too far, the facial image captured by the camera may be unclear, thus affecting the detection effect. In actual implementation, the target user can be determined whether it is within the preset detection range of the humanoid robot based on the above distance and angle.

[0039] Step 4: If it is confirmed that the target user is not within the preset detection range, output a movement command to instruct the target user to move to the preset detection range according to the movement command; When it is confirmed that the target user is not within the preset detection range of the humanoid robot, the humanoid robot can output clear movement instructions, such as "Please take a small step forward" or "Please take a small step back", to instruct the target user to move to the preset detection range according to the movement instructions.

[0040] Step 5: If the target user is confirmed to be within the preset detection range, output a posture guidance instruction to guide the target user to make various facial postures according to the posture guidance instruction; Once the target user is confirmed to be within the humanoid robot's preset detection range, the robot can further output posture guidance commands to guide the user to perform various facial poses according to the standard postures specified in the commands, ensuring the integrity of the subsequently acquired posture images. For example, the screen displays a frontal face icon with a voice prompt: "Please face the screen directly"; then a left-turning face icon is displayed with a prompt: "Please slowly turn your head to your left," etc. Throughout the guidance process, the humanoid robot can use the YOLOv8 facial landmark detection algorithm to track facial orientation in real time and provide confirmation feedback when the preset standard posture is reached.

[0041] like Figure 3The flowchart shown illustrates an implementation of the YOLOv8 facial landmark detection algorithm. After loading the YOLOv8 model, the algorithm first checks if the input image is valid. If invalid, it displays an input error message and terminates the process. If the input image is valid, it preprocesses the image and performs face detection to determine if a face is detected. If no face is detected, it outputs "no detection result" and terminates the process. If a face is detected, it further extracts facial landmarks and performs post-processing on their coordinates to track facial orientation. The implementation of this YOLOv8 facial landmark detection algorithm can be found in related technologies and will not be elaborated upon here.

[0042] Step 6: During the process of the target user making multiple facial poses, for each facial pose, if the facial pose meets the corresponding preset standard pose, the pose image corresponding to the facial pose is acquired. During the process of the target user making various facial poses, for each facial pose, if the pose meets a preset standard pose—for example, if the target user's frontal face pose meets the standard frontal face pose—the humanoid robot can control the high-definition camera on its head to capture images and obtain the pose image corresponding to that facial pose. It can be understood that the acquisition strategy in this embodiment is not a simple single shot, but a systematic multiple shot; that is, this solution implements multi-angle sequential acquisition, capturing at least one high-definition pose image for each facial pose during the guidance process, such as frontal, left, right, and tilted-head poses.

[0043] Step 7: If the facial pose belongs to a preset specified pose, obtain the key area corresponding to the facial pose, and zoom the preset camera to capture a close-up image of the key area. This solution can also achieve multi-focal distance acquisition. Specifically, if the facial pose is a preset specified pose, such as a frontal view, then the corresponding key area is usually the T-zone (including the forehead, nose, etc.). The preset camera is zoomed to acquire a close-up image of the T-zone through the preset camera.

[0044] Step 8: Preprocess each original image according to the preset preprocessing method to obtain the preprocessing result corresponding to each original image; wherein, the original images include: pose images corresponding to each facial pose and close-up images corresponding to each key area; In one embodiment, the original image may further include: a specified image; the specified image includes at least one of the following: a sebum image, a hemoglobin distribution map; the specified image is acquired by: acquiring the specified image by switching the light source of the preset multispectral acquisition hardware module or by enabling a camera with a specified spectrum of the multispectral acquisition hardware module.

[0045] In this embodiment, as an optional enhancement, a multispectral acquisition hardware module can be integrated into the humanoid robot. Multispectral acquisition is performed through this multispectral acquisition hardware module. Specifically, different light sources of the multispectral acquisition hardware module can be switched or a camera with a specified spectrum can be enabled to acquire sebum images that can see through the epidermal oil and hemoglobin distribution maps that can see through the subcutaneous hemoglobin distribution, for more accurate skin texture and redness analysis.

[0046] For ease of understanding, the following table provides a detailed parameter table of a multispectral acquisition hardware module, as shown in Table 2.

[0047] Table 2

[0048] After acquiring each of the above raw images, preprocessing is typically required to provide high-quality, standardized input for subsequent analysis. Preprocessing steps may include the following: 1. Illumination Compensation and Color Correction: Using an algorithm based on white balance and Retinex, the effects of ambient light are corrected to ensure accurate skin tone reproduction. See Table 3 for the input / output parameters and adjustment strategies of one Retinex algorithm. Table 3

[0049] 2. Face Alignment and Normalization: Based on facial key points, images from different angles are subjected to affine transformations and aligned to a standard face model to eliminate analysis errors caused by pose changes. Step 9: Based on the preprocessing results corresponding to each original image, crop out the key region image for skin analysis corresponding to the target user; In practice, based on the aligned facial landmarks mentioned above, key areas such as the forehead, left cheek, right cheek, nose, and chin can be automatically cropped for skin analysis.

[0050] Step 10: For each key region image, input the key region image into the pre-trained skin analysis model. The first sub-model of the skin analysis model outputs the skin color determination result; the second sub-model outputs the skin texture classification result; and the third sub-model outputs the skin problem detection result. Based on the skin color determination result, skin texture classification result, and skin problem detection result, generate the skin analysis result corresponding to the key region image. Step 11: Obtain the target user's historical profile data; To improve the accuracy of recommending targeted skincare products to target users, it is usually necessary to first obtain the target user's historical profile data. This historical profile data can be created through facial recognition or manual input, and typically includes information such as age, gender, past skin problems, and skincare products used. For example, it can include a user table as shown in Table 4 of a user skin management system, a past skin problem record table as shown in Table 5, and a skincare product information table as shown in Table 6.

[0051] Table 4

[0052] Table 5

[0053] Table 6

[0054] Step 12: Merge the skin analysis results corresponding to each key area image with historical archive data to obtain the fusion result; Step 13: Based on the fusion results, determine the core skincare needs of the target users; The skin analysis results corresponding to each key area image can be fused with historical archive data. Based on the fusion results, the core skin care needs of the target user can be mapped.

[0055] In practice, the skin analysis results for each key area image include scores for multiple preset skin detection dimensions. These dimensions may include oiliness and inflammation levels. Each dimension has its own corresponding score, which is typically associated with the severity of the skin problem; generally, a higher score indicates a more severe problem and thus a higher priority. Step thirteen may involve determining the target user's core skincare needs based on each score in the fusion results and historical skin problem records in the historical archive data. For example, for a user with "oily skin with numerous inflammatory acne," the highest priority needs are "oil control" and "anti-acne and anti-inflammatory"; while for a user with "dry skin with fine lines," the core skincare needs are "moisturizing" and "anti-wrinkle." The priority ranking is based on scores derived from the severity of the skin problem. For instance, a pre-set score threshold can be used to identify skin problems exceeding that threshold as the target user's core skincare needs.

[0056] Step Fourteen: Based on core skincare needs, match target skincare products that match the core skincare needs from the preset skincare product knowledge base; The aforementioned pre-built skincare product knowledge base typically stores information such as the ingredients, efficacy, skin type suitability, and user reviews of various skincare products. In actual implementation, once the core skincare needs are identified, a matching query can be performed within the skincare product knowledge base. The matching algorithm can be based on collaborative filtering and content-based recommendation to find the target skincare product with the highest degree of matching to the target user's core skincare needs.

[0057] Step 15: Generate a skincare routine plan based on the target skincare products; In practice, there are usually multiple target skincare products. After matching the target skincare products, a skincare routine plan containing each of the above target skincare products can be generated, such as: "Daytime Routine: Cleanser—Oil-controlling Toner—Vitamin C Serum—Salicylic Acid Acne-fighting Gel—Lightweight Sunscreen, etc." A brief explanation of the principle behind each recommended target skincare product is also provided.

[0058] The process of generating a skincare routine solution from the skincare product knowledge base is as follows: ① Convert the user query into a vector; ② Retrieve the most relevant document fragments from the skincare product knowledge base; ③ Typically return the top-k relevant results; ④ Combine the retrieved document fragments with the original user query; ⑤ Construct enhanced prompts containing contextual information; ⑥ The large language model generates an answer based on the enhanced prompts; ⑦ The answer output by the large language model utilizes both its internally pre-trained parameterized knowledge and external information retrieved from the skincare product knowledge base, which can improve user experience and satisfaction.

[0059] Step 16: Display the skincare routine on the screen and / or play the skincare routine via voice. The humanoid robot can display the above skincare routine on its screen, and / or, through its built-in voice synthesis module, play the generated skincare routine to the target user via voice, thus presenting the skincare routine to the target user in an intuitive and easy-to-understand way.

[0060] Step 17: Display the results on a screen and / or play them aloud; the results include: skin analysis results, as well as images, names, core ingredients, and reasons for recommendation of the target skincare product.

[0061] In practical implementation, the skin analysis results can be displayed on a screen, including a method diagram of problem areas and a scoring radar chart for each dimension, forming a visual report. The screen can also display images, names, core ingredients, and reasons for recommendation of recommended target skincare products, which can be categorized by skincare steps in a tabbed manner. In addition, the humanoid robot provides simultaneous voice explanations, summarizing the analysis results and reasons for recommendation. Specifically, the built-in voice synthesis module can play the above display results to the target user through voice, further enhancing the user experience.

[0062] Step 18: If a question about the target skincare product is received from the target user, the target answer corresponding to the question is extracted from the preset information knowledge base and fed back to the target user.

[0063] For ease of understanding, see Figure 4 The flowchart of another skincare product recommendation method shown mainly includes three parts: proactive interaction and data collection, intelligent analysis and recommendation, and result presentation and feedback. First, after the robot determines that a target user exists, the service starts and provides interactive guidance. Then, multi-angle / multi-spectral image collection is performed, and image preprocessing is carried out to crop out key area images. Based on the key area images, multi-dimensional analysis of skin condition is performed, and finally, personalized recommendations are generated for the target user to generate suitable target skincare products. Skincare process plans can be generated based on target skincare products, and the skincare process plans and display results can be visualized. In addition, the robot can also support user interactive Q&A, ultimately forming a one-stop closed-loop service solution of "skin detection - skin analysis - skincare product recommendation".

[0064] The above-mentioned methods for recommending skincare products mainly have the following key technical points: 1. Actively Guided Image Acquisition: Utilizing the humanoid robot's anthropomorphic interactive capabilities (voice, screen), the robot actively guides the target user to complete standardized position adjustments and facial poses, ensuring the consistency of acquired images and fundamentally improving analysis accuracy.

[0065] 2. Multi-dimensional collaborative analysis model: It adopts a group of deep learning models designed for multi-angle and multi-focal distance facial images to perform collaborative analysis and comprehensive diagnosis of skin color, skin texture and skin problems.

[0066] 3. Dynamic Personalized Recommendation Engine: Combines real-time skin analysis results with the target user's long-term historical data, maps core skin care needs through algorithms, and generates deeply personalized skin care solutions and product recommendations by matching from the skin care product knowledge base.

[0067] 4. Personalized Service Loop: The robot itself (display screen, voice) presents and explains the analysis results and recommended solutions in an integrated and contextualized manner, and supports real-time interactive Q&A, realizing a complete service loop from detection to consultation.

[0068] Compared with existing technologies, this solution has the following outstanding advantages: 1. Significantly improved detection accuracy and reliability By actively guiding and controlling a humanoid robot, the inconsistencies in lighting, angle, and distance caused by user selfies in mobile app solutions are eliminated. The humanoid robot autonomously ensures that each inspection is performed under standardized optimal conditions, providing high-quality, highly consistent image input for subsequent visual analysis. This controlled data acquisition environment, combined with targeted image preprocessing, fundamentally guarantees the accuracy and reliability of skin color determination, skin texture classification, and skin problem identification.

[0069] 2. It achieved a humanized and guided service experience. Unlike fixed, impersonal desktop testing devices or passive mobile apps, this solution fully leverages the unique anthropomorphic interactive advantages of humanoid robots. Like a professional beauty consultant, the humanoid robot proactively communicates with users through voice and animation, guiding them step-by-step through standardized postures and providing real-time feedback. This warm and guided interaction significantly enhances the target user's trust and engagement, transforming a functional testing process into an immersive service experience.

[0070] 3. It breaks the limitations of professional service scenarios and achieves a service closed loop. This solution successfully liberates professional-grade skin analysis capabilities from the confines of beauty salons or clinics. Leveraging its mobility, robots equipped with this system can be flexibly deployed in various consumer and lifestyle scenarios, including shopping malls, brand stores, airports, and even homes. More importantly, it can directly connect to offline shelves or online stores, immediately providing specific product recommendations and even guiding customers to purchase after completing the testing and analysis. This achieves a complete service loop from "skin testing" and "skin analysis" to "skincare product recommendations" and ultimately "product acquisition," resulting in immense commercial value.

[0071] 4. Provides truly dynamic and in-depth personalization. Compared to mobile apps that typically offer static recommendations based on single tests, this solution establishes and maintains historical profiles of target users, enabling the fusion and trend analysis of results from past tests. This allows the system to understand long-term changes in a user's skin condition and, combined with their historical preferences, provide dynamically evolving and deeply personalized skincare solutions. For example, the system can identify seasonal changes in a user's skin or assess the long-term effects of a recommended product, thus providing a more forward-looking intelligent advisory service that goes beyond single-test analysis.

[0072] This invention provides a skincare product recommendation device, which is installed on a robot; such as Figure 5 As shown, the device includes: a guidance module 50, used to receive skin analysis instructions from a target user and output posture guidance instructions according to the skin analysis instructions to guide the target user to make various facial postures according to the posture guidance instructions; an acquisition module 51, used to acquire posture images corresponding to each facial posture and close-up images of key areas corresponding to specified postures during the process of the target user making various facial postures; a determination module 52, used to determine the key area images for skin analysis corresponding to the target user based on the original images; wherein, the original images include: posture images corresponding to each facial posture and close-up images corresponding to each key area; and an analysis module 53, used to analyze each key area image. The key region image is input into a pre-trained skin analysis model. The first sub-model of the skin analysis model outputs the skin color determination result; the second sub-model outputs the skin texture classification result; and the third sub-model outputs the skin problem detection result. Based on the skin color determination result, skin texture classification result, and skin problem detection result, the skin analysis result corresponding to the key region image is generated. The recommendation module 54 is used to recommend target skin care products to the target user based on the skin analysis result of each key region image. The extraction module 55 is used to extract the target answer corresponding to the question from the preset information knowledge base if it receives a question from the target user about the target skin care product, and then feed it back to the target user.

[0073] The aforementioned skincare product recommendation device can determine key region images based on original images containing pose and close-up images, and input them into a skin analysis model. Through the first, second, and third sub-models in this skin analysis model, the device collaboratively analyzes the target user's skin tone, skin texture, and skin problems to generate skin analysis results for each key region image. This allows the device to recommend suitable target skincare products to the user and answer questions from the target user. It can provide a one-stop closed-loop service of "skin detection - skin analysis - skincare product recommendation" in various life scenarios.

[0074] Furthermore, the guidance module is also used to: receive skin analysis instructions from the target user, and acquire a first image containing the target user through a preset camera; calculate the distance between the target user and the robot, and the angle of the target user relative to the robot, based on the first image; determine whether the target user is within the robot's preset detection range based on the distance and angle; if it is confirmed that the target user is not within the preset detection range, output a movement instruction to instruct the target user to move to the preset detection range according to the movement instruction; if it is confirmed that the target user is within the preset detection range, output a posture guidance instruction to guide the target user to make various facial postures according to the posture guidance instruction.

[0075] Furthermore, the acquisition module is also used to: during the process of the target user making multiple facial poses, for each facial pose, if the facial pose meets the corresponding preset standard pose, acquire the pose image corresponding to the facial pose; if the facial pose belongs to a preset specified pose, obtain the key area corresponding to the facial pose, and perform zoom processing on the preset camera to acquire a close-up image corresponding to the key area.

[0076] Furthermore, the original image also includes: a specified image; the specified image includes at least one of the following: a sebum image, a hemoglobin distribution map; the specified image is acquired by: switching the light source of the preset multispectral acquisition hardware module or enabling a camera with a specified spectrum of the multispectral acquisition hardware module to acquire the specified image.

[0077] Furthermore, the determination module is also used to: preprocess each original image according to a preset preprocessing method to obtain the preprocessing result corresponding to each original image; and based on the preprocessing result corresponding to each original image, crop out the key region image for skin analysis corresponding to the target user.

[0078] Furthermore, the recommendation module is also used to: obtain the target user's historical profile data; merge the skin analysis results corresponding to each key area image with the historical profile data to obtain the fusion result; determine the target user's core skin care needs based on the fusion result; and match target skin care products that match the core skin care needs from the preset skin care product knowledge base based on the core skin care needs.

[0079] Furthermore, the skin analysis results corresponding to each key area image include scores for multiple preset skin detection dimensions; the recommendation module is also used to: determine the core skin care needs of the target user based on each score in the fusion results and historical skin problem records in the historical archive data.

[0080] Furthermore, the device is also used to: generate a skincare routine based on the target skincare product; display the skincare routine on a screen, and / or play the skincare routine via voice; display the results on a screen, and / or play the results via voice; wherein the results include: skin analysis results, as well as images, names, core ingredients, and reasons for recommendation of the target skincare product.

[0081] Furthermore, the key region images include at least one of the following: a forehead image, a left cheek image, a right cheek image, a nose wing image, and a chin image.

[0082] The skincare product recommendation device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned skincare product recommendation method embodiment. For the sake of brevity, any parts not mentioned in the skincare product recommendation device embodiment can be referred to the corresponding content in the aforementioned skincare product recommendation method embodiment.

[0083] This invention also provides an electronic device, see [link to relevant documentation]. Figure 6 As shown, the electronic device includes a processor 130 and a memory 131. The memory 131 stores machine-executable instructions that can be executed by the processor 130. The processor 130 executes the machine-executable instructions to implement the above-mentioned skincare product recommendation method.

[0084] Furthermore, Figure 6 The electronic device shown also includes a bus 132 and a communication interface 133, with the processor 130, the communication interface 133 and the memory 131 connected via the bus 132.

[0085] The memory 131 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 133 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 132 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0086] Processor 130 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 130 or by instructions in software form. Processor 130 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 131, and processor 130 reads the information in memory 131 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0087] This invention also provides a machine-readable storage medium storing machine-executable instructions. When these machine-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned skincare product recommendation method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0088] The computer program product of the skin care product recommendation method and apparatus provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0089] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recommending skincare products, characterized in that, The method is applied to a robot; the method includes: Receive skin analysis instructions from the target user, and output posture guidance instructions according to the skin analysis instructions to guide the target user to make various facial postures according to the posture guidance instructions; During the process of the target user making various facial poses, pose images corresponding to each facial pose and close-up images of key areas corresponding to a specified pose are captured. Based on the original image, key region images for skin analysis corresponding to the target user are determined; wherein, the original image includes: a pose image corresponding to each facial pose and a close-up image corresponding to each key region; For each key region image, the key region image is input into a pre-trained skin analysis model. The first sub-model of the skin analysis model outputs a skin color determination result; the second sub-model outputs a skin texture classification result; and the third sub-model outputs a skin problem detection result. Based on the skin color determination result, the skin texture classification result, and the skin problem detection result, a skin analysis result corresponding to the key region image is generated. Based on the skin analysis results of each key region image, target skincare products are recommended to the target user. If a question about the target skincare product is received from the target user, the target answer corresponding to the question is extracted from a preset information knowledge base and fed back to the target user.

2. The method according to claim 1, characterized in that, The steps of receiving skin analysis instructions from a target user and outputting posture guidance instructions based on the skin analysis instructions to guide the target user to perform various facial postures according to the posture guidance instructions include: Receive a skin analysis command from a target user and capture a first image containing the target user through a preset camera; Calculate the distance between the target user and the robot, and the angle of the target user relative to the robot, based on the first image; Based on the distance and the angle, determine whether the target user is within the robot's preset detection range; If it is confirmed that the target user is not within the preset detection range, a movement command is output to instruct the target user to move to the preset detection range according to the movement command; If the target user is confirmed to be within the preset detection range, a posture guidance instruction is output to guide the target user to make various facial postures according to the posture guidance instruction.

3. The method according to claim 1, characterized in that, The steps of acquiring a pose image corresponding to each facial pose and a close-up image of a key area corresponding to a specified pose during the process of the target user making multiple facial poses include: During the process of the target user making multiple facial poses, for each facial pose, if the facial pose meets the corresponding preset standard pose, the pose image corresponding to the facial pose is captured. If the facial pose belongs to a preset specified pose, the key area corresponding to the facial pose is obtained, and the preset camera is zoomed to capture a close-up image of the key area.

4. The method according to claim 1, characterized in that, The original image further includes: a specified image; the specified image includes at least one of the following: a sebum image, a hemoglobin distribution map; the specified image is obtained through the following methods: The specified image is acquired by switching the light source of the preset multispectral acquisition hardware module or by enabling the camera of the specified spectrum of the multispectral acquisition hardware module.

5. The method according to claim 1, characterized in that, The steps for determining the key region image for skin analysis corresponding to the target user based on the original image include: Each original image is preprocessed according to a preset preprocessing method to obtain the preprocessing result corresponding to each original image; Based on the preprocessing results corresponding to each of the original images, the key region images for skin analysis corresponding to the target user are cropped out.

6. The method according to claim 1, characterized in that, The steps for recommending target skincare products to the target user based on the skin analysis results of each key region image include: Obtain the target user's historical profile data; The skin analysis results corresponding to each of the key area images are fused with the historical archive data to obtain the fusion result; Based on the fusion results, the core skincare needs of the target users are determined; Based on the core skincare needs, target skincare products that match the core skincare needs are matched from a preset skincare product knowledge base.

7. The method according to claim 6, characterized in that, The skin analysis results corresponding to each key region image include scores for multiple preset skin detection dimensions. Based on the fusion results, the steps for determining the core skincare needs of the target user include: Based on each score result in the fusion results and the historical skin problem records in the historical archive data, the core skin care needs of the target user are determined.

8. The method according to claim 1, characterized in that, The method further includes: Based on the target skincare product, a skincare process plan is generated; The skincare routine is displayed on a screen and / or played via voice. The results are displayed on the screen and / or played via voice; wherein the results include: the skin analysis results, and the image, name, core ingredients, and reasons for recommendation of the target skin care product.

9. The method according to claim 1, characterized in that, The key region images include at least one of the following: forehead image, left cheek image, right cheek image, nose wing image, and chin image.

10. A skincare product recommendation device, characterized in that, The device is mounted on the robot; the device includes: The guidance module is used to receive skin analysis instructions from the target user and output posture guidance instructions according to the skin analysis instructions to guide the target user to make various facial postures according to the posture guidance instructions; The acquisition module is used to acquire a pose image corresponding to each facial pose and a close-up image of the key area corresponding to a specified pose during the process of the target user making multiple facial poses. The determination module is used to determine, based on the original image, key region images for skin analysis corresponding to the target user; wherein, the original image includes: a pose image corresponding to each facial pose and a close-up image corresponding to each key region; The analysis module is used to input each key region image into a pre-trained skin analysis model, output skin color determination results through the first sub-model of the skin analysis model, output skin texture classification results through the second sub-model, and output skin problem detection results through the third sub-model; and generate skin analysis results corresponding to the key region image based on the skin color determination results, the skin texture classification results, and the skin problem detection results. The recommendation module is used to recommend target skincare products to the target user based on the skin analysis results of each key region image; The extraction module is used to extract the target answer corresponding to the question from a preset information knowledge base if it receives a question from the target user regarding the target skin care product, and then feed it back to the target user.

Images