Architecture for a personalized beauty experience using a large language model

Abstract

Architecture for a personalized beauty experience using a large language model: A computer system passes user input and contextual information to a large language model (LLM) and requests the LLM to confirm that the input text relates to one or more beauty themes. Based on the confirmation, the system requests the LLM to provide a response to the input text to be presented to the user. The response relates to the beauty theme(s) and is based on the input text and contextual information. The confirmation may include requesting the LLM to produce one or more classifications of the input text, which indicate that the input text relates to the beauty theme(s).The system can request the LLM to produce a summary of the input text, generate a vector representation, and identify matches for the vector representation of the user input among other vector representations in the database (for example, for relevant products or content). Figure for the summary: none.

Description

Title of the invention: Architecture for a personalized beauty experience using a large language model SUMMARY

[0001] This summary is provided to present a selection of concepts in a simplified form, which are described in greater detail below in the detailed description. This summary is not intended to identify key features of the claimed subject matter, nor should it be used as an aid in determining the scope of the claimed subject matter.

[0002] In one aspect, a computer system performs operations comprising transmitting user input and contextual information for the user input to a large language model (LLM); requesting the LLM to provide confirmation that the user input relates to one or more beauty themes; receiving confirmation from the LLM that the user input relates to one or more beauty themes; based on the confirmation, requesting the LLM to provide a response to the user input to be presented to a user, in which the response relates to one or more beauty themes and is based at least in part on the user input and contextual information; and receiving the response from the LLM.

[0003] In another aspect, a computer system performs operations comprising receiving user input transmitted by a user via a user interface presented on a client computer device; obtaining contextual information for the user input; transmitting the user input and the contextual information for the user input to an LLM; requesting the LLM to provide confirmation that the user input relates to one or more beauty themes; receiving confirmation from the LLM that the user input relates to one or more beauty themes; based on the confirmation, requesting the LLM to provide a response to the user input to be presented to a user, in which the response relates to one or more beauty themes and is based at least in part on the user input and the contextual information; receiving the response from the LLM;and the act of presenting the answer to the user via the user interface.

[0004] In some embodiments, user input includes text input or voice input. In some embodiments, the user interface includes an online chat interface, and the response includes text presented to the user in the online chat interface. In some embodiments, the user interface includes a user interface element of skin analysis request, and the operations further include receiving an indication to activate the skin analysis request user interface element; and, in response to the indication to activate the skin analysis request user interface element, obtaining a digital model of the user's face; and requesting the LLM to generate a product recommendation or skincare routine recommendation based at least in part on the digital face model.In some embodiments, the user interface includes a product information user interface element configured to allow the user to view information corresponding to the recommended products, and the operations further include requesting the LLM to generate one or more product recommendations; receiving the one or more product recommendations from the LLM; and causing a client computing device to present the one or more product recommendations to the user via the product information user interface element.In some embodiments, the user interface includes a content selection user interface element configured to allow the user to view information corresponding to recommended content, and the operations further include requesting the LLM to generate one or more content recommendations; receiving the one or more content recommendations from the LLM; and causing a client computing device to present the one or more content recommendations to the user via the content selection user interface element.

[0005] In some embodiments, the request to the LLM to provide confirmation that the user input relates to one or more beauty themes includes the request to the LLM to produce one or more classifications of the user input, the receipt of confirmation that the user input relates to one or more beauty themes includes the receipt of one or more classifications of the user input from the LLM, and the one or more classifications indicate that the user input relates to one or more beauty themes.In some embodiments, the request to the LLM to produce one or more classifications of the user input includes requesting the LLM to produce a first classification of the user input; receiving the first classification of the user input from the LLM, in which the first classification indicates that the user input relates to one or more beauty themes; and, in response to the first classification indicating that the user input relates to one or more beauty themes, requesting the LLM to produce a second classification of the user input that further defines the one or more beauty themes. In such... In terms of implementations, the request to the LLM to provide the response to the user input can be based on both the first classification and the second classification.

[0006] In some embodiments, the operations further include requesting the LLM to produce a summary of the user input and may also include receiving the summary of the user input from the LLM and generating a vector representation of the user input. In some embodiments, the operations further include comparing the vector representation of the user input with vector representations of other vector representations in a vector database and identifying a near neighbor match for the vector representation of the user input among the other vector representations in the vector database.

[0007] In some embodiments, the operations further include obtaining a digital model of the user's face and requesting the LLM to generate a product recommendation or a skincare routine recommendation based at least in part on the digital face model. The digital face model may include a plurality of skin characteristics, including information on imperfections, hyperpigmentation, skin texture, complexion, or other information, or a combination thereof.

[0008] In some embodiments, the computer system includes or communicates with a client computer device comprising a camera. In such embodiments, the operations may further include causing the client computer device to request the activation of the camera to capture one or more digital images; receiving the one or more captured digital images; and generating a digital model of the user's face based, at least in part, on the one or more captured digital images. In some embodiments, the operations further include requesting the LLM to generate a product recommendation or a skincare routine recommendation based at least in part on the digital face model.

[0009] Computer-implemented processes, computer-readable media, computer devices and computer systems are disclosed. Brief description of the drawings

[0010] The foregoing aspects and many related advantages of the present invention will be more readily appreciated as they are better understood with reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

[0011] [Fig-1] [Fig. 1] is a diagram that provides a high-level overview of a computer system according to aspects of this disclosure;

[0012] [Fig.2] [Fig.2] is a schematic diagram illustrating an example of a mode of implementation of a data flow and logic in a back-end server computer system according to aspects of this disclosure;

[0013] [Fig.3] [Fig.3] is a schematic diagram illustrating an example of mode implementation of a client IT system according to aspects of this disclosure;

[0014] [Fig 4A-4D] Figures 4A to 4D are screenshot diagrams of an illustrative user interface, according to a described embodiment;

[0015] [Fig. 5] [Fig. 5] is a flowchart that illustrates an example of an embodiment of a method for using an LLM to obtain responses to user input on beauty topics according to aspects of this disclosure;

[0016] [Fig.6] [Fig.6] is a schematic diagram illustrating aspects of an example of appropriate computer equipment for use with embodiments of this disclosure. Detailed description

[0017] The described embodiments include methods and systems for providing personalized beauty recommendations based on user preferences and needs, as well as available products. The described embodiments utilize real-time data from direct user interaction, along with product information, to offer highly personalized recommendations using a large language model (LLM) approach. The described embodiments are further designed to avoid hallucinations in this context, which is a major challenge with LLMs.

[0018] In an illustrative use case, a user interacts with the system using a smartphone, laptop, or other client computing device. The user can log in to the system with account credentials or use the system as a guest. In some embodiments, the system can access the user's information, online chat history, and similar data, and can use this information to personalize the user's interaction with the system. A user interface provides a mechanism for the user to interact with the system (for example, via voice interactions or online chat with a chatbot using an LLM). In the user interface, one or more interaction options are presented to the user, such as sample questions to ask the system or a general invitation to ask any question.The user then asks questions. or makes statements via the user interface, and these questions or statements are given as input to the system for further analysis. In one embodiment, the system is configured to suggest taking a self-portrait photo for further analysis if knowledge of the user's appearance or hair or skin condition is helpful in guiding the system's response. In such an embodiment, the system analyzes the photo to determine characteristics (e.g., gender, hair color, head shape, eye color, complexion, etc.) that can be used to provide personalized answers or recommendations.

[0019] The system is capable of operating with a general-purpose LLM (e.g., GPT-3 or GPT-4, available from OpenAI Inc., LaMDA / Bard, available from Google LLC, etc.), which does not need to be specially trained on beauty topics. In some embodiments, the system specifically controls and personalizes interactions with the LLM to improve performance and reduce the risk of hallucinations or inaccurate responses. In this way, the system significantly improves upon previous dialoguers using a general-purpose LLM. In some embodiments, when the user initiates an online dialogue, the system prompts the LLM to respond from the perspective of a highly trained beauty consultant, which helps ensure that responses are given in a style the user expects and that the responses remain focused on beauty topics.When the user asks a question or makes a statement, the system adds contextual information. In some embodiments, contextual information is given in the form of prompts to provide context for the question or statement, such as special definitions or constraints to be considered by the LLM, user profile information (e.g., preferences, characteristics, products used, etc.), summaries of past conversations, or other contextual information.

[0020] The system uses the LLM to classify the question and determine if it is well-formulated. If the question is not well-formulated, the system asks the user clarifying questions. If the question is well-formulated and the LLM can answer it, the LLM transmits the answer to the system, and the system presents the answer to the user. Classifying the question ensures that answers are given using appropriate resources. For example, if the user is asking about skin problems, the system uses skin treatment information to provide the answer. The system uses a similar approach for other types of questions, including questions about makeup, hair care, and nail care. If the question can be better answered by analyzing a photo of the user, the system can request access to the photos. recorded from the user or a camera. User photos can be analyzed to assess the user's skin concerns and suggest personalized skincare routines or product recommendations. In some embodiments, the camera is used to analyze makeup, hair, or nails in real time, which may include the use of augmented reality virtual try-on applications. In some embodiments, the system obtains additional information, such as location data, which can then be used to tailor responses to the user. For example, location data can be used to identify suitable stores for purchasing products, to assess the user's environmental conditions (e.g., humidity, temperature, UV conditions, etc.).) to guide product recommendations or routine care recommendations, or for other purposes.

[0021] In certain embodiments, the system presents the user with a user interface which, in addition to the online dialogue functionality, allows the user to view product information, view tutorial or promotional content, identify locations for purchasing products in stores or salons, purchase products via a website, or perform other tasks.

[0022] The LLMs described here in the context of embodiments can be monomodal (e.g., receiving text input and responding with text responses) or multimodal (e.g., receiving text and images or other input modes and proposing text and images or other output modes).

[0023] Fig. 1 is a schematic diagram illustrating a system in which various aspects of the present disclosure can be implemented. As illustrated, the system 100 includes one or more client computing devices 104, a front-end server system 106, a back-end server system 110, a skin analysis engine 112, a virtual testing engine 114, a user data store 116, a vector database 120, a product data store 122, a content data store 124, and an LLM 130.

[0024] The client computing device 104 can be used by a consumer to interact with other components of the system 100, such as the front-end server system 106. In one embodiment, the client computing device 104 is a mobile computing device, such as a smartphone or tablet computer. However, any other suitable type of computing device capable of communicating over the network and presenting a user interface, including, but not limited to, a desktop computing device, a portable computing device, augmented / virtual reality glasses, or a dedicated device enhanced by AI such as the Humane AI Pin or Rabbit RI device, a smart speaker or a smartwatch (or combinations of these devices) can be used.

[0025] The front-end server system 106 includes one or more server computers that provide an interface for the client computer device 104 to access the functionality of the system 100. In one embodiment, the front-end server system 106 provides a web interface through which an end user can access such functionality, for example, via an internet browser or a dedicated client application. In one embodiment, the front-end server system 106 is responsible for generating the user interface and communicating user requests to the back-end server system 110. An illustrative user interface is described in more detail below.

[0026] In one embodiment, the front-end server system 106 also provides access to a digital personal care application, which can be implemented by or accessed via the back-end server system 110. The application can offer e-commerce functionalities for purchasing, paying for, and delivering products or services. In an illustrative scenario, a consumer orders a service and a product online via a virtual storefront for a service provider, such as a salon. The consumer can access the virtual storefront via the front-end server system 106, which can then submit orders to the back-end server system 110 for further processing and fulfillment. Fulfillment can include delivering an ordered product (for example, to the consumer's address or to the service provider's location before the consumer's scheduled appointment).The execution may also include the transmission of product information or digital content to the client computer device 104.

[0027] In one embodiment, the front-end server system 106 and / or the back-end server system 110 implement an application programming interface (API) that allows service providers, manufacturers, and others to offer services and products through the application. For example, the application may offer functionality to connect consumers with experts, retailers, or service providers via audio calls, video calls, instant messaging, email, or similar means to receive advice or request information about products or services. The application may provide a platform for experts, service providers, or retailers to offer content, such as videos, articles, or the like, to existing or potential customers.

[0028] The back-end server system 110 includes one or more server computers. As illustrated, the back-end server system 110 communicates with an analysis engine of Skin 112, a virtual testing engine 114, a user data store 116, a vector database 120, and an LLM 130. In one embodiment, the vector database 120 ingests product information from data sources such as the product data store 122 (e.g., product descriptions, product ingredients), the content data store 124 (e.g., product safety information, reviews, video tutorials, promotional videos, websites). The vector database 120 stores embeds (vector data representations) that represent this content. For videos, the audio is transcribed into text (e.g., using the Whisper machine learning model, available from OpenAI Inc.), and embeds of these transcriptions are added to the vector database 120.On the user side, a user's prompts or queries are also represented as embeds, allowing comparison with embeds already present in the vector database. Since these embeds are numerical, it is possible to search for content relevant to a user query by looking for "near neighbors" of an embed in the user query. This enables the retrieval of relevant documents, videos, or other content from the vector database. In some embodiments, embeds of user prompts or queries are also added to the vector database to further enrich it.

[0029] In one embodiment, the LLM 130 plays several roles in the system 100. By way of role illustration, the system uses the LLM 130 to synthesize user prompts or requests, distilling them into a form that can be more easily and accurately transformed into an embed. For example, the back-end server system 110 can transmit a user request with a prompt to the LLM 130 to synthesize this request into two or three sentences, which can be more easily transformed into an embed that accurately represents the user's request. The additional functionalities of the LLM 130 are described below.

[0030] Figure 2 is a schematic diagram illustrating an example of an embodiment of a data flow and user interaction logic in the back-end server system 110 according to aspects of this disclosure. The illustrative design shown in Figure 2 represents the expected operation of the dialogist aspects of the system 100 and helps ensure that the system is used safely, provides relevant results, and reduces the risk of hallucination by the LLM 130. In the example shown in Figure 2, user input is passed to a first classifier 140 which classifies the user input into different categories, and can also be called an input classifier. In one embodiment, the first classifier 140 accomplishes this by passing the user input to the LLM 130 and requesting the LLM 130 to classify the input into one of the following categories: "Health," "Vulnerable," "Harmful," "Safety," "Normal," "Video," or "Ethical." Alternatively, the first classifier 140 can be implemented as a natural language classifier specifically designed to classify the input without needing to access the LLM 130. These classifications can be defined as follows:

[0031] Normal: The request is related to beauty advice and is politically neutral, does not involve ethical concerns, and is not offensive.

[0032] Health: The application suggests that the user needs medical attention, has mental health problems (such as depression or severe anxiety), has serious physical health problems, or is pregnant. This category does not include standard forms of skin problems, such as acne, eczema, redness, or rashes.

[0033] Vulnerable: The application suggests that the user is a vulnerable person (such as a child or a drug addict) who is susceptible to manipulation.

[0034] Harmful: The application is offensive (e.g., racist or discriminatory), unlawful or violent, or manifestly political, or suggests the use of products in a potentially malicious manner.

[0035] Security: The request raises privacy or data security issues, or attempts to hack the dialog (for example, to obtain information about its configuration or to change its functionality or role).

[0036] Ethics: The request relates to the ethics or values ​​of the company.

[0037] Videos: The purpose of the request is to access a video.

[0038] In one embodiment, the backend server system 110 provides instructions to the LLM 130 accordingly by transmitting one or more additional prompts that define these categories, which the LLM 130 uses to categorize user input. The backend server system 110 can transmit additional prompts to the LLM 130, such as a prompt for the LLM 130 to return not only a category, but also a reason why the input is assigned to that category.

[0039] In the example shown in [Fig. 2], if the LLM 130 classifies the entry in the "Health" or "Vulnerable" category, the back-end server system 110 flags the user accordingly in process block 150, with the reason for the flagging (e.g., user likely pregnant as the reason for classification in the "Health" category; user likely a child as the reason for classification in the "Vulnerable" category) and offers a predefined response to the user in process block 152. The flagging information can be added to a profile for that user (for example, in user data store 116). In future interactions with the user, the back-end server system 110 takes this information into account. In an illustrative scenario, a product recommendation considers the previous report when preparing the response and displays a warning message to the user. If LLM 130 classifies the entry as "Harmful" or "Security," the back-end server system 110 displays predefined messages in process block 152. If LLM 130 classifies the entry as "Video" or "Ethics," the back-end server system 110 retrieves corresponding information from a video database or the ethics website in process blocks 154 and 156, respectively.

[0040] For an input initially classified as "Normal Request," the back-end server system 110 performs an additional level of classification at the second classifier 158. In one embodiment, the second classifier 158 accomplishes this by forwarding the request (which is now classified as "Normal Request" by the first classifier 140) to the LLM 130 and prompting the LLM 130 to classify the request into one of the following categories: "Ingredients," "Product Information," "Virtual Trial," "Skin Analysis," or "Other." Thus, the second classifier 158 can also be referred to as the request classifier. This additional level of classification helps to reduce the risk of hallucinations or inaccurate results.In one embodiment, the back-end server system 110 instructs the LLM 130 accordingly by passing one or more prompts that define these categories, which the LLM 130 uses to further categorize the user input. Alternatively, the second classifier 158 can be implemented as a natural language classifier specifically designed to classify the request without needing to access the LLM 130. These additional classifications can be defined as follows.

[0041] Ingredients: A request for information on the ingredients of a product.

[0042] Product recommendation: A request for a product recommendation.

[0043] VTO Request: A virtual trial request.

[0044] Skin analysis: A request for skin analysis.

[0045] Other: Catch-all category for requests that do not fit into any of the other categories.

[0046] In the example shown in [Fig.2], if the LLM 130 classifies the request in the category "Ingredients", the back-end server system 110 obtains corresponding information from an ingredients database in the process block 160. In one embodiment, the back-end server system 110 retrieves the appropriate documents specifying the ingredients of a product and sends them to the user.

[0047] If LLM 130 classifies the request in the "Product Recommendation" category, back-end server system 110 determines whether the user has been previously flagged (for example, in "Health" or "Vulnerable") in decision block 162. In the example shown in [Fig. 2], if the user has been previously flagged, back-end server system 110 obtains a modified product recommendation in process block 164, for example, by sending the user's request to LLM 130 with additional context regarding the flagging to restrict and / or provide a warning with the recommendation. Otherwise, back-end server system 110 obtains a product recommendation in process block 166, for example, by sending the user's request to LLM 130, without such warnings.

[0048] If the LLM 130 classifies the request as a "VTO Request," the back-end server system 110 can activate a camera on the client computer device 104 at process block 168 and transmit the corresponding required product information to an application on the client computer device 104 to initiate a virtual trial process (for example, by running the virtual trial engine 114 to try a cosmetic product in an augmented reality application). The virtual trial process may include interaction with the LLM 130 at process block 172, for example, by using an online chat interface to obtain product information and virtually applying the recommended product in the virtual trial process.

[0049] If the LLM 130 classifies the request in the "Skin Analysis" category, the back-end server system 110 can activate a camera on the client computer device 104 at process block 168 and / or process a self-portrait of the user to obtain a skin analysis report (for example, by the skin analysis engine 112) at process block 170. The skin analysis process may include interaction with the LLM 130 at process block 172, for example, by using an online chat interface to transmit a skin analysis report and obtain a product or skincare routine recommendation from the LLM 130. If the LLM 130 classifies the request in the "Other" category, the back-end server system 110 requests the LLM 130 to provide a response that can be sent to the user (process block 172).Thus, for certain categories, LLM 130 is required to provide a complete answer to the user's query, rather than simply classifying the query.

[0050] The backend server system 110 can use prompt engineering to increase the likelihood of useful and on-topic responses. For example, the backend server system 110 can forward a user's request to the LLM 130, along with one or more prompts that supplement or restrict the request, such as information user profile, user preferences, or instructions on how to respond to the query (for example, recommending only products of a specified type or from a specific company). The back-end server system 110 can further request the LLM 130 to summarize the user query, which is an effective vector database embedding technique (as described above). In some embodiments, multiple LLMs can be used to cross-check responses and reduce the risk of hallucinations or inaccurate responses that can occur with a single LLM operating independently.

[0051] Referring again to [Fig. 1], in one embodiment, the virtual try-on engine 114 allows consumers to apply different styles or features of a product, for example, using augmented reality techniques, to modify an image of the consumer's face, hair, skin, etc. This technology can also be used to match colors, compare products to other products, and test variations in features such as coverage, color, finish, etc. In one embodiment, the skin analysis engine 112 generates a digital model (for example, based on one or more digital images or scans) of a human subject's face. In one embodiment, the skin analysis engine 112 obtains one or more digital images or scans from the client computing device 104, such as a smartphone with an integrated digital camera.In such an embodiment, these images or scans are captured by the client computing device 104 and loaded into the skin analysis engine 112, which generates the digital model and detects clinical signs (e.g., of aging) and / or skin problems of the user. In one embodiment, the digital model includes a highly accurate model of facial features and characteristics, including the edges of the lips and eyes, the size and location of the iris, skin features such as blemishes, texture, and wrinkles, and the like. It will be understood that the skin features and characteristics described herein are only examples, and that other features or combinations thereof are also desirable and fall within the scope of this disclosure.In one embodiment, source images are captured and digital models are generated using Modiface software available from Modiface, Inc.

[0052] Figure 3 is a schematic diagram illustrating an example of an embodiment of a client computing device 104 according to various aspects of this disclosure. Figure 3 presents a non-limiting example of features and configurations of a client computing device; many other features and configurations are possible within the scope of this disclosure.

[0053] In the example shown in [Fig. 3], the client computer device 104 includes a camera 250 and a client application 260. The client application 260 includes a user interface 276, which may include interactive features such as data collection or online voice / dialogue elements, tools for entering or modifying user preferences, tutorials, a virtual "try-on" feature for virtually trying different products or cosmetics, or other features. In one embodiment, the user interface 276 provides functionality for exploring personalized products recommended by healthcare professionals, such as personalized product formulations (e.g., personalized formulations of hair treatment products, makeup products, etc.).), customized product combinations to achieve a particular style or effect, product variations (e.g., color, finish, texture), and the like. Visual elements of the user interface 276 are presented on a display 240, such as a touchscreen display. Customized content, such as personalized product recommendations and skincare routines, can be retrieved by the client computing device 104 (e.g., from the back-end server system 110) and presented via the user interface 276. Details of an illustrative user interface are described below with reference to Figures 4A to 4D.

[0054] In one embodiment, the client application 260 also includes an image capture / scanning module 270, which is configured to capture and process images (e.g., color images, depth images, etc.) or digital scans. In one embodiment, the images or digital scans are transmitted to the back-end server system 110 or to another external computer system where digital skin models are generated. Alternatively, the digital models are generated at the client computer device 104 or at another location. In one embodiment, the digital models include 3D topology and texture information, which can be used to reproduce an accurate representation of the structure and overall appearance of the user's face, as well as for skin diagnosis (e.g., to detect imperfections, areas of hyperpigmentation, visible pores, etc.).In one embodiment, user interface 276 includes user interface elements to help accurately capture the digital images or scans on which these digital skin models are based, such as graphical guides to center the user's face in a self-portrait photo, visual or audio reminders to adjust ambient lighting or keep the camera steady, or the like.

[0055] In one embodiment, a communication module 278 of the client application 260 is used to prepare information to be transmitted to, or to receive and interpret information from other devices or systems, such as the 106 front-end server system. This information may include captured digital images, scans or videos, skincare device settings, personalized skincare routines, user preferences, user IDs, device IDs, or the like.

[0056] Other features of the client computing devices are not shown in [Fig. 3] for ease of illustration. A description of the illustrative computing devices is provided below with reference to [Fig. 6].

[0057] The devices shown in Figures 1 to 3, or other devices used in described embodiments, can communicate with each other via a network (not shown), which may include any suitable communication technology, including, but not limited to, wired technologies such as DSL, Ethernet, fiber optics, USB, and FireWire; wireless technologies such as Wi-Fi, WiMAX, 3G, 4G, LTE, 5G, and Bluetooth; and the Internet. In general, communication between the system components of [Fig. 1] or other computing devices can occur directly or through intermediate devices.

[0058] Numerous alternatives to the arrangement disclosed and described with reference to Figures 1 to 3 are possible. For example, the functionality described as being implemented in multiple components may instead be consolidated into a single component, or the functionality described as being implemented in a single component may be implemented in multiple components shown, or in other components not shown in Figures 1 to 3. As another example, the devices in Figures 1 to 3, which are shown as including particular components, may instead include more components, fewer components, or different components without departing from the scope of the described embodiments.

[0059] In general, the word "engine," as used here, refers to logic embedded in hardware or software instructions written in a programming language, such as C, C++, COBOL, JAVA™, PHP, Perl, H™L, CSS, JavaScript, VBScript, ASPX, Microsoft .NET™, and / or similar. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be called from other engines or from themselves. In general, the engines described herein refer to logic modules that may be merged with other engines or divided into sub-engines. Engines may be stored in any type of computer-readable media or computer storage device and may be stored on and executed by one or more general-purpose computers, thereby creating a specialized computer configured to provide the engine or its functionality.

[0060] As a person skilled in the art would understand, a "data store" as described herein can be any suitable device configured to store data for access by a computing device. An example of a data store is a high-throughput, highly reliable relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technique and / or device capable of quickly and reliably providing the stored data in response to queries can be used, and the computing device can be accessed locally rather than over a network, or can be provided as a cloud service.A data store may also include data stored in an organized manner on a computer-readable storage medium, as described below. Those skilled in the art will recognize that separate data stores described herein may be combined into a single data store, and / or that a single data store described herein may be separated into multiple data stores, without departing from the scope of this disclosure.

[0061] Aspects of an illustrative user interface will now be described with reference to Figures 4A to 4D. It will be understood that this disclosure also encompasses many alternative user interface features and techniques that vary from those described below.

[0062] Figure 4A shows a screenshot of an illustrative user interface 400, according to a described embodiment. In the example shown in Figure 4A, the user interface 400 includes user interface elements in the form of a text box 410 (titled "Ask me a question") in which a user can enter statements or questions and a skin analysis request element (for example, a button 420) to initiate a skin analysis process.

[0063] The 400 user interface also includes an online dialogue interface, which presents user input in addition to system responses. Alternatively, a voice interface or a combination of visual / textual dialogue and a voice interface can be used.

[0064] In one embodiment, user input is processed according to the logic and data flow shown in [Fig. 2]. In the example shown in [Fig. 4A], the online dialogue begins with a general greeting message entered by the user ("Hello"). In one embodiment, this greeting message is processed by the first classifier 140 by sending the input text to the LLM 130, which classifies the input as a "Normal Request." The input text is then processed by the second classifier 158 by sending the input text to the LLM 130, which then classifies it. The prompt is set to "Other." The input text is then sent to LLM 130 for a complete response to process block 172, and the LLM returns a response in the online dialog interface. ("Hello! How can I help you today?")

[0065] As shown in [Fig. 4A], the user continues the interaction with additional text. ("I want a new skincare routine.") In one embodiment, this input text is processed by the first classifier 140 by sending the prompt to the LLM 130, which classifies the input text as a "Normal Request." The input text is then processed by the second classifier 158 by sending the prompt to the LLM 130, which then classifies the input text as "Other." The input text is then sent to the LLM 130 for a complete response to process block 172, and the LLM 130 returns a response in the online dialogue interface. In the example shown in [Fig. 4A], the LLM 130 responds by offering assistance in finding a new skincare routine and requesting information about the user's skin type.In one embodiment, the LLM 130's decision to respond with a request for additional user information is guided by pre-training or prompting the LLM 130 to initially respond to skincare routine requests with a follow-up question to determine the user's skin type. As shown in [Fig. 4A], the user continues the interaction with an additional input that describes their skin type ("I have combination skin.") This input text is also categorized as "Normal Request" and "Other," and the LLM 130 responds with information about combination skin and a request for information about the user's skin concerns and priorities. The LLM 130 also offers a skin analysis as a next step and reminds the user that the 420 button can be used to initiate a skin analysis process.In one embodiment, the LLM 130's decision to respond in this manner is guided by pre-training or inviting the LLM 130 to initially respond to skin type statements with a follow-up question concerning skin problems and / or a suggestion to initiate a skin analysis process.

[0066] Figure 4B shows another illustrative screenshot of the user interface 400, according to a described embodiment. In the example shown in Figure 4B, the user has activated button 420, which performs a skin analysis process. Such a process may include asking the user to take a new self-portrait or to select a previously captured photo from a photo library. The self-portrait is analyzed by facial scanning software (for example, the skin analysis engine 112) to generate a digital model of the user's face. In a mode of In one embodiment, the digital model includes clinical signs of aging, skin problems, and skin characteristics and features. In one embodiment, the characteristics and features include imperfections, areas of hyperpigmentation, texture or wrinkles, or other features or combinations thereof.

[0067] In the example shown in [Fig. 4B], the back-end server system transmitted the skin analysis results to the LLM 130, which responds by summarizing these results and asking if the user agrees with the summary. In one embodiment, the LLM 130's decision to respond with a summary and request confirmation is guided by pre-training or prompting the LLM 130 to respond to skin analysis results in this manner. As shown in [Fig. 4B], the user continues the interaction with an additional input that expresses disagreement with the skin problem summary and proposes a different skin problem. ("I have redness, and that's my main skin problem.") This input text is categorized as "Normal Request" and "Other," and the LLM 130 responds with information about redness as a skin problem for combination skin.The LLM 130 also offers the option of discussing the user's skincare routine. In one embodiment, the LLM 130's decision to respond in this way is guided by prior training or prompting the LLM 130 to initially respond to skin type statements with a follow-up question regarding skin concerns and / or a suggestion to initiate a skin analysis process. The LLM 130's decision can be further guided by the user's previous request on [Fig. 4A], which can be stored in the online dialogue history, to discuss a new skincare routine.

[0068] In one embodiment, the online dialogue history is processed by the LLM 130 to enable the LLM 130 to formulate a response. Different approaches to processing the online dialogue history can be used. For example, the LLM 130 summarizes an online dialogue history and uses the summary it generates to formulate a response, but does not necessarily store the summary it generates. As another example, the user input and a summary of the LLM's response are stored in a storage area, such as a database, and this summary can be retrieved from the storage area if necessary.

[0069] As shown in [Fig. 4B], the user continues the interaction with an additional statement regarding the complexity level of the user's skincare routine. ("I would say I'm somewhere in between!"). In one embodiment, this statement is processed by the first classifier 140 and the second classifier 158 by sending the statement to the LLM 130, which classifies the statement as a "Normal Request" and "Other". Based on this, the system requests a response The LLM 130 completes the process block 172, and returns a response in the online dialogue interface. In the example shown in [Fig. 4B], the LLM 130 responds by offering assistance in finding a new skincare routine. In this example, instead of requesting information about the user's skin type, as in [Fig. 4A], the LLM responds with specific product and routine recommendations. In one embodiment, the LLM 130's decision to respond in this way is guided by prior training or prompting the LLM 130 to offer product and / or routine recommendations once information about the user's skin concerns, skin type, and skin condition has been obtained.In one embodiment, the back-end server system 110 retrieves product information from the vector database 120 and sends the product information to the LLM 130 before the LLM offers product recommendations, in order to improve the likelihood of accurate recommendations and reduce the risk of hallucination by the LLM 130.

[0070] Figure 4C shows another illustrative screenshot of the user interface 400, according to a described embodiment. In the example shown in Figure 4C, the LLM 130 asks the user for their opinion on the recommended skincare products and routine. ("What do you think? Do you have any questions or concerns about any of these products?") The user interface 400 also features a product information user interface element 430, which is configured to allow the user to view information corresponding to recommended products. In the example shown in Figure 4C, the product information user interface element 430 is configured to allow the user to scroll (e.g., horizontally or vertically) through icons corresponding to the recommended products.In one embodiment, user interaction with these icons (e.g., clicking or tapping) results in the display of product pages with additional information (e.g., ingredients, reviews, pricing information, purchase information, etc.). In one embodiment, specific product recommendations by the LLM 130 are guided by incorporating a representation of user information obtained so far (e.g., skin analysis results, skin concerns) and comparing the resulting incorporation with previously stored incorporations representing products in a vector database (e.g., vector database 120). The recommendations can be further guided by pre-training or prompting the LLM 130 to restrict product recommendations, for example, by limiting recommended products to a particular brand.

[0071] As shown in Figure 4C, the user continues the interaction with a further statement requesting the addition of a serum. (“I would like you to add a serum.”) In this example, the LLM 130 responds with specific product and routine care recommendations that include a serum. In one embodiment, the LLM 130's decision to respond in this way is guided by pre-training or prompting the LLM 130 to update the product and / or routine recommendations after an initial set of products and / or routine recommendations has been presented to the user.

[0072] Figure 4D shows another illustrative screenshot of user interface 400, according to a described embodiment. In the example shown in Figure 4D, the user continues the interaction with additional input text requesting a discussion about makeup. ("Could you help me with a makeup question?"). In one embodiment, this question is processed by the first classifier 140 and the second classifier 158 by sending the input text to the LLM 130, which classifies the input text as a "Normal Request" and "Other". The system requests a complete response from process block 172, and the LLM 130 returns a response in the inline dialogue interface. ("Of course! What would you like to discuss?"). The user continues the interaction with an additional statement. ("I'd like to get KJ's iconic red lip").In one embodiment, this statement is processed by the first classifier 140 and the second classifier 158 by sending the statement to the LLM 130, which classifies the prompt as a "Normal Request" and "Other." The system requests a complete response from process block 172, and the LLM 130 returns a response in the online dialog interface. In the example shown in [Fig. 4D], the LLM 130 responds by offering assistance in achieving the desired style and providing specific product recommendations. In one embodiment, the LLM 130's decision to respond in this way is guided by prior training or prompting the LLM 130 to provide product and / or routine recommendations once information about the user's skin concerns, skin type, and skin condition has been obtained.The user interface 400 also features a content selection user interface element 440, which is configured to allow the user to view content (e.g., videos or websites) that relates to the desired style, for example, by horizontally scrolling through icons corresponding to the content. In one embodiment, the user's interaction with these icons (e.g., by clicking or tapping) results in the playback of videos (e.g., in a dedicated video player or web browser) or the display of other content such as websites. In one embodiment, the specific product and content recommendations by the LLM 130 are oriented. by incorporating a representation of user information that has been obtained so far (e.g., interest in a product, interest in a celebrity) and comparing the resulting incorporation with previously stored incorporations representing products or content in a vector database (e.g., vector database 120). Recommendations can be further targeted by pre-training or prompting the LLM 130 to restrict product or content recommendations, for example, by limiting recommended products to a particular brand or content recommendations to a particular content provider.

[0073] Figure 5 is a flowchart illustrating an example of an embodiment of a method for using an LLM to obtain responses to user input on beauty topics. The method 500 is implemented by a computer system. The method 500 can be implemented by one or more components of the system 100 shown in Figure 1, such as the back-end server system 110, or by another computer device or system.

[0074] From a starting block, process 500 proceeds to block 502, where the computer system receives user input. In some embodiments, the user input is transmitted by a user via a user interface presented on a client computer device, such as user interface 400. The user input can be obtained in various ways, for example, by obtaining text typed into a text input field or by translating speech input into text. Process 500 then proceeds to process block 504, where the computer system obtains contextual information for the user input. In some embodiments, the contextual information is given in the form of prompts to provide context for the question or statement, such as special definitions or constraints (e.g., classification definitions, product recommendation constraints, etc.).) to be taken into account by an LLM, user profile information (e.g., user preferences, demographic data, skin characteristics, products used, etc.), summaries of past conversations, or other contextual information. Process 500 passes to process block 506, where the computer system transmits the user input and contextual information to the LLM. In some embodiments, the computer system transmits user input obtained directly from a user interface (e.g., text input from a text input field). In some embodiments, the computer system transmits user input that has been modified or translated into a different form (e.g., text generated from speech input).

[0075] Process 500 proceeds to process block 508, where the computer system requests the LLM to confirm that the user input relates to one or more beauty themes. Process 500 then proceeds to process block 510, where the computer system receives confirmation that the user input relates to the beauty theme(s). In some embodiments, the request to the LLM to confirm that the user input relates to the beauty theme(s) includes a request to the LLM to produce one or more classifications of the user input. In some embodiments, receiving confirmation that the user input relates to the beauty theme(s) includes receiving the classification(s) of the user input from the LLM. In such embodiments, the one or more classifications indicate that the user input relates to one or more beauty themes.In some embodiments, the request to the LLM to produce one or more classifications of the user input includes requesting the LLM to produce a first classification of the user input, receiving the first classification of the user input from the LLM, in which the first classification indicates that the user input relates to one or more beauty themes, and in response to the first classification indicating that the user input relates to one or more beauty themes, requesting the LLM to produce a second classification of the user input that further defines the one or more beauty themes. In an illustrative scenario, the request to the LLM to provide the response to the user input is based on both the first and second classifications.

[0076] Process 500 proceeds to process block 512 where, based on the confirmation, the computer system requests the LLM to provide a response related to the beauty theme(s) and based on user input and contextual information. In process block 514, the computer system receives the response from the LLM, and in process block 516, the computer system triggers the presentation of the response to the user via the user interface. The response may include text, graphic icons, links, videos, images, audio such as a synthesized voice, other information, or combinations thereof. In some embodiments, the response includes text presented to the user in an online dialogue interface, one or more product recommendations, one or more content recommendations, one or more skincare routine recommendations, or a combination thereof.Such recommendations can be presented in the form of text, graphic icons, links, videos, images, or similar formats.

[0077] Figure 6 is a schematic diagram illustrating aspects of an example of a 600 computer device suitable for use with modes of implementation of this disclosure. While [Fig. 6] is described with reference to a computing device that is implemented as a device on a network, the description below is applicable to servers, personal computers, mobile phones, smartphones, tablets, embedded computing devices, and other devices that may be used to implement portions of embodiments of this disclosure. Furthermore, those skilled in the art and others will recognize that the computing device 600 could be any one of a number of devices currently available or yet to be developed.

[0078] In its simplest configuration, the computing device 600 includes at least one processor 602 and a system memory 604 connected by a communication bus 606. Depending on the exact configuration and type of device, the system memory 604 may be volatile or non-volatile, such as read-only memory (“ROM”), random-access memory (“RAM”), EEPROM, flash memory, or similar memory technology. Those skilled in the art and others will recognize that the system memory 604 typically stores data and / or program modules that are immediately accessible and / or currently being used by the processor 602. In this respect, the processor 602 can serve as the computing center of the computing device 600 by handling the execution of instructions.

[0079] As illustrated in more detail in [Fig. 6], the computing device 600 may include a network interface 610 comprising one or more components for communicating with other devices on a network. Embodiments of this disclosure may access basic services that use the network interface 610 to perform communications using common network protocols. The network interface 610 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, LTE, 5G, WiMAX, Bluetooth, and / or similar protocols.

[0080] In the example embodiment shown in [Fig. 6], the computer device 600 also includes a storage medium 608. However, it is possible to access the services using a computer device that does not include means for data persistence on a local storage medium. Therefore, the storage medium 608 shown in [Fig. 6] is represented by a dashed line to indicate that the storage medium 608 is optional. In any case, the storage medium 608 may be volatile or non-volatile, removable or non-removable, implemented using any technology capable of storing information such as, but not limited to, a hard disk drive, an SSD, a CD-ROM, a DVD or any other disk storage medium, magnetic cassettes, magnetic tape, magnetic disk storage medium and / or the like.

[0081] As used herein, the term "computer-readable media" includes volatile and non-volatile, removable and non-removable media implemented in any process or technology capable of storing information, such as computer-readable instructions, data structures, program modules, or other data. In this respect, the system memory 604 and storage medium 608 shown in [Fig. 6] are only examples of computer-readable media. In one embodiment, computer-readable media are used to store data for use by programs.

[0082] Suitable implementations of computing devices including a processor 602, system memory 604, a communication bus 606, a storage medium 608, and a network interface 610 are known and commercially available. For ease of illustration and because it is not important for understanding the claimed subject matter, [Fig. 6] does not show some of the typical components of many computing devices. In this regard, the computing device 600 may include input devices, such as a keyboard, numeric keypad, mouse, microphone, touch input device, touchscreen, tablet, and / or the like. These input devices may be coupled to the computing device 600 by wired or wireless connections, including RF, infrared, serial, parallel, Bluetooth, USB, or other suitable connection protocols using wireless or physical connections.Similarly, the 600 computer device may also include output devices, such as a display, speakers, a printer, etc. Since these devices are well-known in the arts, they are not illustrated or described in further detail here. Extensions and alternatives

[0083] Many alternatives to the systems and devices described herein are possible. For example, individual modules or subsystems may be separated into additional modules or subsystems or combined into fewer modules or subsystems. As another example, modules or subsystems may be omitted or supplemented by other modules or subsystems. As yet another example, functions that are indicated as being performed by a particular device, module, or subsystem may instead be performed by one or more other devices, modules, or subsystems. Although some examples in this disclosure include descriptions of devices comprising specific hardware components in specific arrangements, the techniques and tools described herein may be modified to accommodate different hardware components, combinations, or arrangements.Furthermore, while some examples in this disclosure include descriptions of specific use cases, the techniques and tools described herein can be modified to suit different use cases. A feature that is described as being implemented in software may instead be implemented in hardware, or vice versa.

[0084] Numerous alternatives to the techniques described herein are possible. For example, the processing steps in the various techniques can be separated into additional steps or combined into fewer steps. As another example, processing steps in the various techniques can be omitted or supplemented by other techniques or processing steps. As another example, processing steps described as occurring in a particular order can instead occur in a different order. As yet another example, processing steps described as being carried out in a series of steps can instead be processed in parallel, with several software modules or processes simultaneously processing one or more of the illustrated processing steps.

[0085] Many alternatives to the user interfaces described herein are possible. In practice, the user interfaces described herein can be implemented as separate user interfaces or as different states of the same user interface, and the different states can be presented in response to different events, for example, user input events. The user interfaces can be customized for different devices, different input and output capabilities, and the like. For example, user interfaces can be presented in different ways depending on the display size, display orientation, whether the device is a mobile device or not, and so on. The information and user interface elements displayed in the user interfaces can be modified, supplemented, or replaced by other elements in various possible implementations.For example, various combinations of graphical user interface elements, including text boxes, sliders, drop-down menus, radio buttons, programmable keys, etc., or any other user interface element, including hardware elements such as buttons, switches, scroll wheels, microphones, cameras, etc., can be used to accept user input in various forms. As another example, the user interface elements used in a particular implementation or configuration may depend on whether a device has specific input and / or output capabilities (e.g., a touchscreen). Information and user interface elements can be presented in different spatial, logical, and temporal arrangements across various possible implementations.For example, information or user interface elements shown as being presented simultaneously on a single page or screen may also be presented at different times, on different pages or screens, etc. As another example, certain information or certain... User interface elements can be presented conditionally based on previous input, user preferences, or similar factors.

[0086] Although illustrative embodiments have been shown and described, it should be appreciated that various changes can be made to them without departing from the spirit and scope of the invention.

Claims

Demands

1. A non-transient, computer-readable medium on which are stored instructions configured to, when executed by one or more computing devices of a computer system, cause the computer system to perform operations comprising: - transmitting user input and contextual information for the user input to a large language model (LLM); - requesting the LLM to provide confirmation that the user input relates to one or more beauty themes; - receiving confirmation from the LLM that the user input relates to one or more beauty themes; - on the basis of the confirmation, requesting the LLM to provide a response to the user input to be presented to a user, wherein the response relates to one or more beauty themes and is based at least in part on the user input and contextual information;and - receiving the LLM's response.

2. A computer-readable medium according to claim 1, wherein the request to the LLM to provide confirmation that the user input relates to one or more beauty themes includes the request to the LLM to produce one or more classifications of the user input, wherein the receipt of confirmation that the user input relates to one or more beauty themes includes the receipt of one or more classifications of the user input from the LLM, and wherein the one or more classifications indicate that the user input relates to one or more beauty themes.

3. A computer-readable medium according to claim 2, wherein the request to the LLM to produce one or more classifications of the user input comprises: - the request to the LLM to produce a first classification of the user input; - the receipt of the first classification of the user input from the LLM, wherein the first classification indicates that the user input relates to one or more beauty themes; and - in response to the first classification indicating that the user input relates to one or more beauty themes, the request to the LLM to produce a second classification of the user input that further defines the one or more beauty themes, - in which the request to the LLM to give the response to the user input is based on both the first classification and the second classification.

4. A computer-readable medium according to claim 1, wherein the user input includes text input or voice input.

5. Computer-readable media according to claim 1, operations further comprising requesting the LLM to produce a summary of the user input.

6. Computer-readable media according to claim 5, operations further comprising: - receiving the summary of user input from the LLM; - generating a vector representation of the user input.

7. A computer-readable medium according to claim 6, operations further comprising: - comparing the vector representation of the user input with vector representations of other vector representations in a vector database; and - identifying a near neighbor match for the vector representation of the user input among the other vector representations in the vector database.

8. Computer-readable media according to claim 1, operations further comprising: - obtaining a digital model of the user's face; and - requesting the LLM to generate a product recommendation or a skincare routine recommendation based at least in part on the digital face model.

9. A computer-readable medium according to claim 8, wherein the digital face model includes a plurality of skin features including information on imperfections, information on hyperpigmentation, clinical signs, information on skin problems, information on skin texture, information on complexion, or any combination thereof.

10. A computer-implemented method comprising, by a computer system: - transmitting user input and contextual information for the user input to a large language model (LLM); - requesting the LLM to provide confirmation that the user input relates to one or more beauty themes; - receiving confirmation from the LLM that the user input relates to one or more beauty themes; - on the basis of the confirmation, requesting the LLM to provide a response to the user input to be presented to a user, wherein the response relates to one or more beauty themes and is based at least in part on the user input and contextual information; and - receiving the response from the LLM.

Images