system

Abstract

We provide the system. [Solution] A face recognition means for acquiring a user's face image and analyzing the face image to generate face feature information, A simulation means for applying the color and texture of an accessory selected based on the facial feature information to the user's face in real time, A display means for displaying the simulation results so that the user can visually confirm them, A system that includes this.

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, the problems faced by individual users when selecting the most suitable ornaments for themselves are that the opportunity for trial is limited and they feel dissatisfied with the finish different from their expectations after actually purchasing the product. There are also limitations in trying out products in stores and difficulties in making purchase decisions during busy lives. In such a situation, there is a demand for a system that allows users to efficiently and personally try out ornaments in a short time and make a selection after being convinced. [[ID=!6]]

Means for Solving the Problems

[0006] A "user" refers to an individual who uses the system to take a facial image and try on accessories.

[0007] "Face image" refers to image data of a user's face.

[0008] "Facial recognition means" refers to a function that analyzes a user's facial image and extracts information such as facial features, shape, and color.

[0009] "Facial feature information" refers to data obtained by facial recognition methods regarding the shape of the user's face and skin tone.

[0010] "Decorative items" refer to cosmetics such as lipstick, eyeshadow, and foundation that are applied to the user's face.

[0011] "Simulation means" refers to a method of applying the visual effects of selected accessories to a user's facial image based on facial feature information and displaying them in real time.

[0012] "Display means" refers to a device or function for providing the user with images generated by the simulation means in a visual manner.

[0013] "Communication means" refers to the function for sending and receiving data between the user's terminal and the server.

[0014] "Suggestion method" refers to the function of a system that selects and suggests suitable accessories to a user based on facial feature information. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiment for Implementing the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the language used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides a makeup simulation system that allows users to visually confirm realistic results without having to try individual cosmetic items at home or in a store. Embodiments of this system are realized through a series of steps in which the user acquires a facial image via a terminal, and the server analyzes and simulates that image. Specific embodiments of this invention will be described below.

[0037] User actions and terminal functions

[0038] The user first launches a dedicated application installed on their device. The application provides a function to take a picture of the user's face with the camera and record the facial image. After taking the picture, the user selects an item of interest from a list of decorative items displayed on the screen. The selected information is sent from the device to the server in real time.

[0039] Server Processing

[0040] The server receives facial images transmitted from the terminal and analyzes facial features using a facial recognition algorithm. Specifically, it extracts feature information such as facial shape, skin tone, and the position of the eyes and lips. Based on this information, it optimizes the color and texture of selected accessories to match those features and performs a simulation to apply them to the user's face.

[0041] Displaying the simulation results

[0042] The simulation results generated on the server are transmitted to the terminal in real time and displayed on the user's screen. The terminal continues to display the simulation results seamlessly, even as the user's face moves dynamically. This process allows the user to visually confirm what the finished look would be like with actual makeup applied, supporting their pre-purchase decision.

[0043] Specific example

[0044] If a user wants to try red lipstick, they take a picture of their face using the device and select red lipstick. The server then simulates applying the lipstick color to the user's face and sends the result back to the device. The device displays the red lipstick on the user's face in real time, allowing the user to check the result and make a purchase decision based on their satisfaction.

[0045] Thus, the system of the present invention provides end users with a personalized makeup experience, enabling them to make smarter and more effective choices before purchasing.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The user launches an application on their device and takes a picture of their face with the camera. The device then prepares to send the captured face image to the server.

[0049] Step 2:

[0050] The device transmits captured facial image data to the server in real time. The image data is compressed before transmission to ensure efficient data transfer.

[0051] Step 3:

[0052] The server analyzes the received facial image data and extracts the user's facial features using a facial recognition algorithm. This includes contour detection, skin tone analysis, and identification of the location of major facial features.

[0053] Step 4:

[0054] The user selects an item they want to try from a list of accessories displayed on the device screen. The device then sends the user's selection information to the server.

[0055] Step 5:

[0056] The server combines the user's facial feature information with data on selected accessories and starts the simulation. During the simulation, deep learning technology is used to apply the selected colors and textures to the facial image in a realistic way.

[0057] Step 6:

[0058] The server sends the generated simulation results to the terminal. This data is optimized to provide a smooth and natural transition for the user's visual perception.

[0059] Step 7:

[0060] The device displays the received simulation results on the screen, allowing the user to check them in real time. The user can evaluate how the virtually applied makeup looks on them and use this information to help them make their selection.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] The challenge is to provide a system that allows users to realistically see the effects of cosmetics on their own faces without actually trying them. This would enable users to visually confirm the results before purchasing cosmetics, allowing them to make more effective purchasing decisions.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for acquiring a facial image via an application installed on a device used by the user, processing means for analyzing the facial image and generating facial feature information, and simulation means for applying the color and texture of selected cosmetics to the user's face in real time based on the facial feature information. This allows the user to check the finished look of the cosmetics in real time.

[0066] "Applications installed on devices used by the user" refers to programs launched on electronic devices operated by the user, and are means of acquiring facial images and providing interfaces.

[0067] "Facial feature information" refers to information that quantifies or digitizes features such as shape, skin tone, and the position of eyes and lips, which are analyzed from a user's facial image.

[0068] A "simulation method" is a method or process that applies selected cosmetics to a user's facial image based on analyzed facial feature information and visually simulates the results.

[0069] "Display means" refers to digital screens or output devices that provide users with simulated results and enable them to dynamically verify them.

[0070] "Communication means" refers to network equipment and communication protocols used to send and receive data between a user's device and a server.

[0071] A "guidance tool" refers to a system or function that provides suggestions and recommendations to help users select the most suitable cosmetics based on their facial features.

[0072] This invention begins with the user acquiring a facial image using a dedicated application installed on a given device. The device uses its camera to capture the user's face and temporarily stores the image data.

[0073] The captured facial image, along with information about the cosmetics selected by the user, is sent to the server. The server processes the facial image using image analysis technology and extracts facial feature information. This process includes algorithms that quantify characteristics such as facial shape, skin tone, and the position of the eyes and lips.

[0074] Based on the facial feature information obtained, the server optimizes the color and texture of the selected cosmetics and performs a simulation. This allows the user to visually confirm the finished look when actually using the cosmetics.

[0075] The simulation results are transmitted to the device in real time, and the user can view them on the device's display. Dynamic display technology allows the results to be displayed seamlessly even if the user's face moves.

[0076] For example, if a user wants to try red lipstick, they simply take a picture of their face with their device and select red lipstick in the application. The server analyzes the facial features, performs a simulation with the red lipstick applied, and returns the results to the device. The user can then see how the red lipstick looks on their face on the screen and decide whether to purchase it.

[0077] An example of a prompt would be: "Describe the process in which a user takes a photo of their face to try on red lipstick, and the server analyzes the photo and performs a simulation."

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user launches an application installed on their device. The user then uses the application's camera function to capture a photo of their face. The input is the user's face, and the output is a digital image temporarily stored on the device.

[0081] Step 2:

[0082] The terminal provides an interface for recording the accessories selected by the user. The user provides information about the selected items as input. As output, the accessory selection information is generated and ready to be sent to the server.

[0083] Step 3:

[0084] The device sends facial images and accessory selection information to the server. The input is the facial image and accessory information stored on the device, and the output is the dataset that arrives on the server. Communication technology is used to ensure that the data is transferred quickly and securely.

[0085] Step 4:

[0086] The server starts the process of analyzing the received face image and extracting facial features. The input is the transmitted face image, and the output is facial feature information including face shape, skin tone, and the positions of the eyes and lips. By using a face recognition algorithm, the necessary data is extracted quickly and accurately.

[0087] Step 5:

[0088] The server performs simulations based on facial feature information and accessory selection data. The inputs are facial feature information and accessory information, and the output is a simulated image of the user's face with the accessories applied. A generative AI model is used to reproduce realistic textures and colors.

[0089] Step 6:

[0090] The server transmits the simulation results to the terminal in real time. The input is the generated simulation image, and the output is image data for display on the user's terminal. The results are displayed to the user without interruption and are dynamically updated.

[0091] Step 7:

[0092] The user views the simulation results on the device's display. The presented results allow the user to visually confirm the finished look of the makeup and use them as a basis for making a purchase decision.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] The challenge is to provide a means for users to accurately simulate and visually confirm the suitability of makeup and accessories in a virtual environment, something that is difficult to do realistically at home or in a regular store. With current technology, it is difficult to provide personalized visual feedback that adapts to the user's preferences in real time.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes face recognition means for acquiring and analyzing a user's facial image, simulation means for applying accessories in real time based on facial feature information, and virtual reality representation means for enabling trial use in a virtual environment. This allows users to check the finished look of accessories in real time and make the optimal choice while at home or in a virtual store.

[0098] A "user" refers to an individual who uses the system to take a facial image and perform a simulation of how accessories would look on them.

[0099] "Face image" refers to image data obtained by capturing a user's face using a camera or other imaging device.

[0100] "Facial feature information" refers to characteristic data such as skin tone, facial shape, and the position of eyes and lips, which are analyzed from facial images.

[0101] "Decorative items" refer to elements that change a user's appearance by being applied to their face, such as makeup products and accessories.

[0102] "Simulation means" refers to the process of applying accessories in real time based on facial feature information and generating the results.

[0103] "Display means" refers to devices or interfaces that visually provide simulation results to the user.

[0104] A "virtual environment" refers to a digital space that utilizes virtual reality and augmented reality to allow users to experience simulations in real time.

[0105] "Virtual reality representation means" refers to technologies that generate realistic visual experiences in a virtual environment for trying on decorative items.

[0106] To implement this system, users first launch a dedicated application using a device such as a smartphone or smart glasses to acquire a facial image. The acquired facial image is then transmitted from the device to the server via communication. The server analyzes facial feature information based on this facial image using software libraries such as OpenCV and TENSORFLOW®. This analysis extracts data such as skin tone, facial shape, and the position of the eyes and lips.

[0107] Next, the user selects information about decorative items they are interested in within the app. This information is sent to a server, which then performs a simulation based on the characteristics of the selected items. Cloud services such as AWS® are used in this process, and data is processed in real time. The simulation results are sent to the terminal via virtual reality representation and presented to the user visually.

[0108] Users can visually see how decorative items will match within this virtual environment. This feature allows users to simulate the finished look without actually handling the product, enabling them to make smarter purchasing decisions.

[0109] For example, if a user wants to try a specific color of eyeshadow, they can select that color and instantly see the result through the application. The system is optimized so that users can realistically see the color and texture under any lighting conditions. This technology can also be used to prompt a generative AI model and instruct it to try out specific color and texture combinations of accessories. For instance, by using a prompt such as, "Receive a facial image and apply the specified lipstick color. Make the effect visible in real time," the system can provide the user with the most suitable simulation.

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The user launches a dedicated application on their device to acquire a facial image. The input is a facial image of the user, which is captured by the device's camera. The camera recognizes the face and automatically adjusts the focus and exposure to take the picture. The output is the captured facial image data.

[0113] Step 2:

[0114] The terminal sends the acquired facial image data to the server. The input is the facial image data obtained in step 1. This is uploaded to the server via internet communication. The output is the facial image data received on the server.

[0115] Step 3:

[0116] The server applies a face recognition algorithm to the received face image data and analyzes the facial feature information. The input is face image data, and the output is the analyzed facial feature information. The server uses libraries such as OpenCV and TensorFlow to identify the shape of the face, the position of the eyes, and the skin tone.

[0117] Step 4:

[0118] The user selects an accessory within the application and sends that information to the server. The input is the user's selected accessory information, and the output is the accessory identification information received by the server. The user interacts with the application by browsing a list on the screen and tapping the item they want to try.

[0119] Step 5:

[0120] The server performs simulations based on facial feature information and accessory information. The inputs are facial feature information and accessory information. The output is the simulation result, i.e., an image of the face after the accessories have been applied. The server utilizes cloud infrastructure such as AWS to perform calculations that apply texture and color in real time.

[0121] Step 6:

[0122] The server sends the simulation results to the terminal and displays them to the user. The input is the simulation result data, and the output is the user's face image displayed on the terminal's screen. The terminal's display shows the results in full color, and the screen continues to update smoothly even as the user moves.

[0123] Step 7:

[0124] The user visually reviews the displayed results and considers purchasing the accessories. The input is a facial image displayed on the device's screen. The output is the user's purchase intentions and preferences, and the user adds the accessories to their cart as needed. An example of a prompt used to query the generating AI model is, "Receive the facial image and apply the specified lipstick color. Allow real-time confirmation of the effect."

[0125] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0126] This invention provides a makeup simulation system equipped with advanced personalization capabilities that utilize emotion recognition. The system aims to understand the user's emotions from their facial expressions and select and apply the most appropriate accessories accordingly. Specific embodiments are described in detail below.

[0127] User interaction and emotion engine functionality

[0128] The user launches a dedicated application on their device and takes a picture of their face. The device sends this image to a server, which extracts facial feature information using facial recognition technology. Furthermore, an emotion engine simultaneously analyzes the user's facial expressions and recognizes their current emotional state. This recognition result is used as foundational information for user customization.

[0129] Server processing and application of decorations

[0130] The server combines facial feature information and emotional information to select accessories suitable for the user. The data for the selected accessories is adjusted in terms of color and texture according to the user's emotional state, enabling a more personalized simulation. For example, if the server determines that the user is depressed, accessories with bright colors that will lift their spirits will be recommended.

[0131] Displaying the simulation results

[0132] The simulation results with the adjusted decorations applied are transmitted to the terminal in real time and displayed for the user to visually confirm. The terminal interface is designed to provide natural and continuous feedback even as the user moves, allowing for the selection and adjustment of decorations while viewing the results.

[0133] Specific example

[0134] For example, suppose a user takes a selfie at home using their device. If the system detects that the user is smiling, it will suggest a blue eyeshadow to give the user a refreshing image. The simulation results then show the blue eyeshadow being applied to the user's face in real time, which can be viewed on the device. Based on these results, the user can select a product.

[0135] Thus, the present invention takes into account the user's real-time emotions and enables a makeup experience tailored to individual needs. This system is intended to enhance personalization through emotion recognition technology and improve user satisfaction.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The user launches an application on their device and takes a picture of their face using the camera. The device acquires this face image and immediately prepares to send it to the server.

[0139] Step 2:

[0140] The device sends the acquired facial image data to the server. This data transfer is optimized for high speed by compressing the images.

[0141] Step 3:

[0142] The server receives the facial image sent from the terminal and applies a facial recognition algorithm to extract facial feature information. This process includes identifying skin color, face shape, and the positions of the eyes, nose, and mouth.

[0143] Step 4:

[0144] The server uses an emotion engine in addition to facial image analysis results to identify the user's current emotional state from their facial expressions. The emotion engine recognizes emotions such as smiling, surprise, and sadness in real time.

[0145] Step 5:

[0146] The user selects an item they want to try from a list of accessories presented on the device's interface. The device then sends the selected accessory data and the user's emotional state to the server.

[0147] Step 6:

[0148] The server adjusts the color and texture of selected accessories based on facial feature information and emotional information. For example, if the user is feeling happy, colors that emphasize freshness will be applied. This adjustment data is then input into the simulation system.

[0149] Step 7:

[0150] The server generates simulation results, which are then sent to the terminal. The terminal displays a real-time, adjusted simulation of the adornment on its screen. This allows the user to instantly visualize and confirm the virtual makeup applied to their face.

[0151] Step 8:

[0152] The user selects and further adjusts decorative items based on the displayed simulation results. When the user makes selections or changes, the terminal sends the relevant data back to the server to obtain the latest simulation results.

[0153] This entire process allows users to try on personalized accessories in real time, supporting flexible, emotion-based selections.

[0154] (Example 2)

[0155] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0156] Conventional makeup simulation systems primarily select accessories based on the user's static facial information, and suffer from the problem of insufficient personalization that takes into account the user's emotional changes. Furthermore, real-time adjustment and display of accessories are difficult, making it challenging to improve the user experience.

[0157] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0158] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate feature information; application means for selecting an accessory based on the feature information and emotional information, and for adjusting its color and texture; and display means for simulating and displaying the adjusted accessory on the user's face in real time so that the user can confirm it. This makes it possible to select a more highly personalized accessory and provide real-time feedback, taking into account the user's emotional state.

[0159] A "user" refers to an individual who uses the makeup simulation system by providing a facial image.

[0160] "Facial image" refers to digital image data of the user's face acquired by the device for the purpose of performing makeup simulations.

[0161] "Feature information" refers to user-specific physical characteristics and shape information extracted from facial images.

[0162] "Emotional information" refers to data about a user's emotional state obtained by analyzing their facial expressions.

[0163] "Recognition means" refers to a processing device or program for analyzing facial images and extracting feature information from them.

[0164] "Application means" refers to a processing device or program for selecting ornaments based on characteristic information and emotional information, and for adjusting the parameters of those ornaments.

[0165] "Display means" refers to a device or interface for simulating and displaying an adjusted ornament on a user's face and providing the user with the result.

[0166] "Accessories" refers to data related to cosmetics and accessories applied to the user in the makeup simulation.

[0167] "Real-time" refers to a state where processing and display occur almost instantaneously in response to user actions.

[0168] This invention provides a system that simulates optimal makeup based on the user's emotions. The user first launches a dedicated application on their device and takes a facial image using the camera function. This device is assumed to be a typical smartphone or tablet. The facial image taken by the user is sent from the device to the server. This data transmission is performed via HTTPS to ensure security.

[0169] When the server receives a face image, it first activates a recognition mechanism to extract facial feature information from the image. Here, open-source image processing libraries such as OpenCV and dlib are used to obtain the contours of the face and the positional information of its features. Next, an emotion engine is activated to obtain emotional information, and machine learning frameworks such as TensorFlow and PyTorch are used to analyze facial expressions. This makes it possible to recognize the emotional state of the user, such as whether they are smiling or frowning.

[0170] Based on the characteristic and emotional information obtained, the server selects the most suitable ornament and adjusts its color and texture in real time. MongoDB and MySQL (registered trademark) are used as databases to search and retrieve information about ornaments. During the adjustment phase, color adjustments are made using image editing libraries such as Pillow, and the suggested ornaments change according to the user's emotions.

[0171] The simulation results of the adjusted ornaments are sent from the server to the terminal and displayed in the user interface. This interface is built on a cross-platform framework such as React Native and allows for real-time feedback. Users can further fine-tune the displayed simulation results.

[0172] For example, if a user takes a selfie and smiles, and the emotion recognition system determines that the user is "happy," the server will suggest a pink blush that gives the user a warm impression. The simulation results are applied to the user's face, and the effect can be viewed on the device. In this way, the user receives an optimal makeup experience tailored to their emotions.

[0173] An example of a prompt for a generative AI model is "How to design a makeup simulation system that responds to emotional states," and this prompt can yield results that align with the system's objectives.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The user launches a dedicated application on their device and uses the camera function to take a picture of their face. When the user presses the capture button, the face image is input to the device. The device converts the acquired face image data into JPEG or PNG format and prepares it for transmission to the server. The output of this step is the face image data to be sent to the server.

[0177] Step 2:

[0178] The server receives face image data sent from the terminal. Next, it analyzes this face image using an image processing library to extract facial feature information. This process executes a face recognition algorithm to identify the positions of the eyes, nose, and mouth within the image. The input is face image data, and the output is feature information regarding the shape and position of the user's face.

[0179] Step 3:

[0180] An emotion engine operates within the server, recognizing the user's emotional state based on facial feature information. A machine learning model is used for emotion recognition, performing facial expression analysis. During this process, the facial recognition results are input into an existing model, and emotional information is output. This output information includes emotion labels such as "happy" and "surprised."

[0181] Step 4:

[0182] The server integrates facial feature information and emotional information, and based on this, selects the most suitable accessory for the user from the database. After selection, it adjusts the color and texture of the accessory to match the emotion. The input is facial feature information and emotional information, and the output is data of the adjusted accessory. The database used here contains information on the type, color, and texture of the accessory.

[0183] Step 5:

[0184] The server sends the adjusted accessory data to the terminal. The terminal generates simulation results on the user interface based on the received data and displays them to the user. Real-time processing allows the display to change smoothly in response to the user's movements. The input is the accessory data, and the output is the simulation display on the terminal.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] Currently, providing personalized makeup experiences tailored to users' emotional states is challenging in both physical beauty stores and online platforms. In particular, systems that offer real-time makeup suggestions based on each customer's current emotions to improve satisfaction are still insufficient. To address these challenges, a system is needed that recognizes user emotions and selects and suggests cosmetics best suited to that emotional state.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate biometric information; simulation means for applying the properties of selected cosmetics to the user's face in real time based on the biometric information and emotional information; and display means for displaying the simulation results so that the user can visually confirm them. This enables personalized makeup suggestions and simulations that correspond to the user's emotional state.

[0190] A "user" refers to a person who uses the system to experience the makeup simulation.

[0191] "Face image" refers to digital image data of a user's face.

[0192] "Biometric characteristic information" refers to data about the structure and shape of the face obtained through the analysis of facial images.

[0193] "Emotional information" refers to data about the emotional state inferred from the user's facial expressions.

[0194] "Recognition means" refers to methods and devices for acquiring a user's facial image and generating biometric information and emotional information.

[0195] "Cosmetics" refers to products that have colors and textures used for makeup purposes.

[0196] "Simulation means" refers to a method or device that applies the characteristics of selected cosmetics to a user's facial image and displays them visually.

[0197] "Display means" refers to screens or devices that allow users to check the simulation results.

[0198] "Suggestion means" refers to methods and devices for presenting the most suitable cosmetics to a user based on the user's biometric characteristics and emotional information.

[0199] This invention is a system implemented using a smart mirror installed in a physical store. Users can perform a makeup simulation by standing in front of this smart mirror.

[0200] The smart mirror is equipped with a camera, display, and network connectivity. The camera captures the user's facial image, and the acquired data is transmitted to a server via the network. The server analyzes biometric and emotional information using facial recognition and emotion recognition means, thereby understanding the user's emotional state. Based on this information, the server selects the optimal cosmetic product. The simulation results of the selected cosmetic product are displayed in real time on the smart mirror's display, providing the user with visual feedback.

[0201] If the user is smiling, the system will suggest cosmetics in colors that give a refreshing impression. In this way, users can try makeup that matches their mood.

[0202] A concrete example of a prompt would be, "Please list makeup styles suitable for a woman in her 30s whose current emotion is joy." Based on this prompt, the generative AI model creates a list of optimal cosmetics and suggests them to the user throughout the system.

[0203] This system allows users to select and experience makeup styles that match their emotional state, making their makeup choices in physical stores more personalized.

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The smart mirror on the device acquires an image of the user's face using its camera. The input is optical information of the user's face, and the output is digitized facial image data. This data is sent to a server for later analysis.

[0207] Step 2:

[0208] The server analyzes the received facial image data using facial recognition technology and extracts biometric information. The input is facial image data, and the output is characteristic data related to the structure and shape of the user's face. Here, information such as the facial contour, eyes, nose, and mouth positions is analyzed.

[0209] Step 3:

[0210] The server uses emotion recognition to obtain emotional information from facial image data. The input is facial image data, and the output is the user's current emotional state. The emotion recognition algorithm estimates the emotion (joy, sadness, surprise, etc.) from the user's facial expression. Based on this result, it determines which emotional state is strongest.

[0211] Step 4:

[0212] The server integrates biometric and emotional information to generate prompt statements for a generative AI model. The input is biometric and emotional information, and the output is a prompt statement. This prompt statement is used by the generative AI model as a criterion for selecting the most suitable cosmetics.

[0213] Step 5:

[0214] A generative AI model lists the optimal cosmetic characteristics based on the prompt text. The input is the prompt text, and the output is data about the color and texture of the cosmetics. This data is used to suggest a cosmetic style that matches the user's emotions.

[0215] Step 6:

[0216] The server applies optimized cosmetic data to the user's facial image using a simulation mechanism and generates simulation results in real time. The input is cosmetic characteristic data and the user's facial image, and the output is a virtual facial image with the cosmetics applied.

[0217] Step 7:

[0218] The smart mirror on the device displays a simulated facial image on its screen, providing the user with visual feedback. The input is the image data of the simulation result, and the output is the visual information displayed on the screen. This allows the user to see and experience suggested cosmetics in real time.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention provides a makeup simulation system that allows users to visually confirm realistic results without having to try individual cosmetic items at home or in a store. Embodiments of this system are realized through a series of steps in which the user acquires a facial image via a terminal, and the server analyzes and simulates that image. Specific embodiments of this invention will be described below.

[0236] User actions and terminal functions

[0237] The user first launches a dedicated application installed on their device. The application provides a function to take a picture of the user's face with the camera and record the facial image. After taking the picture, the user selects an item of interest from a list of decorative items displayed on the screen. The selected information is sent from the device to the server in real time.

[0238] Server Processing

[0239] The server receives facial images transmitted from the terminal and analyzes facial features using a facial recognition algorithm. Specifically, it extracts feature information such as facial shape, skin tone, and the position of the eyes and lips. Based on this information, it optimizes the color and texture of selected accessories to match those features and performs a simulation to apply them to the user's face.

[0240] Displaying the simulation results

[0241] The simulation results generated on the server are transmitted to the terminal in real time and displayed on the user's screen. The terminal continues to display the simulation results seamlessly, even as the user's face moves dynamically. This process allows the user to visually confirm what the finished look would be like with actual makeup applied, supporting their pre-purchase decision.

[0242] Specific example

[0243] If a user wants to try red lipstick, they take a picture of their face using the device and select red lipstick. The server then simulates applying the lipstick color to the user's face and sends the result back to the device. The device displays the red lipstick on the user's face in real time, allowing the user to check the result and make a purchase decision based on their satisfaction.

[0244] Thus, the system of the present invention provides end users with a personalized makeup experience, enabling them to make smarter and more effective choices before purchasing.

[0245] The following describes the processing flow.

[0246] Step 1:

[0247] The user launches an application on their device and takes a picture of their face with the camera. The device then prepares to send the captured face image to the server.

[0248] Step 2:

[0249] The device transmits captured facial image data to the server in real time. The image data is compressed before transmission to ensure efficient data transfer.

[0250] Step 3:

[0251] The server analyzes the received facial image data and extracts the user's facial features using a facial recognition algorithm. This includes contour detection, skin tone analysis, and identification of the location of major facial features.

[0252] Step 4:

[0253] The user selects an item they want to try from a list of accessories displayed on the device screen. The device then sends the user's selection information to the server.

[0254] Step 5:

[0255] The server combines the user's facial feature information with data on selected accessories and starts the simulation. During the simulation, deep learning technology is used to apply the selected colors and textures to the facial image in a realistic way.

[0256] Step 6:

[0257] The server sends the generated simulation results to the terminal. This data is optimized to provide a smooth and natural transition for the user's visual perception.

[0258] Step 7:

[0259] The device displays the received simulation results on the screen, allowing the user to check them in real time. The user can evaluate how the virtually applied makeup looks on them and use this information to help them make their selection.

[0260] (Example 1)

[0261] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0262] The challenge is to provide a system that allows users to realistically see the effects of cosmetics on their own faces without actually trying them. This would enable users to visually confirm the results before purchasing cosmetics, allowing them to make more effective purchasing decisions.

[0263] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0264] In this invention, the server includes means for acquiring a facial image via an application installed on a device used by the user, processing means for analyzing the facial image and generating facial feature information, and simulation means for applying the color and texture of selected cosmetics to the user's face in real time based on the facial feature information. This allows the user to check the finished look of the cosmetics in real time.

[0265] "Applications installed on devices used by the user" refers to programs launched on electronic devices operated by the user, and are means of acquiring facial images and providing interfaces.

[0266] "Facial feature information" refers to information that quantifies or digitizes features such as shape, skin tone, and the position of eyes and lips, which are analyzed from a user's facial image.

[0267] A "simulation method" is a method or process that applies selected cosmetics to a user's facial image based on analyzed facial feature information and visually simulates the results.

[0268] "Display means" refers to digital screens or output devices that provide users with simulated results and enable them to dynamically verify them.

[0269] "Communication means" refers to network equipment and communication protocols used to send and receive data between a user's device and a server.

[0270] A "guidance tool" refers to a system or function that provides suggestions and recommendations to help users select the most suitable cosmetics based on their facial features.

[0271] This invention begins with the user acquiring a facial image using a dedicated application installed on a given device. The device uses its camera to capture the user's face and temporarily stores the image data.

[0272] The captured facial image, along with information about the cosmetics selected by the user, is sent to the server. The server processes the facial image using image analysis technology and extracts facial feature information. This process includes algorithms that quantify characteristics such as facial shape, skin tone, and the position of the eyes and lips.

[0273] Based on the facial feature information obtained, the server optimizes the color and texture of the selected cosmetics and performs a simulation. This allows the user to visually confirm the finished look when actually using the cosmetics.

[0274] The simulation results are transmitted to the device in real time, and the user can view them on the device's display. Dynamic display technology allows the results to be displayed seamlessly even if the user's face moves.

[0275] For example, if a user wants to try red lipstick, they simply take a picture of their face with their device and select red lipstick in the application. The server analyzes the facial features, performs a simulation with the red lipstick applied, and returns the results to the device. The user can then see how the red lipstick looks on their face on the screen and decide whether to purchase it.

[0276] An example of a prompt would be: "Describe the process in which a user takes a photo of their face to try on red lipstick, and the server analyzes the photo and performs a simulation."

[0277] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0278] Step 1:

[0279] The user launches an application installed on their device. The user then uses the application's camera function to capture a photo of their face. The input is the user's face, and the output is a digital image temporarily stored on the device.

[0280] Step 2:

[0281] The terminal provides an interface for recording the ornaments selected by the user. The user provides the information of the selected item as input. As output, the selection information of the ornament is generated and is ready to be sent to the server.

[0282] Step 3:

[0283] The terminal sends the face image and the selection information of the ornament to the server. The input is the face image and the ornament information stored in the terminal, and the output is the dataset that reaches the server. By utilizing communication technology, the data is transferred quickly and securely.

[0284] Step 4:

[0285] The server starts the process of analyzing the received face image and extracting the facial features. The input is the transmitted face image, and the output is the facial feature information including the shape of the face, skin tone, and the positions of eyes and lips. By using face recognition algorithms, the necessary data is extracted quickly and accurately.

[0286] Step 5:

[0287] The server conducts a simulation based on the facial feature information and the selection data of the ornament. The input is the facial feature information and the ornament information, and the output is the simulated image of the user's face with the ornament applied. By utilizing a generative AI model, realistic textures and colors are reproduced.

[0288] Step 6:

[0289] The server sends the simulation result to the terminal in real time. The input is the generated simulated image, and the output is the image data for display on the user's terminal. The result is displayed to the user without interruption and is updated dynamically.

[0290] Step 7:

[0291] The user views the simulation results on the device's display. The presented results allow the user to visually confirm the finished look of the makeup and use them as a basis for making a purchase decision.

[0292] (Application Example 1)

[0293] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0294] The challenge is to provide a means for users to accurately simulate and visually confirm the suitability of makeup and accessories in a virtual environment, something that is difficult to do realistically at home or in a regular store. With current technology, it is difficult to provide personalized visual feedback that adapts to the user's preferences in real time.

[0295] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0296] In this invention, the server includes face recognition means for acquiring and analyzing a user's facial image, simulation means for applying accessories in real time based on facial feature information, and virtual reality representation means for enabling trial use in a virtual environment. This allows users to check the finished look of accessories in real time and make the optimal choice while at home or in a virtual store.

[0297] A "user" refers to an individual who uses the system to take a facial image and perform a simulation of how accessories would look on them.

[0298] "Face image" refers to image data obtained by capturing a user's face using a camera or other imaging device.

[0299] "Facial feature information" refers to characteristic data such as skin tone, facial shape, and the position of eyes and lips, which are analyzed from facial images.

[0300] "Decorative items" refer to elements that change a user's appearance by being applied to their face, such as makeup products and accessories.

[0301] "Simulation means" refers to the process of applying accessories in real time based on facial feature information and generating the results.

[0302] "Display means" refers to devices or interfaces that visually provide simulation results to the user.

[0303] A "virtual environment" refers to a digital space that utilizes virtual reality and augmented reality to allow users to experience simulations in real time.

[0304] "Virtual reality representation means" refers to technologies that generate realistic visual experiences in a virtual environment for trying on decorative items.

[0305] To implement this system, users first launch a dedicated application using a device such as a smartphone or smart glasses to acquire a facial image. The acquired facial image is then transmitted from the device to the server via communication. The server analyzes facial feature information based on this facial image using software libraries such as OpenCV and TensorFlow. This analysis extracts data such as skin tone, facial shape, and the positions of the eyes and lips.

[0306] Next, the user selects information about decorative items they are interested in within the app. This information is sent to a server, which then performs a simulation based on the characteristics of the selected items. Cloud services such as AWS are used in this process, and data is processed in real time. The simulation results are sent to the device via virtual reality representation and presented to the user visually.

[0307] The user can visually confirm how the ornaments match within this virtual environment. With this function, the user can simulate the finished look after application without actually picking up the product, enabling a more informed purchasing decision.

[0308] As a specific example, when the user wants to try a particular color of eyeshadow, they can select that color and immediately view the application result through the application. The system is adjusted so that the user can realistically confirm the color and texture under any light source. By using this technology, it is also possible to input a prompt into the generative AI model and give an instruction to try a combination of colors and textures of specific ornaments. For example, by using a prompt sentence such as "Receive a face image and apply the specified lipstick color. Make it possible to confirm the effect in real time.", the system provides the user with an optimal simulation.

[0309] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0310] Step 1:

[0311] The user launches a dedicated application on the terminal and acquires a face image. The input is the user's face image, which is captured by the camera device of the terminal. The camera recognizes the face and automatically adjusts the focus and exposure for shooting. The output is the captured face image data.

[0312] Step 2:

[0313] The terminal transmits the acquired face image data to the server. The input is the face image data obtained in Step 1. This is uploaded to the server via Internet communication. The output is the face image data received on the server.

[0314] Step 3:

[0315] The server applies a face recognition algorithm to the received face image data and analyzes the facial feature information. The input is face image data, and the output is the analyzed facial feature information. The server uses libraries such as OpenCV and TensorFlow to identify the shape of the face, the position of the eyes, and the skin tone.

[0316] Step 4:

[0317] The user selects an accessory within the application and sends that information to the server. The input is the user's selected accessory information, and the output is the accessory identification information received by the server. The user interacts with the application by browsing a list on the screen and tapping the item they want to try.

[0318] Step 5:

[0319] The server performs simulations based on facial feature information and accessory information. The inputs are facial feature information and accessory information. The output is the simulation result, i.e., an image of the face after the accessories have been applied. The server utilizes cloud infrastructure such as AWS to perform calculations that apply texture and color in real time.

[0320] Step 6:

[0321] The server sends the simulation results to the terminal and displays them to the user. The input is the simulation result data, and the output is the user's face image displayed on the terminal's screen. The terminal's display shows the results in full color, and the screen continues to update smoothly even as the user moves.

[0322] Step 7:

[0323] The user visually reviews the displayed results and considers purchasing the accessories. The input is a facial image displayed on the device's screen. The output is the user's purchase intentions and preferences, and the user adds the accessories to their cart as needed. An example of a prompt used to query the generating AI model is, "Receive the facial image and apply the specified lipstick color. Allow real-time confirmation of the effect."

[0324] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0325] This invention provides a makeup simulation system equipped with advanced personalization capabilities that utilize emotion recognition. The system aims to understand the user's emotions from their facial expressions and select and apply the most appropriate accessories accordingly. Specific embodiments are described in detail below.

[0326] User interaction and emotion engine functionality

[0327] The user launches a dedicated application on their device and takes a picture of their face. The device sends this image to a server, which extracts facial feature information using facial recognition technology. Furthermore, an emotion engine simultaneously analyzes the user's facial expressions and recognizes their current emotional state. This recognition result is used as foundational information for user customization.

[0328] Server processing and application of decorations

[0329] The server combines facial feature information and emotional information to select accessories suitable for the user. The data for the selected accessories is adjusted in terms of color and texture according to the user's emotional state, enabling a more personalized simulation. For example, if the server determines that the user is depressed, accessories with bright colors that will lift their spirits will be recommended.

[0330] Displaying the simulation results

[0331] The simulation results with the adjusted decorations applied are transmitted to the terminal in real time and displayed for the user to visually confirm. The terminal interface is designed to provide natural and continuous feedback even as the user moves, allowing for the selection and adjustment of decorations while viewing the results.

[0332] Specific example

[0333] For example, suppose a user takes a selfie at home using their device. If the system detects that the user is smiling, it will suggest a blue eyeshadow to give the user a refreshing image. The simulation results then show the blue eyeshadow being applied to the user's face in real time, which can be viewed on the device. Based on these results, the user can select a product.

[0334] Thus, the present invention takes into account the user's real-time emotions and enables a makeup experience tailored to individual needs. This system is intended to enhance personalization through emotion recognition technology and improve user satisfaction.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] The user launches an application on their device and takes a picture of their face using the camera. The device acquires this face image and immediately prepares to send it to the server.

[0338] Step 2:

[0339] The device sends the acquired facial image data to the server. This data transfer is optimized for high speed by compressing the images.

[0340] Step 3:

[0341] The server receives the facial image sent from the terminal and applies a facial recognition algorithm to extract facial feature information. This process includes identifying skin color, face shape, and the positions of the eyes, nose, and mouth.

[0342] Step 4:

[0343] The server uses an emotion engine in addition to facial image analysis results to identify the user's current emotional state from their facial expressions. The emotion engine recognizes emotions such as smiling, surprise, and sadness in real time.

[0344] Step 5:

[0345] The user selects an item they want to try from a list of accessories presented on the device's interface. The device then sends the selected accessory data and the user's emotional state to the server.

[0346] Step 6:

[0347] The server adjusts the color and texture of selected accessories based on facial feature information and emotional information. For example, if the user is feeling happy, colors that emphasize freshness will be applied. This adjustment data is then input into the simulation system.

[0348] Step 7:

[0349] The server generates simulation results, which are then sent to the terminal. The terminal displays a real-time, adjusted simulation of the adornment on its screen. This allows the user to instantly visualize and confirm the virtual makeup applied to their face.

[0350] Step 8:

[0351] The user selects and further adjusts decorative items based on the displayed simulation results. When the user makes selections or changes, the terminal sends the relevant data back to the server to obtain the latest simulation results.

[0352] This entire process allows users to try on personalized accessories in real time, supporting flexible, emotion-based selections.

[0353] (Example 2)

[0354] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0355] Conventional makeup simulation systems primarily select accessories based on the user's static facial information, and suffer from the problem of insufficient personalization that takes into account the user's emotional changes. Furthermore, real-time adjustment and display of accessories are difficult, making it challenging to improve the user experience.

[0356] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0357] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate feature information; application means for selecting an accessory based on the feature information and emotional information, and for adjusting its color and texture; and display means for simulating and displaying the adjusted accessory on the user's face in real time so that the user can confirm it. This makes it possible to select a more highly personalized accessory and provide real-time feedback, taking into account the user's emotional state.

[0358] A "user" refers to an individual who uses the makeup simulation system by providing a facial image.

[0359] "Facial image" refers to digital image data of the user's face acquired by the device for the purpose of performing makeup simulations.

[0360] "Feature information" refers to user-specific physical characteristics and shape information extracted from facial images.

[0361] "Emotional information" refers to data about a user's emotional state obtained by analyzing their facial expressions.

[0362] "Recognition means" refers to a processing device or program for analyzing facial images and extracting feature information from them.

[0363] "Application means" refers to a processing device or program for selecting ornaments based on characteristic information and emotional information, and for adjusting the parameters of those ornaments.

[0364] "Display means" refers to a device or interface for simulating and displaying an adjusted ornament on a user's face and providing the user with the result.

[0365] "Accessories" refers to data related to cosmetics and accessories applied to the user in the makeup simulation.

[0366] "Real-time" refers to a state where processing and display occur almost instantaneously in response to user actions.

[0367] This invention provides a system that simulates optimal makeup based on the user's emotions. The user first launches a dedicated application on their device and takes a facial image using the camera function. This device is assumed to be a typical smartphone or tablet. The facial image taken by the user is sent from the device to the server. This data transmission is performed via HTTPS to ensure security.

[0368] When the server receives a face image, it first activates a recognition mechanism to extract facial feature information from the image. Here, open-source image processing libraries such as OpenCV and dlib are used to obtain the contours of the face and the positional information of its features. Next, an emotion engine is activated to obtain emotional information, and machine learning frameworks such as TensorFlow and PyTorch are used to analyze facial expressions. This makes it possible to recognize the emotional state of the user, such as whether they are smiling or frowning.

[0369] Based on the characteristic and emotional information obtained, the server selects the most suitable ornament and adjusts its color and texture in real time. MongoDB or MySQL is used as the database to search and retrieve information about the ornaments. During the adjustment phase, color adjustments are made using image editing libraries such as Pillow, and the suggested ornaments change according to the user's emotions.

[0370] The simulation results of the adjusted ornaments are sent from the server to the terminal and displayed in the user interface. This interface is built on a cross-platform framework such as React Native and allows for real-time feedback. Users can further fine-tune the displayed simulation results.

[0371] For example, if a user takes a selfie and smiles, and the emotion recognition system determines that the user is "happy," the server will suggest a pink blush that gives the user a warm impression. The simulation results are applied to the user's face, and the effect can be viewed on the device. In this way, the user receives an optimal makeup experience tailored to their emotions.

[0372] An example of a prompt for a generative AI model is "How to design a makeup simulation system that responds to emotional states," and this prompt can yield results that align with the system's objectives.

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The user launches a dedicated application on their device and uses the camera function to take a picture of their face. When the user presses the capture button, the face image is input to the device. The device converts the acquired face image data into JPEG or PNG format and prepares it for transmission to the server. The output of this step is the face image data to be sent to the server.

[0376] Step 2:

[0377] The server receives face image data sent from the terminal. Next, it analyzes this face image using an image processing library to extract facial feature information. This process executes a face recognition algorithm to identify the positions of the eyes, nose, and mouth within the image. The input is face image data, and the output is feature information regarding the shape and position of the user's face.

[0378] Step 3:

[0379] An emotion engine operates within the server, recognizing the user's emotional state based on facial feature information. A machine learning model is used for emotion recognition, performing facial expression analysis. During this process, the facial recognition results are input into an existing model, and emotional information is output. This output information includes emotion labels such as "happy" and "surprised."

[0380] Step 4:

[0381] The server integrates facial feature information and emotional information, and based on this, selects the most suitable accessory for the user from the database. After selection, it adjusts the color and texture of the accessory to match the emotion. The input is facial feature information and emotional information, and the output is data of the adjusted accessory. The database used here contains information on the type, color, and texture of the accessory.

[0382] Step 5:

[0383] The server sends the adjusted accessory data to the terminal. The terminal generates simulation results on the user interface based on the received data and displays them to the user. Real-time processing allows the display to change smoothly in response to the user's movements. The input is the accessory data, and the output is the simulation display on the terminal.

[0384] (Application Example 2)

[0385] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0386] Currently, providing personalized makeup experiences tailored to users' emotional states is challenging in both physical beauty stores and online platforms. In particular, systems that offer real-time makeup suggestions based on each customer's current emotions to improve satisfaction are still insufficient. To address these challenges, a system is needed that recognizes user emotions and selects and suggests cosmetics best suited to that emotional state.

[0387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0388] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate biometric information; simulation means for applying the properties of selected cosmetics to the user's face in real time based on the biometric information and emotional information; and display means for displaying the simulation results so that the user can visually confirm them. This enables personalized makeup suggestions and simulations that correspond to the user's emotional state.

[0389] A "user" refers to a person who uses the system to experience the makeup simulation.

[0390] "Face image" refers to digital image data of a user's face.

[0391] "Biometric characteristic information" refers to data about the structure and shape of the face obtained through the analysis of facial images.

[0392] "Emotional information" refers to data about the emotional state inferred from the user's facial expressions.

[0393] "Recognition means" refers to methods and devices for acquiring a user's facial image and generating biometric information and emotional information.

[0394] "Cosmetics" refers to products that have colors and textures used for makeup purposes.

[0395] "Simulation means" refers to a method or device that applies the characteristics of selected cosmetics to a user's facial image and displays them visually.

[0396] "Display means" refers to screens or devices that allow users to check the simulation results.

[0397] "Suggestion means" refers to methods and devices for presenting the most suitable cosmetics to a user based on the user's biometric characteristics and emotional information.

[0398] This invention is a system implemented using a smart mirror installed in a physical store. Users can perform a makeup simulation by standing in front of this smart mirror.

[0399] The smart mirror is equipped with a camera, display, and network connectivity. The camera captures the user's facial image, and the acquired data is transmitted to a server via the network. The server analyzes biometric and emotional information using facial recognition and emotion recognition means, thereby understanding the user's emotional state. Based on this information, the server selects the optimal cosmetic product. The simulation results of the selected cosmetic product are displayed in real time on the smart mirror's display, providing the user with visual feedback.

[0400] If the user is smiling, the system will suggest cosmetics in colors that give a refreshing impression. In this way, users can try makeup that matches their mood.

[0401] A concrete example of a prompt would be, "Please list makeup styles suitable for a woman in her 30s whose current emotion is joy." Based on this prompt, the generative AI model creates a list of optimal cosmetics and suggests them to the user throughout the system.

[0402] This system allows users to select and experience makeup styles that match their emotional state, making their makeup choices in physical stores more personalized.

[0403] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0404] Step 1:

[0405] The smart mirror on the device acquires an image of the user's face using its camera. The input is optical information of the user's face, and the output is digitized facial image data. This data is sent to a server for later analysis.

[0406] Step 2:

[0407] The server analyzes the received facial image data using facial recognition technology and extracts biometric information. The input is facial image data, and the output is characteristic data related to the structure and shape of the user's face. Here, information such as the facial contour, eyes, nose, and mouth positions is analyzed.

[0408] Step 3:

[0409] The server uses emotion recognition to obtain emotional information from facial image data. The input is facial image data, and the output is the user's current emotional state. The emotion recognition algorithm estimates the emotion (joy, sadness, surprise, etc.) from the user's facial expression. Based on this result, it determines which emotional state is strongest.

[0410] Step 4:

[0411] The server integrates biometric and emotional information to generate prompt statements for a generative AI model. The input is biometric and emotional information, and the output is a prompt statement. This prompt statement is used by the generative AI model as a criterion for selecting the most suitable cosmetics.

[0412] Step 5:

[0413] A generative AI model lists the optimal cosmetic characteristics based on the prompt text. The input is the prompt text, and the output is data about the color and texture of the cosmetics. This data is used to suggest a cosmetic style that matches the user's emotions.

[0414] Step 6:

[0415] The server applies optimized cosmetic data to the user's facial image using a simulation mechanism and generates simulation results in real time. The input is cosmetic characteristic data and the user's facial image, and the output is a virtual facial image with the cosmetics applied.

[0416] Step 7:

[0417] The smart mirror on the device displays a simulated facial image on its screen, providing the user with visual feedback. The input is the image data of the simulation result, and the output is the visual information displayed on the screen. This allows the user to see and experience suggested cosmetics in real time.

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0419] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0421] [Third Embodiment]

[0422] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0423] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0424] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0426] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0428] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0429] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0430] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0433] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0434] This invention provides a makeup simulation system that allows users to visually confirm realistic results without having to try individual cosmetic items at home or in a store. Embodiments of this system are realized through a series of steps in which the user acquires a facial image via a terminal, and the server analyzes and simulates that image. Specific embodiments of this invention will be described below.

[0435] User actions and terminal functions

[0436] The user first launches a dedicated application installed on their device. The application provides a function to take a picture of the user's face with the camera and record the facial image. After taking the picture, the user selects an item of interest from a list of decorative items displayed on the screen. The selected information is sent from the device to the server in real time.

[0437] Server Processing

[0438] The server receives facial images transmitted from the terminal and analyzes facial features using a facial recognition algorithm. Specifically, it extracts feature information such as facial shape, skin tone, and the position of the eyes and lips. Based on this information, it optimizes the color and texture of selected accessories to match those features and performs a simulation to apply them to the user's face.

[0439] Displaying the simulation results

[0440] The simulation results generated on the server are transmitted to the terminal in real time and displayed on the user's screen. The terminal continues to display the simulation results seamlessly, even as the user's face moves dynamically. This process allows the user to visually confirm what the finished look would be like with actual makeup applied, supporting their pre-purchase decision.

[0441] Specific example

[0442] If a user wants to try red lipstick, they take a picture of their face using the device and select red lipstick. The server then simulates applying the lipstick color to the user's face and sends the result back to the device. The device displays the red lipstick on the user's face in real time, allowing the user to check the result and make a purchase decision based on their satisfaction.

[0443] Thus, the system of the present invention provides end users with a personalized makeup experience, enabling them to make smarter and more effective choices before purchasing.

[0444] The following describes the processing flow.

[0445] Step 1:

[0446] The user launches an application on their device and takes a picture of their face with the camera. The device then prepares to send the captured face image to the server.

[0447] Step 2:

[0448] The device transmits captured facial image data to the server in real time. The image data is compressed before transmission to ensure efficient data transfer.

[0449] Step 3:

[0450] The server analyzes the received facial image data and extracts the user's facial features using a facial recognition algorithm. This includes contour detection, skin tone analysis, and identification of the location of major facial features.

[0451] Step 4:

[0452] The user selects an item they want to try from a list of accessories displayed on the device screen. The device then sends the user's selection information to the server.

[0453] Step 5:

[0454] The server combines the user's facial feature information with data on selected accessories and starts the simulation. During the simulation, deep learning technology is used to apply the selected colors and textures to the facial image in a realistic way.

[0455] Step 6:

[0456] The server sends the generated simulation results to the terminal. This data is optimized to provide a smooth and natural transition for the user's visual perception.

[0457] Step 7:

[0458] The device displays the received simulation results on the screen, allowing the user to check them in real time. The user can evaluate how the virtually applied makeup looks on them and use this information to help them make their selection.

[0459] (Example 1)

[0460] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0461] The challenge is to provide a system that allows users to realistically see the effects of cosmetics on their own faces without actually trying them. This would enable users to visually confirm the results before purchasing cosmetics, allowing them to make more effective purchasing decisions.

[0462] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0463] In this invention, the server includes means for acquiring a facial image via an application installed on a device used by the user, processing means for analyzing the facial image and generating facial feature information, and simulation means for applying the color and texture of selected cosmetics to the user's face in real time based on the facial feature information. This allows the user to check the finished look of the cosmetics in real time.

[0464] "Applications installed on devices used by the user" refers to programs launched on electronic devices operated by the user, and are means of acquiring facial images and providing interfaces.

[0465] "Facial feature information" refers to information that quantifies or digitizes features such as shape, skin tone, and the position of eyes and lips, which are analyzed from a user's facial image.

[0466] A "simulation method" is a method or process that applies selected cosmetics to a user's facial image based on analyzed facial feature information and visually simulates the results.

[0467] "Display means" refers to digital screens or output devices that provide users with simulated results and enable them to dynamically verify them.

[0468] "Communication means" refers to network equipment and communication protocols used to send and receive data between a user's device and a server.

[0469] A "guidance tool" refers to a system or function that provides suggestions and recommendations to help users select the most suitable cosmetics based on their facial features.

[0470] This invention begins with the user acquiring a facial image using a dedicated application installed on a given device. The device uses its camera to capture the user's face and temporarily stores the image data.

[0471] The captured facial image, along with information about the cosmetics selected by the user, is sent to the server. The server processes the facial image using image analysis technology and extracts facial feature information. This process includes algorithms that quantify characteristics such as facial shape, skin tone, and the position of the eyes and lips.

[0472] Based on the facial feature information obtained, the server optimizes the color and texture of the selected cosmetics and performs a simulation. This allows the user to visually confirm the finished look when actually using the cosmetics.

[0473] The simulation results are transmitted to the device in real time, and the user can view them on the device's display. Dynamic display technology allows the results to be displayed seamlessly even if the user's face moves.

[0474] For example, if a user wants to try red lipstick, they simply take a picture of their face with their device and select red lipstick in the application. The server analyzes the facial features, performs a simulation with the red lipstick applied, and returns the results to the device. The user can then see how the red lipstick looks on their face on the screen and decide whether to purchase it.

[0475] An example of a prompt would be: "Describe the process in which a user takes a photo of their face to try on red lipstick, and the server analyzes the photo and performs a simulation."

[0476] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0477] Step 1:

[0478] The user launches an application installed on their device. The user then uses the application's camera function to capture a photo of their face. The input is the user's face, and the output is a digital image temporarily stored on the device.

[0479] Step 2:

[0480] The terminal provides an interface for recording the accessories selected by the user. The user provides information about the selected items as input. As output, the accessory selection information is generated and ready to be sent to the server.

[0481] Step 3:

[0482] The device sends facial images and accessory selection information to the server. The input is the facial image and accessory information stored on the device, and the output is the dataset that arrives on the server. Communication technology is used to ensure that the data is transferred quickly and securely.

[0483] Step 4:

[0484] The server starts the process of analyzing the received face image and extracting facial features. The input is the transmitted face image, and the output is facial feature information including face shape, skin tone, and the positions of the eyes and lips. By using a face recognition algorithm, the necessary data is extracted quickly and accurately.

[0485] Step 5:

[0486] The server performs simulations based on facial feature information and accessory selection data. The inputs are facial feature information and accessory information, and the output is a simulated image of the user's face with the accessories applied. A generative AI model is used to reproduce realistic textures and colors.

[0487] Step 6:

[0488] The server transmits the simulation results to the terminal in real time. The input is the generated simulation image, and the output is image data for display on the user's terminal. The results are displayed to the user without interruption and are dynamically updated.

[0489] Step 7:

[0490] The user views the simulation results on the device's display. The presented results allow the user to visually confirm the finished look of the makeup and use them as a basis for making a purchase decision.

[0491] (Application Example 1)

[0492] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0493] The challenge is to provide a means for users to accurately simulate and visually confirm the suitability of makeup and accessories in a virtual environment, something that is difficult to do realistically at home or in a regular store. With current technology, it is difficult to provide personalized visual feedback that adapts to the user's preferences in real time.

[0494] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0495] In this invention, the server includes face recognition means for acquiring and analyzing a user's facial image, simulation means for applying accessories in real time based on facial feature information, and virtual reality representation means for enabling trial use in a virtual environment. This allows users to check the finished look of accessories in real time and make the optimal choice while at home or in a virtual store.

[0496] A "user" refers to an individual who uses the system to take a facial image and perform a simulation of how accessories would look on them.

[0497] "Face image" refers to image data obtained by capturing a user's face using a camera or other imaging device.

[0498] "Facial feature information" refers to characteristic data such as skin tone, facial shape, and the position of eyes and lips, which are analyzed from facial images.

[0499] "Decorative items" refer to elements that change a user's appearance by being applied to their face, such as makeup products and accessories.

[0500] "Simulation means" refers to the process of applying accessories in real time based on facial feature information and generating the results.

[0501] "Display means" refers to devices or interfaces that visually provide simulation results to the user.

[0502] A "virtual environment" refers to a digital space that utilizes virtual reality and augmented reality to allow users to experience simulations in real time.

[0503] "Virtual reality representation means" refers to technologies that generate realistic visual experiences in a virtual environment for trying on decorative items.

[0504] To implement this system, users first launch a dedicated application using a device such as a smartphone or smart glasses to acquire a facial image. The acquired facial image is then transmitted from the device to the server via communication. The server analyzes facial feature information based on this facial image using software libraries such as OpenCV and TensorFlow. This analysis extracts data such as skin tone, facial shape, and the positions of the eyes and lips.

[0505] Next, the user selects information about decorative items they are interested in within the app. This information is sent to a server, which then performs a simulation based on the characteristics of the selected items. Cloud services such as AWS are used in this process, and data is processed in real time. The simulation results are sent to the device via virtual reality representation and presented to the user visually.

[0506] Users can visually see how decorative items will match within this virtual environment. This feature allows users to simulate the finished look without actually handling the product, enabling them to make smarter purchasing decisions.

[0507] For example, if a user wants to try a specific color of eyeshadow, they can select that color and instantly see the result through the application. The system is optimized so that users can realistically see the color and texture under any lighting conditions. This technology can also be used to prompt a generative AI model and instruct it to try out specific color and texture combinations of accessories. For instance, by using a prompt such as, "Receive a facial image and apply the specified lipstick color. Make the effect visible in real time," the system can provide the user with the most suitable simulation.

[0508] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0509] Step 1:

[0510] The user launches a dedicated application on their device to acquire a facial image. The input is a facial image of the user, which is captured by the device's camera. The camera recognizes the face and automatically adjusts the focus and exposure to take the picture. The output is the captured facial image data.

[0511] Step 2:

[0512] The terminal sends the acquired facial image data to the server. The input is the facial image data obtained in step 1. This is uploaded to the server via internet communication. The output is the facial image data received on the server.

[0513] Step 3:

[0514] The server applies a face recognition algorithm to the received face image data and analyzes the facial feature information. The input is face image data, and the output is the analyzed facial feature information. The server uses libraries such as OpenCV and TensorFlow to identify the shape of the face, the position of the eyes, and the skin tone.

[0515] Step 4:

[0516] The user selects an accessory within the application and sends that information to the server. The input is the user's selected accessory information, and the output is the accessory identification information received by the server. The user interacts with the application by browsing a list on the screen and tapping the item they want to try.

[0517] Step 5:

[0518] The server performs simulations based on facial feature information and accessory information. The inputs are facial feature information and accessory information. The output is the simulation result, i.e., an image of the face after the accessories have been applied. The server utilizes cloud infrastructure such as AWS to perform calculations that apply texture and color in real time.

[0519] Step 6:

[0520] The server sends the simulation results to the terminal and displays them to the user. The input is the simulation result data, and the output is the user's face image displayed on the terminal's screen. The terminal's display shows the results in full color, and the screen continues to update smoothly even as the user moves.

[0521] Step 7:

[0522] The user visually reviews the displayed results and considers purchasing the accessories. The input is a facial image displayed on the device's screen. The output is the user's purchase intentions and preferences, and the user adds the accessories to their cart as needed. An example of a prompt used to query the generating AI model is, "Receive the facial image and apply the specified lipstick color. Allow real-time confirmation of the effect."

[0523] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0524] This invention provides a makeup simulation system equipped with advanced personalization capabilities that utilize emotion recognition. The system aims to understand the user's emotions from their facial expressions and select and apply the most appropriate accessories accordingly. Specific embodiments are described in detail below.

[0525] User interaction and emotion engine functionality

[0526] The user launches a dedicated application on their device and takes a picture of their face. The device sends this image to a server, which extracts facial feature information using facial recognition technology. Furthermore, an emotion engine simultaneously analyzes the user's facial expressions and recognizes their current emotional state. This recognition result is used as foundational information for user customization.

[0527] Server processing and application of decorations

[0528] The server combines facial feature information and emotional information to select accessories suitable for the user. The data for the selected accessories is adjusted in terms of color and texture according to the user's emotional state, enabling a more personalized simulation. For example, if the server determines that the user is depressed, accessories with bright colors that will lift their spirits will be recommended.

[0529] Displaying the simulation results

[0530] The simulation results with the adjusted decorations applied are transmitted to the terminal in real time and displayed for the user to visually confirm. The terminal interface is designed to provide natural and continuous feedback even as the user moves, allowing for the selection and adjustment of decorations while viewing the results.

[0531] Specific example

[0532] For example, suppose a user takes a selfie at home using their device. If the system detects that the user is smiling, it will suggest a blue eyeshadow to give the user a refreshing image. The simulation results then show the blue eyeshadow being applied to the user's face in real time, which can be viewed on the device. Based on these results, the user can select a product.

[0533] Thus, the present invention takes into account the user's real-time emotions and enables a makeup experience tailored to individual needs. This system is intended to enhance personalization through emotion recognition technology and improve user satisfaction.

[0534] The following describes the processing flow.

[0535] Step 1:

[0536] The user launches an application on their device and takes a picture of their face using the camera. The device acquires this face image and immediately prepares to send it to the server.

[0537] Step 2:

[0538] The device sends the acquired facial image data to the server. This data transfer is optimized for high speed by compressing the images.

[0539] Step 3:

[0540] The server receives the facial image sent from the terminal and applies a facial recognition algorithm to extract facial feature information. This process includes identifying skin color, face shape, and the positions of the eyes, nose, and mouth.

[0541] Step 4:

[0542] The server uses an emotion engine in addition to facial image analysis results to identify the user's current emotional state from their facial expressions. The emotion engine recognizes emotions such as smiling, surprise, and sadness in real time.

[0543] Step 5:

[0544] The user selects an item they want to try from a list of accessories presented on the device's interface. The device then sends the selected accessory data and the user's emotional state to the server.

[0545] Step 6:

[0546] The server adjusts the color and texture of selected accessories based on facial feature information and emotional information. For example, if the user is feeling happy, colors that emphasize freshness will be applied. This adjustment data is then input into the simulation system.

[0547] Step 7:

[0548] The server generates simulation results, which are then sent to the terminal. The terminal displays a real-time, adjusted simulation of the adornment on its screen. This allows the user to instantly visualize and confirm the virtual makeup applied to their face.

[0549] Step 8:

[0550] The user selects and further adjusts decorative items based on the displayed simulation results. When the user makes selections or changes, the terminal sends the relevant data back to the server to obtain the latest simulation results.

[0551] This entire process allows users to try on personalized accessories in real time, supporting flexible, emotion-based selections.

[0552] (Example 2)

[0553] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0554] Conventional makeup simulation systems primarily select accessories based on the user's static facial information, and suffer from the problem of insufficient personalization that takes into account the user's emotional changes. Furthermore, real-time adjustment and display of accessories are difficult, making it challenging to improve the user experience.

[0555] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0556] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate feature information; application means for selecting an accessory based on the feature information and emotional information, and for adjusting its color and texture; and display means for simulating and displaying the adjusted accessory on the user's face in real time so that the user can confirm it. This makes it possible to select a more highly personalized accessory and provide real-time feedback, taking into account the user's emotional state.

[0557] A "user" refers to an individual who uses the makeup simulation system by providing a facial image.

[0558] "Facial image" refers to digital image data of the user's face acquired by the device for the purpose of performing makeup simulations.

[0559] "Feature information" refers to user-specific physical characteristics and shape information extracted from facial images.

[0560] "Emotional information" refers to data about a user's emotional state obtained by analyzing their facial expressions.

[0561] "Recognition means" refers to a processing device or program for analyzing facial images and extracting feature information from them.

[0562] "Application means" refers to a processing device or program for selecting ornaments based on characteristic information and emotional information, and for adjusting the parameters of those ornaments.

[0563] "Display means" refers to a device or interface for simulating and displaying an adjusted ornament on a user's face and providing the user with the result.

[0564] "Accessories" refers to data related to cosmetics and accessories applied to the user in the makeup simulation.

[0565] "Real-time" refers to a state where processing and display occur almost instantaneously in response to user actions.

[0566] This invention provides a system that simulates optimal makeup based on the user's emotions. The user first launches a dedicated application on their device and takes a facial image using the camera function. This device is assumed to be a typical smartphone or tablet. The facial image taken by the user is sent from the device to the server. This data transmission is performed via HTTPS to ensure security.

[0567] When the server receives a face image, it first activates a recognition mechanism to extract facial feature information from the image. Here, open-source image processing libraries such as OpenCV and dlib are used to obtain the contours of the face and the positional information of its features. Next, an emotion engine is activated to obtain emotional information, and machine learning frameworks such as TensorFlow and PyTorch are used to analyze facial expressions. This makes it possible to recognize the emotional state of the user, such as whether they are smiling or frowning.

[0568] Based on the characteristic and emotional information obtained, the server selects the most suitable ornament and adjusts its color and texture in real time. MongoDB or MySQL is used as the database to search and retrieve information about the ornaments. During the adjustment phase, color adjustments are made using image editing libraries such as Pillow, and the suggested ornaments change according to the user's emotions.

[0569] The simulation results of the adjusted ornaments are sent from the server to the terminal and displayed in the user interface. This interface is built on a cross-platform framework such as React Native and allows for real-time feedback. Users can further fine-tune the displayed simulation results.

[0570] For example, if a user takes a selfie and smiles, and the emotion recognition system determines that the user is "happy," the server will suggest a pink blush that gives the user a warm impression. The simulation results are applied to the user's face, and the effect can be viewed on the device. In this way, the user receives an optimal makeup experience tailored to their emotions.

[0571] An example of a prompt for a generative AI model is "How to design a makeup simulation system that responds to emotional states," and this prompt can yield results that align with the system's objectives.

[0572] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0573] Step 1:

[0574] The user launches a dedicated application on their device and uses the camera function to take a picture of their face. When the user presses the capture button, the face image is input to the device. The device converts the acquired face image data into JPEG or PNG format and prepares it for transmission to the server. The output of this step is the face image data to be sent to the server.

[0575] Step 2:

[0576] The server receives face image data sent from the terminal. Next, it analyzes this face image using an image processing library to extract facial feature information. This process executes a face recognition algorithm to identify the positions of the eyes, nose, and mouth within the image. The input is face image data, and the output is feature information regarding the shape and position of the user's face.

[0577] Step 3:

[0578] An emotion engine operates within the server, recognizing the user's emotional state based on facial feature information. A machine learning model is used for emotion recognition, performing facial expression analysis. During this process, the facial recognition results are input into an existing model, and emotional information is output. This output information includes emotion labels such as "happy" and "surprised."

[0579] Step 4:

[0580] The server integrates facial feature information and emotional information, and based on this, selects the most suitable accessory for the user from the database. After selection, it adjusts the color and texture of the accessory to match the emotion. The input is facial feature information and emotional information, and the output is data of the adjusted accessory. The database used here contains information on the type, color, and texture of the accessory.

[0581] Step 5:

[0582] The server sends the adjusted accessory data to the terminal. The terminal generates simulation results on the user interface based on the received data and displays them to the user. Real-time processing allows the display to change smoothly in response to the user's movements. The input is the accessory data, and the output is the simulation display on the terminal.

[0583] (Application Example 2)

[0584] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] Currently, providing personalized makeup experiences tailored to users' emotional states is challenging in both physical beauty stores and online platforms. In particular, systems that offer real-time makeup suggestions based on each customer's current emotions to improve satisfaction are still insufficient. To address these challenges, a system is needed that recognizes user emotions and selects and suggests cosmetics best suited to that emotional state.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0587] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate biometric information; simulation means for applying the properties of selected cosmetics to the user's face in real time based on the biometric information and emotional information; and display means for displaying the simulation results so that the user can visually confirm them. This enables personalized makeup suggestions and simulations that correspond to the user's emotional state.

[0588] A "user" refers to a person who uses the system to experience the makeup simulation.

[0589] "Face image" refers to digital image data of a user's face.

[0590] "Biometric characteristic information" refers to data about the structure and shape of the face obtained through the analysis of facial images.

[0591] "Emotional information" refers to data about the emotional state inferred from the user's facial expressions.

[0592] "Recognition means" refers to methods and devices for acquiring a user's facial image and generating biometric information and emotional information.

[0593] "Cosmetics" refers to products that have colors and textures used for makeup purposes.

[0594] "Simulation means" refers to a method or device that applies the characteristics of selected cosmetics to a user's facial image and displays them visually.

[0595] "Display means" refers to screens or devices that allow users to check the simulation results.

[0596] "Suggestion means" refers to methods and devices for presenting the most suitable cosmetics to a user based on the user's biometric characteristics and emotional information.

[0597] This invention is a system implemented using a smart mirror installed in a physical store. Users can perform a makeup simulation by standing in front of this smart mirror.

[0598] The smart mirror is equipped with a camera, display, and network connectivity. The camera captures the user's facial image, and the acquired data is transmitted to a server via the network. The server analyzes biometric and emotional information using facial recognition and emotion recognition means, thereby understanding the user's emotional state. Based on this information, the server selects the optimal cosmetic product. The simulation results of the selected cosmetic product are displayed in real time on the smart mirror's display, providing the user with visual feedback.

[0599] If the user is smiling, the system will suggest cosmetics in colors that give a refreshing impression. In this way, users can try makeup that matches their mood.

[0600] A concrete example of a prompt would be, "Please list makeup styles suitable for a woman in her 30s whose current emotion is joy." Based on this prompt, the generative AI model creates a list of optimal cosmetics and suggests them to the user throughout the system.

[0601] This system allows users to select and experience makeup styles that match their emotional state, making their makeup choices in physical stores more personalized.

[0602] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0603] Step 1:

[0604] The smart mirror on the device acquires an image of the user's face using its camera. The input is optical information of the user's face, and the output is digitized facial image data. This data is sent to a server for later analysis.

[0605] Step 2:

[0606] The server analyzes the received facial image data using facial recognition technology and extracts biometric information. The input is facial image data, and the output is characteristic data related to the structure and shape of the user's face. Here, information such as the facial contour, eyes, nose, and mouth positions is analyzed.

[0607] Step 3:

[0608] The server uses emotion recognition to obtain emotional information from facial image data. The input is facial image data, and the output is the user's current emotional state. The emotion recognition algorithm estimates the emotion (joy, sadness, surprise, etc.) from the user's facial expression. Based on this result, it determines which emotional state is strongest.

[0609] Step 4:

[0610] The server integrates biometric and emotional information to generate prompt statements for a generative AI model. The input is biometric and emotional information, and the output is a prompt statement. This prompt statement is used by the generative AI model as a criterion for selecting the most suitable cosmetics.

[0611] Step 5:

[0612] A generative AI model lists the optimal cosmetic characteristics based on the prompt text. The input is the prompt text, and the output is data about the color and texture of the cosmetics. This data is used to suggest a cosmetic style that matches the user's emotions.

[0613] Step 6:

[0614] The server applies optimized cosmetic data to the user's facial image using a simulation mechanism and generates simulation results in real time. The input is cosmetic characteristic data and the user's facial image, and the output is a virtual facial image with the cosmetics applied.

[0615] Step 7:

[0616] The smart mirror on the device displays a simulated facial image on its screen, providing the user with visual feedback. The input is the image data of the simulation result, and the output is the visual information displayed on the screen. This allows the user to see and experience suggested cosmetics in real time.

[0617] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0618] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0620] [Fourth Embodiment]

[0621] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0622] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0623] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0625] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0627] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0628] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0629] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0630] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0631] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0632] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0633] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] This invention provides a makeup simulation system that allows users to visually confirm realistic results without having to try individual cosmetic items at home or in a store. Embodiments of this system are realized through a series of steps in which the user acquires a facial image via a terminal, and the server analyzes and simulates that image. Specific embodiments of this invention will be described below.

[0635] User actions and terminal functions

[0636] The user first launches a dedicated application installed on their device. The application provides a function to take a picture of the user's face with the camera and record the facial image. After taking the picture, the user selects an item of interest from a list of decorative items displayed on the screen. The selected information is sent from the device to the server in real time.

[0637] Server Processing

[0638] The server receives facial images transmitted from the terminal and analyzes facial features using a facial recognition algorithm. Specifically, it extracts feature information such as facial shape, skin tone, and the position of the eyes and lips. Based on this information, it optimizes the color and texture of selected accessories to match those features and performs a simulation to apply them to the user's face.

[0639] Displaying the simulation results

[0640] The simulation results generated on the server are transmitted to the terminal in real time and displayed on the user's screen. The terminal continues to display the simulation results seamlessly, even as the user's face moves dynamically. This process allows the user to visually confirm what the finished look would be like with actual makeup applied, supporting their pre-purchase decision.

[0641] Specific example

[0642] If a user wants to try red lipstick, they take a picture of their face using the device and select red lipstick. The server then simulates applying the lipstick color to the user's face and sends the result back to the device. The device displays the red lipstick on the user's face in real time, allowing the user to check the result and make a purchase decision based on their satisfaction.

[0643] Thus, the system of the present invention provides end users with a personalized makeup experience, enabling them to make smarter and more effective choices before purchasing.

[0644] The following describes the processing flow.

[0645] Step 1:

[0646] The user launches an application on their device and takes a picture of their face with the camera. The device then prepares to send the captured face image to the server.

[0647] Step 2:

[0648] The device transmits captured facial image data to the server in real time. The image data is compressed before transmission to ensure efficient data transfer.

[0649] Step 3:

[0650] The server analyzes the received facial image data and extracts the user's facial features using a facial recognition algorithm. This includes contour detection, skin tone analysis, and identification of the location of major facial features.

[0651] Step 4:

[0652] The user selects an item they want to try from a list of accessories displayed on the device screen. The device then sends the user's selection information to the server.

[0653] Step 5:

[0654] The server combines the user's facial feature information with data on selected accessories and starts the simulation. During the simulation, deep learning technology is used to apply the selected colors and textures to the facial image in a realistic way.

[0655] Step 6:

[0656] The server sends the generated simulation results to the terminal. This data is optimized to provide a smooth and natural transition for the user's visual perception.

[0657] Step 7:

[0658] The device displays the received simulation results on the screen, allowing the user to check them in real time. The user can evaluate how the virtually applied makeup looks on them and use this information to help them make their selection.

[0659] (Example 1)

[0660] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0661] The challenge is to provide a system that allows users to realistically see the effects of cosmetics on their own faces without actually trying them. This would enable users to visually confirm the results before purchasing cosmetics, allowing them to make more effective purchasing decisions.

[0662] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0663] In this invention, the server includes means for acquiring a facial image via an application installed on a device used by the user, processing means for analyzing the facial image and generating facial feature information, and simulation means for applying the color and texture of selected cosmetics to the user's face in real time based on the facial feature information. This allows the user to check the finished look of the cosmetics in real time.

[0664] "Applications installed on devices used by the user" refers to programs launched on electronic devices operated by the user, and are means of acquiring facial images and providing interfaces.

[0665] "Facial feature information" refers to information that quantifies or digitizes features such as shape, skin tone, and the position of eyes and lips, which are analyzed from a user's facial image.

[0666] A "simulation method" is a method or process that applies selected cosmetics to a user's facial image based on analyzed facial feature information and visually simulates the results.

[0667] "Display means" refers to digital screens or output devices that provide users with simulated results and enable them to dynamically verify them.

[0668] "Communication means" refers to network equipment and communication protocols used to send and receive data between a user's device and a server.

[0669] A "guidance tool" refers to a system or function that provides suggestions and recommendations to help users select the most suitable cosmetics based on their facial features.

[0670] This invention begins with the user acquiring a facial image using a dedicated application installed on a given device. The device uses its camera to capture the user's face and temporarily stores the image data.

[0671] The captured facial image, along with information about the cosmetics selected by the user, is sent to the server. The server processes the facial image using image analysis technology and extracts facial feature information. This process includes algorithms that quantify characteristics such as facial shape, skin tone, and the position of the eyes and lips.

[0672] Based on the facial feature information obtained, the server optimizes the color and texture of the selected cosmetics and performs a simulation. This allows the user to visually confirm the finished look when actually using the cosmetics.

[0673] The simulation results are transmitted to the device in real time, and the user can view them on the device's display. Dynamic display technology allows the results to be displayed seamlessly even if the user's face moves.

[0674] For example, if a user wants to try red lipstick, they simply take a picture of their face with their device and select red lipstick in the application. The server analyzes the facial features, performs a simulation with the red lipstick applied, and returns the results to the device. The user can then see how the red lipstick looks on their face on the screen and decide whether to purchase it.

[0675] An example of a prompt would be: "Describe the process in which a user takes a photo of their face to try on red lipstick, and the server analyzes the photo and performs a simulation."

[0676] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0677] Step 1:

[0678] The user launches an application installed on their device. The user then uses the application's camera function to capture a photo of their face. The input is the user's face, and the output is a digital image temporarily stored on the device.

[0679] Step 2:

[0680] The terminal provides an interface for recording the accessories selected by the user. The user provides information about the selected items as input. As output, the accessory selection information is generated and ready to be sent to the server.

[0681] Step 3:

[0682] The device sends facial images and accessory selection information to the server. The input is the facial image and accessory information stored on the device, and the output is the dataset that arrives on the server. Communication technology is used to ensure that the data is transferred quickly and securely.

[0683] Step 4:

[0684] The server starts the process of analyzing the received face image and extracting facial features. The input is the transmitted face image, and the output is facial feature information including face shape, skin tone, and the positions of the eyes and lips. By using a face recognition algorithm, the necessary data is extracted quickly and accurately.

[0685] Step 5:

[0686] The server performs simulations based on facial feature information and accessory selection data. The inputs are facial feature information and accessory information, and the output is a simulated image of the user's face with the accessories applied. A generative AI model is used to reproduce realistic textures and colors.

[0687] Step 6:

[0688] The server transmits the simulation results to the terminal in real time. The input is the generated simulation image, and the output is image data for display on the user's terminal. The results are displayed to the user without interruption and are dynamically updated.

[0689] Step 7:

[0690] The user views the simulation results on the device's display. The presented results allow the user to visually confirm the finished look of the makeup and use them as a basis for making a purchase decision.

[0691] (Application Example 1)

[0692] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0693] The challenge is to provide a means for users to accurately simulate and visually confirm the suitability of makeup and accessories in a virtual environment, something that is difficult to do realistically at home or in a regular store. With current technology, it is difficult to provide personalized visual feedback that adapts to the user's preferences in real time.

[0694] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0695] In this invention, the server includes face recognition means for acquiring and analyzing a user's facial image, simulation means for applying accessories in real time based on facial feature information, and virtual reality representation means for enabling trial use in a virtual environment. This allows users to check the finished look of accessories in real time and make the optimal choice while at home or in a virtual store.

[0696] A "user" refers to an individual who uses the system to take a facial image and perform a simulation of how accessories would look on them.

[0697] "Face image" refers to image data obtained by capturing a user's face using a camera or other imaging device.

[0698] "Facial feature information" refers to characteristic data such as skin tone, facial shape, and the position of eyes and lips, which are analyzed from facial images.

[0699] "Decorative items" refer to elements that change a user's appearance by being applied to their face, such as makeup products and accessories.

[0700] "Simulation means" refers to the process of applying accessories in real time based on facial feature information and generating the results.

[0701] "Display means" refers to devices or interfaces that visually provide simulation results to the user.

[0702] A "virtual environment" refers to a digital space that utilizes virtual reality and augmented reality to allow users to experience simulations in real time.

[0703] "Virtual reality representation means" refers to technologies that generate realistic visual experiences in a virtual environment for trying on decorative items.

[0704] To implement this system, users first launch a dedicated application using a device such as a smartphone or smart glasses to acquire a facial image. The acquired facial image is then transmitted from the device to the server via communication. The server analyzes facial feature information based on this facial image using software libraries such as OpenCV and TensorFlow. This analysis extracts data such as skin tone, facial shape, and the positions of the eyes and lips.

[0705] Next, the user selects information about decorative items they are interested in within the app. This information is sent to a server, which then performs a simulation based on the characteristics of the selected items. Cloud services such as AWS are used in this process, and data is processed in real time. The simulation results are sent to the device via virtual reality representation and presented to the user visually.

[0706] Users can visually see how decorative items will match within this virtual environment. This feature allows users to simulate the finished look without actually handling the product, enabling them to make smarter purchasing decisions.

[0707] For example, if a user wants to try a specific color of eyeshadow, they can select that color and instantly see the result through the application. The system is optimized so that users can realistically see the color and texture under any lighting conditions. This technology can also be used to prompt a generative AI model and instruct it to try out specific color and texture combinations of accessories. For instance, by using a prompt such as, "Receive a facial image and apply the specified lipstick color. Make the effect visible in real time," the system can provide the user with the most suitable simulation.

[0708] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0709] Step 1:

[0710] The user launches a dedicated application on their device to acquire a facial image. The input is a facial image of the user, which is captured by the device's camera. The camera recognizes the face and automatically adjusts the focus and exposure to take the picture. The output is the captured facial image data.

[0711] Step 2:

[0712] The terminal sends the acquired facial image data to the server. The input is the facial image data obtained in step 1. This is uploaded to the server via internet communication. The output is the facial image data received on the server.

[0713] Step 3:

[0714] The server applies a face recognition algorithm to the received face image data and analyzes the facial feature information. The input is face image data, and the output is the analyzed facial feature information. The server uses libraries such as OpenCV and TensorFlow to identify the shape of the face, the position of the eyes, and the skin tone.

[0715] Step 4:

[0716] The user selects an accessory within the application and sends that information to the server. The input is the user's selected accessory information, and the output is the accessory identification information received by the server. The user interacts with the application by browsing a list on the screen and tapping the item they want to try.

[0717] Step 5:

[0718] The server performs simulations based on facial feature information and accessory information. The inputs are facial feature information and accessory information. The output is the simulation result, i.e., an image of the face after the accessories have been applied. The server utilizes cloud infrastructure such as AWS to perform calculations that apply texture and color in real time.

[0719] Step 6:

[0720] The server sends the simulation results to the terminal and displays them to the user. The input is the simulation result data, and the output is the user's face image displayed on the terminal's screen. The terminal's display shows the results in full color, and the screen continues to update smoothly even as the user moves.

[0721] Step 7:

[0722] The user visually reviews the displayed results and considers purchasing the accessories. The input is a facial image displayed on the device's screen. The output is the user's purchase intentions and preferences, and the user adds the accessories to their cart as needed. An example of a prompt used to query the generating AI model is, "Receive the facial image and apply the specified lipstick color. Allow real-time confirmation of the effect."

[0723] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0724] This invention provides a makeup simulation system equipped with advanced personalization capabilities that utilize emotion recognition. The system aims to understand the user's emotions from their facial expressions and select and apply the most appropriate accessories accordingly. Specific embodiments are described in detail below.

[0725] User interaction and emotion engine functionality

[0726] The user launches a dedicated application on their device and takes a picture of their face. The device sends this image to a server, which extracts facial feature information using facial recognition technology. Furthermore, an emotion engine simultaneously analyzes the user's facial expressions and recognizes their current emotional state. This recognition result is used as foundational information for user customization.

[0727] Server processing and application of decorations

[0728] The server combines facial feature information and emotional information to select accessories suitable for the user. The data for the selected accessories is adjusted in terms of color and texture according to the user's emotional state, enabling a more personalized simulation. For example, if the server determines that the user is depressed, accessories with bright colors that will lift their spirits will be recommended.

[0729] Displaying the simulation results

[0730] The simulation results with the adjusted decorations applied are transmitted to the terminal in real time and displayed for the user to visually confirm. The terminal interface is designed to provide natural and continuous feedback even as the user moves, allowing for the selection and adjustment of decorations while viewing the results.

[0731] Specific example

[0732] For example, suppose a user takes a selfie at home using their device. If the system detects that the user is smiling, it will suggest a blue eyeshadow to give the user a refreshing image. The simulation results then show the blue eyeshadow being applied to the user's face in real time, which can be viewed on the device. Based on these results, the user can select a product.

[0733] Thus, the present invention takes into account the user's real-time emotions and enables a makeup experience tailored to individual needs. This system is intended to enhance personalization through emotion recognition technology and improve user satisfaction.

[0734] The following describes the processing flow.

[0735] Step 1:

[0736] The user launches an application on their device and takes a picture of their face using the camera. The device acquires this face image and immediately prepares to send it to the server.

[0737] Step 2:

[0738] The device sends the acquired facial image data to the server. This data transfer is optimized for high speed by compressing the images.

[0739] Step 3:

[0740] The server receives the facial image sent from the terminal and applies a facial recognition algorithm to extract facial feature information. This process includes identifying skin color, face shape, and the positions of the eyes, nose, and mouth.

[0741] Step 4:

[0742] The server uses an emotion engine in addition to facial image analysis results to identify the user's current emotional state from their facial expressions. The emotion engine recognizes emotions such as smiling, surprise, and sadness in real time.

[0743] Step 5:

[0744] The user selects an item they want to try from a list of accessories presented on the device's interface. The device then sends the selected accessory data and the user's emotional state to the server.

[0745] Step 6:

[0746] The server adjusts the color and texture of selected accessories based on facial feature information and emotional information. For example, if the user is feeling happy, colors that emphasize freshness will be applied. This adjustment data is then input into the simulation system.

[0747] Step 7:

[0748] The server generates simulation results, which are then sent to the terminal. The terminal displays a real-time, adjusted simulation of the adornment on its screen. This allows the user to instantly visualize and confirm the virtual makeup applied to their face.

[0749] Step 8:

[0750] The user selects and further adjusts decorative items based on the displayed simulation results. When the user makes selections or changes, the terminal sends the relevant data back to the server to obtain the latest simulation results.

[0751] This entire process allows users to try on personalized accessories in real time, supporting flexible, emotion-based selections.

[0752] (Example 2)

[0753] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0754] Conventional makeup simulation systems primarily select accessories based on the user's static facial information, and suffer from the problem of insufficient personalization that takes into account the user's emotional changes. Furthermore, real-time adjustment and display of accessories are difficult, making it challenging to improve the user experience.

[0755] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0756] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate feature information; application means for selecting an accessory based on the feature information and emotional information, and for adjusting its color and texture; and display means for simulating and displaying the adjusted accessory on the user's face in real time so that the user can confirm it. This makes it possible to select a more highly personalized accessory and provide real-time feedback, taking into account the user's emotional state.

[0757] A "user" refers to an individual who uses the makeup simulation system by providing a facial image.

[0758] "Facial image" refers to digital image data of the user's face acquired by the device for the purpose of performing makeup simulations.

[0759] "Feature information" refers to user-specific physical characteristics and shape information extracted from facial images.

[0760] "Emotional information" refers to data about a user's emotional state obtained by analyzing their facial expressions.

[0761] "Recognition means" refers to a processing device or program for analyzing facial images and extracting feature information from them.

[0762] "Application means" refers to a processing device or program for selecting ornaments based on characteristic information and emotional information, and for adjusting the parameters of those ornaments.

[0763] "Display means" refers to a device or interface for simulating and displaying an adjusted ornament on a user's face and providing the user with the result.

[0764] "Accessories" refers to data related to cosmetics and accessories applied to the user in the makeup simulation.

[0765] "Real-time" refers to a state where processing and display occur almost instantaneously in response to user actions.

[0766] This invention provides a system that simulates optimal makeup based on the user's emotions. The user first launches a dedicated application on their device and takes a facial image using the camera function. This device is assumed to be a typical smartphone or tablet. The facial image taken by the user is sent from the device to the server. This data transmission is performed via HTTPS to ensure security.

[0767] When the server receives a face image, it first activates a recognition mechanism to extract facial feature information from the image. Here, open-source image processing libraries such as OpenCV and dlib are used to obtain the contours of the face and the positional information of its features. Next, an emotion engine is activated to obtain emotional information, and machine learning frameworks such as TensorFlow and PyTorch are used to analyze facial expressions. This makes it possible to recognize the emotional state of the user, such as whether they are smiling or frowning.

[0768] Based on the characteristic and emotional information obtained, the server selects the most suitable ornament and adjusts its color and texture in real time. MongoDB or MySQL is used as the database to search and retrieve information about the ornaments. During the adjustment phase, color adjustments are made using image editing libraries such as Pillow, and the suggested ornaments change according to the user's emotions.

[0769] The simulation results of the adjusted ornaments are sent from the server to the terminal and displayed in the user interface. This interface is built on a cross-platform framework such as React Native and allows for real-time feedback. Users can further fine-tune the displayed simulation results.

[0770] For example, if a user takes a selfie and smiles, and the emotion recognition system determines that the user is "happy," the server will suggest a pink blush that gives the user a warm impression. The simulation results are applied to the user's face, and the effect can be viewed on the device. In this way, the user receives an optimal makeup experience tailored to their emotions.

[0771] An example of a prompt for a generative AI model is "How to design a makeup simulation system that responds to emotional states," and this prompt can yield results that align with the system's objectives.

[0772] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0773] Step 1:

[0774] The user launches a dedicated application on their device and uses the camera function to take a picture of their face. When the user presses the capture button, the face image is input to the device. The device converts the acquired face image data into JPEG or PNG format and prepares it for transmission to the server. The output of this step is the face image data to be sent to the server.

[0775] Step 2:

[0776] The server receives face image data sent from the terminal. Next, it analyzes this face image using an image processing library to extract facial feature information. This process executes a face recognition algorithm to identify the positions of the eyes, nose, and mouth within the image. The input is face image data, and the output is feature information regarding the shape and position of the user's face.

[0777] Step 3:

[0778] An emotion engine operates within the server, recognizing the user's emotional state based on facial feature information. A machine learning model is used for emotion recognition, performing facial expression analysis. During this process, the facial recognition results are input into an existing model, and emotional information is output. This output information includes emotion labels such as "happy" and "surprised."

[0779] Step 4:

[0780] The server integrates facial feature information and emotional information, and based on this, selects the most suitable accessory for the user from the database. After selection, it adjusts the color and texture of the accessory to match the emotion. The input is facial feature information and emotional information, and the output is data of the adjusted accessory. The database used here contains information on the type, color, and texture of the accessory.

[0781] Step 5:

[0782] The server sends the adjusted accessory data to the terminal. The terminal generates simulation results on the user interface based on the received data and displays them to the user. Real-time processing allows the display to change smoothly in response to the user's movements. The input is the accessory data, and the output is the simulation display on the terminal.

[0783] (Application Example 2)

[0784] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] Currently, providing personalized makeup experiences tailored to users' emotional states is challenging in both physical beauty stores and online platforms. In particular, systems that offer real-time makeup suggestions based on each customer's current emotions to improve satisfaction are still insufficient. To address these challenges, a system is needed that recognizes user emotions and selects and suggests cosmetics best suited to that emotional state.

[0786] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0787] In this invention, the server includes recognition means for acquiring a user's facial image and analyzing the facial image to generate biometric information; simulation means for applying the properties of selected cosmetics to the user's face in real time based on the biometric information and emotional information; and display means for displaying the simulation results so that the user can visually confirm them. This enables personalized makeup suggestions and simulations that correspond to the user's emotional state.

[0788] A "user" refers to a person who uses the system to experience the makeup simulation.

[0789] "Face image" refers to digital image data of a user's face.

[0790] "Biometric characteristic information" refers to data about the structure and shape of the face obtained through the analysis of facial images.

[0791] "Emotional information" refers to data about the emotional state inferred from the user's facial expressions.

[0792] "Recognition means" refers to methods and devices for acquiring a user's facial image and generating biometric information and emotional information.

[0793] "Cosmetics" refers to products that have colors and textures used for makeup purposes.

[0794] "Simulation means" refers to a method or device that applies the characteristics of selected cosmetics to a user's facial image and displays them visually.

[0795] "Display means" refers to screens or devices that allow users to check the simulation results.

[0796] "Suggestion means" refers to methods and devices for presenting the most suitable cosmetics to a user based on the user's biometric characteristics and emotional information.

[0797] This invention is a system implemented using a smart mirror installed in a physical store. Users can perform a makeup simulation by standing in front of this smart mirror.

[0798] The smart mirror is equipped with a camera, display, and network connectivity. The camera captures the user's facial image, and the acquired data is transmitted to a server via the network. The server analyzes biometric and emotional information using facial recognition and emotion recognition means, thereby understanding the user's emotional state. Based on this information, the server selects the optimal cosmetic product. The simulation results of the selected cosmetic product are displayed in real time on the smart mirror's display, providing the user with visual feedback.

[0799] If the user is smiling, the system will suggest cosmetics in colors that give a refreshing impression. In this way, users can try makeup that matches their mood.

[0800] A concrete example of a prompt would be, "Please list makeup styles suitable for a woman in her 30s whose current emotion is joy." Based on this prompt, the generative AI model creates a list of optimal cosmetics and suggests them to the user throughout the system.

[0801] This system allows users to select and experience makeup styles that match their emotional state, making their makeup choices in physical stores more personalized.

[0802] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0803] Step 1:

[0804] The smart mirror on the device acquires an image of the user's face using its camera. The input is optical information of the user's face, and the output is digitized facial image data. This data is sent to a server for later analysis.

[0805] Step 2:

[0806] The server analyzes the received facial image data using facial recognition technology and extracts biometric information. The input is facial image data, and the output is characteristic data related to the structure and shape of the user's face. Here, information such as the facial contour, eyes, nose, and mouth positions is analyzed.

[0807] Step 3:

[0808] The server uses emotion recognition to obtain emotional information from facial image data. The input is facial image data, and the output is the user's current emotional state. The emotion recognition algorithm estimates the emotion (joy, sadness, surprise, etc.) from the user's facial expression. Based on this result, it determines which emotional state is strongest.

[0809] Step 4:

[0810] The server integrates biometric and emotional information to generate prompt statements for a generative AI model. The input is biometric and emotional information, and the output is a prompt statement. This prompt statement is used by the generative AI model as a criterion for selecting the most suitable cosmetics.

[0811] Step 5:

[0812] A generative AI model lists the optimal cosmetic characteristics based on the prompt text. The input is the prompt text, and the output is data about the color and texture of the cosmetics. This data is used to suggest a cosmetic style that matches the user's emotions.

[0813] Step 6:

[0814] The server applies optimized cosmetic data to the user's facial image using a simulation mechanism and generates simulation results in real time. The input is cosmetic characteristic data and the user's facial image, and the output is a virtual facial image with the cosmetics applied.

[0815] Step 7:

[0816] The smart mirror on the device displays a simulated facial image on its screen, providing the user with visual feedback. The input is the image data of the simulation result, and the output is the visual information displayed on the screen. This allows the user to see and experience suggested cosmetics in real time.

[0817] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0818] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0819] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0820] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0821] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0822] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0823] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0824] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0825] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0826] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0827] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0828] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0829] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0830] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0831] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0832] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0833] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0834] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0835] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0836] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0837] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0838] The following is further disclosed regarding the embodiments described above.

[0839] (Claim 1)

[0840] A face recognition means for acquiring a user's face image and analyzing the face image to generate face feature information,

[0841] A simulation means for applying the color and texture of an accessory selected based on the facial feature information to the user's face in real time,

[0842] A display means for displaying the simulation results so that the user can visually confirm them,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, comprising communication means for processing data generated in real time according to the selection of the ornament.

[0846] (Claim 3)

[0847] The system according to claim 1, further comprising a suggestion means for suggesting the most suitable accessory to the user based on the facial feature information.

[0848] "Example 1"

[0849] (Claim 1)

[0850] A means of acquiring a facial image via an application installed on a device used by the user,

[0851] Processing means for analyzing the face image and generating face feature information,

[0852] A simulation means for applying the color and texture of cosmetics selected based on the facial feature information to the user's face in real time,

[0853] A display means for displaying the simulated results and allowing the user to dynamically check them,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, comprising communication means for processing information generated in real time according to the selection of the cosmetic product.

[0857] (Claim 3)

[0858] The system according to claim 1, further comprising guidance means for suggesting the most suitable cosmetics to the user based on the facial feature information.

[0859] "Application Example 1"

[0860] (Claim 1)

[0861] A face recognition means for acquiring a user's face image and analyzing the face image to generate face feature information,

[0862] A simulation means for applying the color and texture of an accessory selected based on the facial feature information to the user's face in real time,

[0863] A display means for displaying the simulation results so that the user can visually confirm them,

[0864] A virtual reality representation method for users to try out decorative items selected in a virtual environment,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, comprising communication means for processing data generated in real time according to the selection of the ornament.

[0868] (Claim 3)

[0869] The system according to claim 1, further comprising a suggestion means for suggesting the most suitable accessory to the user based on the facial feature information.

[0870] "Example 2 of combining an emotion engine"

[0871] (Claim 1)

[0872] A recognition means for acquiring a user's facial image, analyzing the facial image, and generating feature information,

[0873] A means for selecting decorative items based on characteristic information and emotional information, and for adjusting their color and texture,

[0874] A display means for simulating and displaying the adjusted ornament on the user's face in real time so that the user can confirm it,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, comprising communication means for transmitting and processing information necessary for the selection and adjustment of the ornament in real time.

[0878] (Claim 3)

[0879] The system according to claim 1, further comprising a suggestion means for suggesting the most suitable ornament to the user based on the characteristic information and emotional information.

[0880] "Application example 2 when combining with an emotional engine"

[0881] (Claim 1)

[0882] A recognition means for acquiring a user's facial image, analyzing the facial image, and generating biometric information,

[0883] A simulation means for applying the characteristics of cosmetics selected based on the bio-characteristic information and emotional information to the user's face in real time,

[0884] A display means for displaying the simulation results so that the user can visually confirm them,

[0885] A means of suggesting cosmetics that are appropriate to the user's emotions in real time,

[0886] A system that includes this.

[0887] (Claim 2)

[0888] The system according to claim 1, comprising communication means for processing data generated in real time according to the selection of the cosmetic product.

[0889] (Claim 3)

[0890] The system according to claim 1, comprising analytical means for analyzing the user's emotional state and performing individually optimized makeup simulations according to the emotional state. [Explanation of Symbols]

[0891] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A face recognition means for acquiring a user's face image and analyzing the face image to generate face feature information, A simulation means for applying the color and texture of an accessory selected based on the facial feature information to the user's face in real time, A display means for displaying the simulation results so that the user can visually confirm them, A system that includes this.

2. The system according to claim 1, further comprising communication means for processing data generated in real time according to the selection of the ornament.

3. The system according to claim 1, further comprising a suggestion means for suggesting the most suitable ornament to the user based on the facial feature information.

Images