system

Abstract

We provide the system. [Solution] A means for recognizing the features of a user's face and analyzing their style based on those features, A means of reflecting the analyzed style onto the user's image in real time, A means of linking with a database to recommend a style suitable for the user, A means of displaying product information related to the style selected by the user and proceeding with the order, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, many people have difficulty finding makeup and hairstyles that suit them. In particular, for styles that they have never challenged before, there are many cases where they hesitate to try because they fear failure. Against this background, there is a need for technology that enables users to confidently challenge new styles.

Means for Solving the Problems

[0005] This invention provides a technology that recognizes the facial features of a user and analyzes the optimal makeup and hairstyle based on that data. Furthermore, it reflects the analysis results on the user's video in real time, allowing them to easily try different styles. It also integrates with an online database to recommend suitable styles to the user, displaying product information related to the selected style and enabling the user to proceed with the purchase. This makes it easier and safer for users to discover and try new styles.

[0006] "User facial features" refer to the individual physical characteristics of the face, including the contours of the face and the position, shape, and size of each facial feature.

[0007] "Means of analyzing style" refers to software or algorithms that evaluate and suggest appropriate makeup and hairstyles based on the user's facial features.

[0008] "Means of reflecting the user's image in real time" refers to technologies that utilize computer graphics and image processing techniques to instantly reproduce the style selected by the user on the screen.

[0009] "Means of interacting with a database" refers to a process or interface that communicates with an external or internal information storage system to retrieve, store, and manage necessary data.

[0010] "Means for displaying product information and processing orders" refers to the user interface and backend processes that display related products based on the selected style and assist the user in purchasing them online. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention presents an embodiment of a system that allows users to easily find makeup and hairstyles that suit them. This system uses facial recognition technology to scan the user's facial features and optimizes the style based on that data. The specific implementation of each element is described below.

[0033] User facial recognition

[0034] The device has a built-in high-resolution camera that automatically activates when the user stands in front of the smart mirror. The device uses an advanced facial recognition algorithm to capture the user's facial features in real time and analyze the contours, eyes, nose, and mouth position and shape. This data is sent to a server and used for further analysis.

[0035] Style recommendations and displays

[0036] The server uses the received facial feature data to match it against a large style database. This allows it to suggest multiple optimal makeup and hairstyle styles for the user. The suggested styles are displayed on the device's screen, and the user can select and try different styles using touch controls.

[0037] Virtual Simulation

[0038] The device overlays the selected style onto the user's face in real time, visualizing virtual makeup and hairstyles. This simulation follows the user's movements, allowing for viewing from different angles. The system utilizes GPU-driven image processing to achieve this.

[0039] Product recommendation and purchase processing

[0040] Products matching the user's selected style are fed back to the server and displayed as a list on the device. If the user wishes to purchase a product, they can proceed with the online order process through the device, and payment information is sent to the server via a secure communication method. The server records this information and stores the purchase history in a database.

[0041] As a concrete example, consider a scenario where User A stands in front of a smart mirror and tries to change their hairstyle. User A's face is recognized, and several hairstyle styles are recommended from the server. User A selects their favorite hairstyle, and that hairstyle is virtually simulated. During this time, User A can check whether it suits their face from various angles, and if they like it, they can purchase related products on the spot.

[0042] Through this process, users can easily try out new styles and purchase related products.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The device activates its camera and captures a video of the user's face. When the user stands in front of the smart mirror, it automatically recognizes the face area and extracts feature data.

[0046] Step 2:

[0047] The server receives the user's facial feature data sent from the terminal. Based on this, it compares it with data in a pre-built style database and selects a style that suits the user.

[0048] Step 3:

[0049] The server generates a list of selected makeup and hairstyle styles and sends this information to the terminal. The terminal displays these recommended styles to the user and prompts them to make a selection.

[0050] Step 4:

[0051] The user selects a style they are interested in, and the device overlays the selected style onto the user's face in real time. This allows the user to see how the style will look on their own face.

[0052] Step 5:

[0053] The server analyzes the user's selections and creates an optimized list of product recommendations. This product information is sent to the terminal and presented to the user.

[0054] Step 6:

[0055] The user selects a product they like and expresses their intention to purchase it via their device. The device then uses secure communication with the server to process the order and verify payment information.

[0056] Step 7:

[0057] The server processes purchase information and stores it in the database as purchase history. This allows for more personalized suggestions to the user the next time they use the service.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] There is a problem in that it is difficult for individual users to easily and effectively find the style that is best suited to them. Furthermore, there is a lack of a process for users to easily purchase products related to their chosen style, as they cannot see how the style will actually look in real time.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for identifying the features of the user's face and analyzing expressions based on those features, means for reflecting the analyzed expressions on the user's video in real time, and means for coordinating with a data storage device for recommending expressions suitable for the user. This enables the user to quickly find the style that is best suited to them, confirm that style in real time, and smoothly purchase related products.

[0063] A "user" refers to an individual who uses this system to select and experiment with styles.

[0064] "Facial features" refer to data about the position and shape of an individual's eyes, nose, mouth, and facial contours.

[0065] "Expression" refers to styles and designs that change a user's appearance, such as makeup and hairstyles.

[0066] "Identification" refers to the process of extracting and recognizing specific features from images or data using a particular algorithm.

[0067] "Analysis" is the process of thoroughly examining collected data, extracting information, and understanding it.

[0068] A "generative AI model" refers to an algorithm or program that uses AI technology to automatically create new designs and styles.

[0069] A "data storage device" refers to a system for systematically storing and managing multiple styles and product information.

[0070] "Real-time" refers to the processing and reflection of information occurring almost instantaneously or in a very short time.

[0071] "This invention" refers to a system designed to enable users to select and experience styles that are optimized for them.

[0072] A description of embodiments for carrying out the present invention will be provided.

[0073] This system is designed to help users easily find styles that suit them. The device incorporates a high-resolution camera and display, and when a user stands in front of the device, it automatically captures the user's face and identifies their facial features. This utilizes advanced facial recognition algorithms to analyze the contours, eyes, nose, and mouth position and shape in real time.

[0074] This analysis data is sent to a server, which uses a generative AI model to generate multiple styles based on the data. The server then compares these styles with a large-scale data storage system and recommends the style best suited to the user. This recommendation is personalized based on the user's individual characteristics.

[0075] The device displays recommended styles on its screen, and the user can try out the styles using touch controls. The device simulates the selected style on the user's face in real time. This process is made possible by the device's GPU technology, allowing the user to view the styles from various angles while moving.

[0076] Furthermore, the server retrieves product information related to the selected style from a data storage device and displays it as a list on the terminal. The user can then purchase related products based on this information. The purchase process is completed online through the terminal, payment information is securely transmitted to the server, and the purchase history is recorded in the database.

[0077] For example, if user A wants to try a new hairstyle using this system, the hairstyle style will be simulated on their face in real time, and if they like it, the corresponding care products will be immediately displayed and available for purchase.

[0078] An example of a prompt message would be: "Use facial recognition technology to suggest makeup and hairstyle styles that suit me. Please simulate them virtually in real time and also display a list of related products."

[0079] In this way, users can easily and efficiently try out styles that suit them, and purchase related products seamlessly.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The device automatically activates its high-resolution camera when a user stands in front of it. The camera captures the user's face and acquires image data. Based on this input data, the device uses a facial recognition algorithm to identify the user's facial features, determining the contours and the position and shape of the eyes, nose, and mouth. As a result, analyzed feature data is output.

[0083] Step 2:

[0084] The server receives facial feature data transmitted from the device and inputs this data into a generating AI model. The server uses the data to generate the optimal style for the user and outputs several possible styles. The server then creates a unique style and prepares to make optimal suggestions.

[0085] Step 3:

[0086] The server compares the generated styles with style data stored in the data storage device. Through this process, the server identifies the style best suited to the user and outputs prioritized style information.

[0087] Step 4:

[0088] Upon receiving style information from the server, the terminal displays these styles on its screen. Based on this display, the user selects the style they wish to try via the touchscreen. This selection process then passes the selected style data to the next stage.

[0089] Step 5:

[0090] The device overlays the selected style onto the user's face in real time, displaying it for the user to see. The GPU is used to perform style simulations, dynamically adjusting the style to different angles and lighting conditions. The output is a real-time visual simulation.

[0091] Step 6:

[0092] Based on the user's interests, the server retrieves relevant product information from the data storage device and sends it to the terminal. The output provides a list of products related to the desired style, allowing the user to make a purchase selection.

[0093] Step 7:

[0094] If a user wishes to purchase a selected product, the terminal initiates the ordering process. Payment information is sent to the server via secure communication, which receives this information and records it in the purchase history database. As a result, the online purchase is completed, and the purchase history is updated.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] Traditional beauty experiences require users to spend a lot of time and effort trying out new looks, and the process of actually purchasing the products associated with the chosen look is often cumbersome. Therefore, there is a need for an efficient system that allows users to easily try out new looks and purchase related products on the spot.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for recognizing the user's facial features and analyzing their appearance based on those features; means for reflecting the analyzed appearance on the user's video in real time; and means for displaying product information related to the appearance selected by the user and facilitating the purchase process. This makes it possible for users to interactively try out new appearances in physical stores and efficiently purchase related products on the spot.

[0100] A "user" is an individual who operates the system and tries out different appearances to find what suits them best.

[0101] "Facial features" refers to the physical characteristics of a user's face, such as its shape, contours, eyes, nose, and mouth.

[0102] "Appearance" refers to styles such as makeup and hairstyles that users can try out.

[0103] An "information collection" is a database used to provide users with recommended appearance options.

[0104] "Product information" refers to information about products and services related to the appearance selected by the user.

[0105] The "purchase process" refers to the series of steps involved in a user ordering a selected product online and completing the payment.

[0106] "Physical stores" refer to shopping facilities and beauty salons that exist in a physical location.

[0107] An "interactive experience" is a process in which users can interact with the system in both directions, try out the appearance in real time, and make a purchase.

[0108] The system for implementing this invention mainly consists of interactions between a server, a terminal, and a user.

[0109] The server analyzes the user's facial features using advanced facial recognition algorithms. Specifically, it processes the user's facial image captured by a high-resolution camera, identifying the contours and positions of the eyes, nose, and mouth. Based on this data, the server works in conjunction with an information database to generate a selection of options to suggest the most suitable appearance for the user.

[0110] The terminal incorporates a mirror-type display and a camera. The terminal overlays suggested appearances received from the server onto the user's video in real time, displaying them interactively. Through the display, users can try out different styles and instantly view product information related to their selected appearance.

[0111] When a user selects a specific appearance, the system displays relevant product information on the terminal, allowing them to proceed with the purchase immediately. Once the user completes the payment, the information is securely transmitted to the server, and the purchase history is recorded.

[0112] As a concrete example, imagine a customer experiencing this interactive system for the first time in a department store's cosmetics counter. The customer stands in front of a smart mirror, tries out several makeup styles, and by selecting one, can order related cosmetics on the spot. This process takes only a few minutes from start to finish, providing the customer with an efficient and intuitive shopping experience.

[0113] Furthermore, the generative AI model associated with this system can generate new styles using prompt statements like the following.

[0114] "Design an interactive system that uses facial recognition technology to suggest recommended makeup and hairstyles based on the user's facial features in real time, and allows users to purchase related products."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The server receives a user's facial image sent from the terminal. Based on this input image, it uses a facial recognition algorithm to analyze facial features such as the eyes, nose, and mouth, and outputs their coordinate data. This facial data is basic information necessary for subsequent style recommendation.

[0118] Step 2:

[0119] The server uses facial feature data to match it with a database and creates a list of appearances suitable for the user. The input consists of facial feature data and user preference data, which are then used to output candidate appearance styles, which are sent to the terminal.

[0120] Step 3:

[0121] The terminal displays style candidates received from the server, overlaid on the user's video in real time. The input consists of style data and the user's video, and multiple styles are applied to the displayed video, allowing the user to visually try them out.

[0122] Step 4:

[0123] Users select their preferred appearance style through the device's interface. This selection information is sent to a server, where product information related to that appearance is collected. The selected appearance serves as an indicator in the purchase process.

[0124] Step 5:

[0125] The server sends product information related to the selected appearance to the terminal, which then displays this information to the user. At this stage, the product price, detailed information, and purchase availability are provided as output.

[0126] Step 6:

[0127] The user selects the product they wish to purchase and completes the purchase process via their device. Based on the user's selection, the purchase information is securely transmitted to the server, and the purchase history is updated. This is the final transaction processing step.

[0128] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0129] This invention not only provides a style that suits the user, but also includes a system configuration that incorporates user emotion recognition. This system operates by combining a terminal including a camera, a server that manages style and emotion information, and an emotion engine that appropriately determines reactions.

[0130] emotion recognition

[0131] When the device scans the user's face, it analyzes not only facial feature data but also the user's micro-expressions and skin condition using an emotion analysis algorithm. The emotion engine responsible for this process identifies various emotional states, such as joy, surprise, and anxiety, in real time based on the characteristics of the user's facial expressions. This data is sent to a server and used in the style recommendation process.

[0132] Recommended Style

[0133] The server considers the user's facial features and emotional data to select the most suitable makeup and hairstyle from the database. For example, if the user has a tense expression, it can recommend styles and makeup that are expected to have a relaxing effect. This makes the user experience more emotionally responsive and provides a more personalized service.

[0134] Virtual simulation and emotional feedback

[0135] The device virtually displays the selected style over the user's face. At this point, the emotion engine operates again to evaluate the user's reaction. The evaluation results are used to re-select styles and recommend products. For example, styles that the user expressed pleasure with will be given priority in subsequent recommendations.

[0136] Product information provision and purchase support

[0137] Product information related to the style selected by the user is displayed, including recommendation reasons that match their emotional state. Users can then proceed with the online purchase process based on the product information. The server stores purchase history data and uses it to improve recommendation accuracy in the future.

[0138] As a concrete example, when user B changes to an interesting hairstyle while using the system, the terminal instantly reflects the change, and the emotion engine analyzes user B's micro-expressions. If user B shows a happy reaction at this time, products related to that hairstyle are recommended in more detail. This allows user B to confidently select and purchase styles and products that fit their emotions.

[0139] Thus, the present invention provides a concrete example of a system that provides users with more satisfying personalized services by combining the analysis of emotional information.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The device activates its built-in camera and captures the user's face in real time. It analyzes not only facial features but also micro-expressions to recognize emotions.

[0143] Step 2:

[0144] The device sends extracted facial feature data and emotion data to the server. The server receives this data, activates the emotion engine, and identifies the user's current emotional state.

[0145] Step 3:

[0146] The server accesses the database and recommends the style that best matches the user's facial features and emotional state. The recommended style takes the user's emotional state into consideration, and the one that best suits a specific emotion is selected.

[0147] Step 4:

[0148] The server sends back the selected style information to the terminal. The terminal receives this information and virtually applies the style to the user's face in real time, visualizing it.

[0149] Step 5:

[0150] The emotion engine is restarted on the device to analyze how the user is reacting to the virtual simulation. Emotional feedback is sent to the server and used to improve the recommended style.

[0151] Step 6:

[0152] The server selects the most suitable product information based on the user's emotional feedback and purchase history, and sends it to the device. The product information includes the reasons for the recommendation and is presented visually to the user.

[0153] Step 7:

[0154] The user selects their preferred product and completes the purchase process via their device. The server securely processes the purchase information, stores it in a database as a history, and uses it to improve recommendations for future purchases.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] Traditional style recommendation systems performed style analysis based on the user's facial features, but they could not take into account the user's emotional state, making it difficult to provide the optimal style for each individual user. Furthermore, they could not fully utilize the user's selection history or reactions, resulting in limited information for providing a personalized experience.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for analyzing the style based on the user's facial features and emotional state, means for acquiring emotional feedback and reflecting it in the user's image in real time, and means for utilizing emotional information and linking with a database. This makes it possible to take the user's emotional state into consideration and provide a personalized style that is optimal for each individual user.

[0160] "Facial features" refer to the specific patterns and shapes that make up a user's face, and are fundamental data used in selecting a style.

[0161] "Emotional state" refers to the psychological state obtained from the user's facial expressions and behavior, and includes specific emotional responses such as joy, surprise, and anxiety.

[0162] "Style" refers to the aesthetic or functional appearance recommended to the user, and includes choices of appearance such as hairstyle and makeup.

[0163] "Analysis" is the process of collecting data from the user's facial features and emotional state, and using that data to identify an optimized style.

[0164] A "database" is an information processing system that systematically stores information necessary for style recommendations and allows for quick access when needed.

[0165] "Emotional feedback" refers to information collected through an emotional engine, which reflects the user's reaction to recommended styles and is used to improve style selection in the future.

[0166] "Real-time reflection" refers to the process of instantly overlaying the analyzed style onto the user's video, allowing for immediate confirmation.

[0167] This invention is a system that provides users with emotion-based style suggestions. This system primarily consists of a terminal, a server, and various analysis engines.

[0168] terminal

[0169] The device is a piece of equipment that includes a camera and a display. The device's camera captures the user's face and acquires image data. This image data is sent in real time to the emotion engine, which analyzes the user's micro-expressions and skin condition. Image processing technologies such as the OpenCV library are used for the analysis.

[0170] server

[0171] The server receives emotion data and facial feature data sent from the terminal and selects the optimal style based on this data. The database stores past user history and various style information, and the server refers to this database to recommend styles while utilizing emotion information. Specifically, databases such as MongoDB and MySQL (registered trademark) can be used.

[0172] Emotion engine and virtual simulation

[0173] The emotion engine analyzes the user's emotional state in real time and plays a role in obtaining emotional feedback. This feedback information is reflected in future style recommendations. The selected style is virtually overlaid on the user's face using AR technology on the device's display. Available platforms include Unity and Vuforia.

[0174] Concrete examples of user experience

[0175] For example, consider a scenario where a user wants to try a new makeup style and uses the system. The device immediately captures the user's face with its camera and analyzes it using an emotion engine. If the user is relaxed, the server selects a suitable makeup style from its database and displays it virtually on the device. All of these actions occur in real time.

[0176] An example of a prompt message would be, "Recommend the optimal style in real time based on the user's facial features and emotion data." This allows the user experience to be highly personalized to the individual's emotions.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The device captures the user's face with its camera. The user's face image is acquired as input and sent to the emotion engine. The image data is processed based on detected facial features to generate micro-expression data. This process utilizes image recognition technology and tools such as the OpenCV library.

[0180] Step 2:

[0181] The device uses an emotion engine to analyze the user's emotions from acquired image data. It receives facial feature data as input and generates emotion data as output through analysis. This analysis process uses algorithms that analyze microexpressions and skin condition in detail, allowing the user's emotional state (e.g., joy, surprise) to be identified in real time.

[0182] Step 3:

[0183] The device sends the analyzed emotion data to the server. Here, the emotion data is received from the emotion engine and sent to the server for comparison with the database. To ensure security, the data is encrypted before transmission. The input is emotion data, and the output is a database lookup request on the server side.

[0184] Step 4:

[0185] The server selects the optimal style from a database based on the user's emotional data and facial features. It takes emotional data and facial feature data as input and generates recommended styles as output. In this step, it refers to similar user patterns and past style selection history, and uses a database management system (e.g., MongoDB) to select the optimal makeup and hairstyle.

[0186] Step 5:

[0187] The server sends the selected style information to the terminal. Here, the recommended style is retrieved from the server and sent to the user's display device in real time. The recommended style data is received from the server as input, and style information is provided to the terminal as output. Through this process, the user can see a virtual simulation of the recommended style.

[0188] Step 6:

[0189] The device overlays the selected style onto the user's face and obtains emotional feedback again. It receives recommended style data as input and virtually overlays it onto the user's face. The output generates a virtual simulation display and emotional feedback data. The emotion engine re-evaluates the feedback, and the feedback is used for future recommendations.

[0190] Step 7:

[0191] The device provides the user with the final style and related product information, and supports their purchase. It receives style feedback as input and presents relevant product information. It generates purchase instructions as output. In this step, the user can confidently select and purchase products while understanding the reasons for the recommendations.

[0192] (Application Example 2)

[0193] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0194] Traditional style recommendation systems determine styles based solely on the user's facial features, which limits their ability to provide personalized recommendations that take into account the user's emotional state. Furthermore, there is a lack of style evaluation and product recommendations that utilize real-time emotional responses, highlighting the need for improved user experience.

[0195] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0196] In this invention, the server includes means for recognizing the user's facial features and emotional state, and analyzing a style based on those features and state; means for reflecting the analyzed style on the user's image in real time and evaluating the emotional response; means for linking with a personal information database for recommending a style suitable for the user based on the evaluation results; and means for displaying product information related to the style selected by the user, presenting recommendation reasons based on emotions, and facilitating the ordering process. This makes it possible to provide highly accurate style recommendations to users that take emotions into consideration, and a highly satisfying purchasing experience.

[0197] "Facial features" refer to the unique shape and arrangement information present in a user's face, and are data used for style analysis.

[0198] "Emotional state" refers to information that indicates the inner feelings and psychological state obtained from the user's facial expressions and actions.

[0199] "Style" refers to external elements such as hairstyle, makeup, and clothing applied to a user, and is selected based on individual preferences and feelings.

[0200] "Real-time" means that information processing and the reflection of results are performed instantly without delay.

[0201] "Evaluation results" are conclusions regarding the appropriateness and suitability of a particular style or product, derived from user response data analyzed by the emotion engine.

[0202] A "personal information database" is a source of information that stores data on users' facial features, emotional history, and style preferences, and is used for style recommendations.

[0203] "Product information" refers to detailed information about the product name, features, price, and purchase process related to the style the user has shown interest in.

[0204] The "order process" refers to the series of steps required when a user purchases a selected product, including confirmation, payment, and shipping.

[0205] This invention is a system that uses a smart mirror installed in stores or homes to recommend styles to users. The server captures the user's facial features and emotional state through a built-in camera and analyzes their style based on this data. The analysis uses a facial recognition library (such as OpenCV) and an emotion analysis engine. The emotion engine evaluates the user's micro-expressions in real time, and the server stores the evaluation results. As a result, the most suitable fashion items and related products are selected from the database and displayed on the screen as a virtual try-on experience. The display follows the user's movements and smoothly provides try-on results and product information. The user can check product information related to the styles they are interested in and, if necessary, proceed with the ordering process on the smart mirror.

[0206] As a concrete example, when a user tries on their favorite style in front of a smart mirror, the system analyzes the user's micro-expressions to detect feelings of happiness and curiosity. As a result, it recommends casual and relaxing clothing to the user, creating a more satisfying shopping experience. An example of a prompt to the generative AI model is, "Design an algorithm for a virtual try-on app that analyzes the user's micro-expressions and suggests casual clothing if they appear happy."

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The device uses its built-in camera to capture the user's facial features and emotional state. The input is image data of the user from the front, captured by the camera, and the output is facial feature data and micro-expression data. This data is analyzed using a facial recognition library (such as OpenCV).

[0210] Step 2:

[0211] The server receives facial feature data and microexpression data sent from the terminal. The input is data from the terminal, and the output is the emotion analysis result. The server uses an emotion analysis engine to identify emotions such as joy and surprise from the user's microexpressions.

[0212] Step 3:

[0213] The server initiates style recommendations based on the sentiment analysis results. The input is the sentiment analysis data, and the output is style information suitable for the user. The server interacts with a person information database to search for styles that match the analysis results.

[0214] Step 4:

[0215] The terminal receives style information sent from the server and provides the user with a virtual try-on experience on the display. The input is style information, and the output is a virtual try-on image displayed on the screen. The display follows the user's movements in real time to support the try-on experience.

[0216] Step 5:

[0217] The user reviews the virtual try-on experience and expresses emotional responses to styles that interest them. The input is the user's micro-facial expressions, and the output is the emotional analysis data. The server performs the analysis again via the emotion engine to evaluate the user's satisfaction level.

[0218] Step 6:

[0219] The server provides users with relevant product information based on the evaluation results. The input is sentiment evaluation data, and the output is product information and reasons for recommendation. Based on this, the user proceeds with the order process and completes the purchase on their terminal.

[0220] Step 7:

[0221] User purchase history and sentiment data are stored on the server to be used to improve future style recommendations. The input is purchase and sentiment history data, and the output is an updated database.

[0222] This entire process allows us to optimize the user experience and provide personalized styles and products.

[0223] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0224] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0225] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0226] [Second Embodiment]

[0227] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0228] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0229] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0230] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0231] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0233] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0234] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0235] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0236] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0237] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0238] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0239] This invention presents an embodiment of a system that allows users to easily find makeup and hairstyles that suit them. This system uses facial recognition technology to scan the user's facial features and optimizes the style based on that data. The specific implementation of each element is described below.

[0240] User facial recognition

[0241] The device has a built-in high-resolution camera that automatically activates when the user stands in front of the smart mirror. The device uses an advanced facial recognition algorithm to capture the user's facial features in real time and analyze the contours, eyes, nose, and mouth position and shape. This data is sent to a server and used for further analysis.

[0242] Style recommendations and displays

[0243] The server uses the received facial feature data to match it against a large style database. This allows it to suggest multiple optimal makeup and hairstyle styles for the user. The suggested styles are displayed on the device's screen, and the user can select and try different styles using touch controls.

[0244] Virtual Simulation

[0245] The device overlays the selected style onto the user's face in real time, visualizing virtual makeup and hairstyles. This simulation follows the user's movements, allowing for viewing from different angles. The system utilizes GPU-driven image processing to achieve this.

[0246] Product recommendation and purchase processing

[0247] Products matching the user's selected style are fed back to the server and displayed as a list on the device. If the user wishes to purchase a product, they can proceed with the online order process through the device, and payment information is sent to the server via a secure communication method. The server records this information and stores the purchase history in a database.

[0248] As a concrete example, consider a scenario where User A stands in front of a smart mirror and tries to change their hairstyle. User A's face is recognized, and several hairstyle styles are recommended from the server. User A selects their favorite hairstyle, and that hairstyle is virtually simulated. During this time, User A can check whether it suits their face from various angles, and if they like it, they can purchase related products on the spot.

[0249] Through this process, users can easily try out new styles and purchase related products.

[0250] The following describes the processing flow.

[0251] Step 1:

[0252] The device activates its camera and captures a video of the user's face. When the user stands in front of the smart mirror, it automatically recognizes the face area and extracts feature data.

[0253] Step 2:

[0254] The server receives the user's facial feature data sent from the terminal. Based on this, it compares it with data in a pre-built style database and selects a style that suits the user.

[0255] Step 3:

[0256] The server generates a list of selected makeup and hairstyle styles and sends this information to the terminal. The terminal displays these recommended styles to the user and prompts them to make a selection.

[0257] Step 4:

[0258] The user selects a style they are interested in, and the device overlays the selected style onto the user's face in real time. This allows the user to see how the style will look on their own face.

[0259] Step 5:

[0260] The server analyzes the user's selections and creates an optimized list of product recommendations. This product information is sent to the terminal and presented to the user.

[0261] Step 6:

[0262] The user selects a product they like and expresses their intention to purchase it via their device. The device then uses secure communication with the server to process the order and verify payment information.

[0263] Step 7:

[0264] The server processes purchase information and stores it in the database as purchase history. This allows for more personalized suggestions to the user the next time they use the service.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] There is a problem in that it is difficult for individual users to easily and effectively find the style that is best suited to them. Furthermore, there is a lack of a process for users to easily purchase products related to their chosen style, as they cannot see how the style will actually look in real time.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for identifying the features of the user's face and analyzing expressions based on those features, means for reflecting the analyzed expressions on the user's video in real time, and means for coordinating with a data storage device for recommending expressions suitable for the user. This enables the user to quickly find the style that is best suited to them, confirm that style in real time, and smoothly purchase related products.

[0270] A "user" refers to an individual who uses this system to select and experiment with styles.

[0271] "Facial features" refer to data about the position and shape of an individual's eyes, nose, mouth, and facial contours.

[0272] "Expression" refers to styles and designs that change a user's appearance, such as makeup and hairstyles.

[0273] "Identification" refers to the process of extracting and recognizing specific features from images or data using a particular algorithm.

[0274] "Analysis" is the process of thoroughly examining collected data, extracting information, and understanding it.

[0275] A "generative AI model" refers to an algorithm or program that uses AI technology to automatically create new designs and styles.

[0276] A "data storage device" refers to a system for systematically storing and managing multiple styles and product information.

[0277] "Real-time" refers to the processing and reflection of information occurring almost instantaneously or in a very short time.

[0278] "This invention" refers to a system designed to enable users to select and experience styles that are optimized for them.

[0279] A mode for implementing the present invention will be described.

[0280] This system is designed for users to easily find styles that suit them. The terminal is equipped with a high-resolution camera and a display. When the user stands in front of the device, it automatically captures the user's face and identifies facial features. This utilizes advanced face recognition algorithms to analyze the contours, positions, and shapes of the eyes, nose, and mouth in real time.

[0281] This analysis data is sent to the server, and the server uses a generative AI model to generate multiple styles based on the data. The server compares these styles with a large-scale data storage device and recommends the style most suitable for the user. This recommendation is personalized based on the user's individual characteristics.

[0282] The terminal displays the recommended styles on the display, and the user can try the styles through touch operations. The terminal superimposes and simulates the selected style on the user's face in real time. This process is realized by the terminal's GPU technology, and the user can view the style from various angles while moving.

[0283] Furthermore, the server obtains product information related to the selected style from the data storage device and displays it as a list on the terminal. The user can purchase related products based on this information. The purchase procedure is completed online through the terminal, and the payment information is securely sent to the server, and the purchase history is recorded in the database.

[0284] As a specific example, when User A wants to try a new hairstyle using this system, the hairstyle style is simulated on the face in real time. If they like it, the corresponding care products will be immediately displayed and available for purchase.

[0285] Examples of prompt sentences include: "Using facial recognition technology, please propose makeup and hairstyle styles that suit me. Virtually simulate them in real time and display a list of related products."

[0286] In this way, the user can easily and efficiently try styles that suit them and seamlessly purchase related products.

[0287] The flow of the specific process in Example 1 will be described using Figure 11.

[0288] Step 1:

[0289] When the user stands in front of the device, the terminal automatically activates the high-resolution camera. The camera captures the user's face and acquires image data. Based on this input data, the terminal uses a facial recognition algorithm to identify the features of the user's face and specify the positions and shapes of the contours, eyes, nose, and mouth. As a result, the analyzed feature data is output.

[0290] Step 2:

[0291] The server that receives the facial feature data transmitted from the terminal inputs this data into the generation AI model. The server uses the data to generate the optimal style for the user and outputs a plurality of styles that can be proposed. The server thereby creates a unique style and prepares for making an optimal proposal. <000*******> Step 3:

[0293] The server compares the generated style with the style data stored in the data storage device. Through this process, the server identifies the style most suitable for the user and outputs style information with priorities.

[0294] Step 4:

[0295] Upon receiving style information from the server, the terminal displays these styles on its screen. Based on this display, the user selects the style they wish to try via the touchscreen. This selection process then passes the selected style data to the next stage.

[0296] Step 5:

[0297] The device overlays the selected style onto the user's face in real time, displaying it for the user to see. The GPU is used to perform style simulations, dynamically adjusting the style to different angles and lighting conditions. The output is a real-time visual simulation.

[0298] Step 6:

[0299] Based on the user's interests, the server retrieves relevant product information from the data storage device and sends it to the terminal. The output provides a list of products related to the desired style, allowing the user to make a purchase selection.

[0300] Step 7:

[0301] If a user wishes to purchase a selected product, the terminal initiates the ordering process. Payment information is sent to the server via secure communication, which receives this information and records it in the purchase history database. As a result, the online purchase is completed, and the purchase history is updated.

[0302] (Application Example 1)

[0303] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0304] In the conventional beauty experience, users had to spend a lot of time and effort to try out a new appearance, and the process until actually purchasing the products for the selected appearance was complicated. Therefore, there is a need for an efficient system that allows users to easily try out a new appearance and purchase related products on the spot.

[0305] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0306] In this invention, the server includes means for recognizing the facial features of a user, analyzing the appearance based on the features, reflecting the analyzed appearance in real time on the user's video, and displaying product information related to the appearance selected by the user and performing a purchase procedure. As a result, in a physical store, users can interactively try out a new appearance and efficiently purchase related products on the spot.

[0307] A "user" is an individual who operates the system and aims to try out an appearance suitable for themselves.

[0308] "Facial features" refer to the physical characteristics of a user's face, such as the shape, contour, eyes, nose, mouth, etc.

[0309] "Appearance" refers to styles such as makeup and hairstyles that users can try.

[0310] "Information collection" is a database used to provide options for appearances recommended to users.

[0311] "Product information" refers to information on products and services related to the appearance selected by the user.

[0312] "Purchase procedure" is a series of processes in which a user orders a product selected online and completes the payment.

[0313] A "physical store" refers to a shopping facility or beauty salon existing in a physical location.

[0314] An "interactive experience" is a process in which users can interact with the system in both directions, try out the appearance in real time, and make a purchase.

[0315] The system for implementing this invention mainly consists of interactions between a server, a terminal, and a user.

[0316] The server analyzes the user's facial features using advanced facial recognition algorithms. Specifically, it processes the user's facial image captured by a high-resolution camera, identifying the contours and positions of the eyes, nose, and mouth. Based on this data, the server works in conjunction with an information database to generate a selection of options to suggest the most suitable appearance for the user.

[0317] The terminal incorporates a mirror-type display and a camera. The terminal overlays suggested appearances received from the server onto the user's video in real time, displaying them interactively. Through the display, users can try out different styles and instantly view product information related to their selected appearance.

[0318] When a user selects a specific appearance, the system displays relevant product information on the terminal, allowing them to proceed with the purchase immediately. Once the user completes the payment, the information is securely transmitted to the server, and the purchase history is recorded.

[0319] As a concrete example, imagine a customer experiencing this interactive system for the first time in a department store's cosmetics counter. The customer stands in front of a smart mirror, tries out several makeup styles, and by selecting one, can order related cosmetics on the spot. This process takes only a few minutes from start to finish, providing the customer with an efficient and intuitive shopping experience.

[0320] Furthermore, the generative AI model associated with this system can generate new styles using prompt statements like the following.

[0321] "Design an interactive system that uses facial recognition technology to suggest recommended makeup and hairstyles based on the user's facial features in real time, and allows users to purchase related products."

[0322] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0323] Step 1:

[0324] The server receives a user's facial image sent from the terminal. Based on this input image, it uses a facial recognition algorithm to analyze facial features such as the eyes, nose, and mouth, and outputs their coordinate data. This facial data is basic information necessary for subsequent style recommendation.

[0325] Step 2:

[0326] The server uses facial feature data to match it with a database and creates a list of appearances suitable for the user. The input consists of facial feature data and user preference data, which are then used to output candidate appearance styles, which are sent to the terminal.

[0327] Step 3:

[0328] The terminal displays style candidates received from the server, overlaid on the user's video in real time. The input consists of style data and the user's video, and multiple styles are applied to the displayed video, allowing the user to visually try them out.

[0329] Step 4:

[0330] Users select their preferred appearance style through the device's interface. This selection information is sent to a server, where product information related to that appearance is collected. The selected appearance serves as an indicator in the purchase process.

[0331] Step 5:

[0332] The server sends product information related to the selected appearance to the terminal, which then displays this information to the user. At this stage, the product price, detailed information, and purchase availability are provided as output.

[0333] Step 6:

[0334] The user selects the product they wish to purchase and completes the purchase process via their device. Based on the user's selection, the purchase information is securely transmitted to the server, and the purchase history is updated. This is the final transaction processing step.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] This invention not only provides a style that suits the user, but also includes a system configuration that incorporates user emotion recognition. This system operates by combining a terminal including a camera, a server that manages style and emotion information, and an emotion engine that appropriately determines reactions.

[0337] emotion recognition

[0338] When the device scans the user's face, it analyzes not only facial feature data but also the user's micro-expressions and skin condition using an emotion analysis algorithm. The emotion engine responsible for this process identifies various emotional states, such as joy, surprise, and anxiety, in real time based on the characteristics of the user's facial expressions. This data is sent to a server and used in the style recommendation process.

[0339] Recommended Style

[0340] The server considers the user's facial features and emotional data to select the most suitable makeup and hairstyle from the database. For example, if the user has a tense expression, it can recommend styles and makeup that are expected to have a relaxing effect. This makes the user experience more emotionally responsive and provides a more personalized service.

[0341] Virtual simulation and emotional feedback

[0342] The device virtually displays the selected style over the user's face. At this point, the emotion engine operates again to evaluate the user's reaction. The evaluation results are used to re-select styles and recommend products. For example, styles that the user expressed pleasure with will be given priority in subsequent recommendations.

[0343] Product information provision and purchase support

[0344] Product information related to the style selected by the user is displayed, including recommendation reasons that match their emotional state. Users can then proceed with the online purchase process based on the product information. The server stores purchase history data and uses it to improve recommendation accuracy in the future.

[0345] As a concrete example, when user B changes to an interesting hairstyle while using the system, the terminal instantly reflects the change, and the emotion engine analyzes user B's micro-expressions. If user B shows a happy reaction at this time, products related to that hairstyle are recommended in more detail. This allows user B to confidently select and purchase styles and products that fit their emotions.

[0346] Thus, the present invention provides a concrete example of a system that provides users with more satisfying personalized services by combining the analysis of emotional information.

[0347] The following describes the processing flow.

[0348] Step 1:

[0349] The device activates its built-in camera and captures the user's face in real time. It analyzes not only facial features but also micro-expressions to recognize emotions.

[0350] Step 2:

[0351] The device sends extracted facial feature data and emotion data to the server. The server receives this data, activates the emotion engine, and identifies the user's current emotional state.

[0352] Step 3:

[0353] The server accesses the database and recommends the style that best matches the user's facial features and emotional state. The recommended style takes the user's emotional state into consideration, and the one that best suits a specific emotion is selected.

[0354] Step 4:

[0355] The server sends back the selected style information to the terminal. The terminal receives this information and virtually applies the style to the user's face in real time, visualizing it.

[0356] Step 5:

[0357] The emotion engine is restarted on the device to analyze how the user is reacting to the virtual simulation. Emotional feedback is sent to the server and used to improve the recommended style.

[0358] Step 6:

[0359] The server selects the most suitable product information based on the user's emotional feedback and purchase history, and sends it to the device. The product information includes the reasons for the recommendation and is presented visually to the user.

[0360] Step 7:

[0361] The user selects their preferred product and completes the purchase process via their device. The server securely processes the purchase information, stores it in a database as a history, and uses it to improve recommendations for future purchases.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] Traditional style recommendation systems performed style analysis based on the user's facial features, but they could not take into account the user's emotional state, making it difficult to provide the optimal style for each individual user. Furthermore, they could not fully utilize the user's selection history or reactions, resulting in limited information for providing a personalized experience.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes means for analyzing the style based on the user's facial features and emotional state, means for acquiring emotional feedback and reflecting it in the user's image in real time, and means for utilizing emotional information and linking with a database. This makes it possible to take the user's emotional state into consideration and provide a personalized style that is optimal for each individual user.

[0367] "Facial features" refer to the specific patterns and shapes that make up a user's face, and are fundamental data used in selecting a style.

[0368] "Emotional state" refers to the psychological state obtained from the user's facial expressions and behavior, and includes specific emotional responses such as joy, surprise, and anxiety.

[0369] "Style" refers to the aesthetic or functional appearance recommended to the user, and includes choices of appearance such as hairstyle and makeup.

[0370] "Analysis" is the process of collecting data from the user's facial features and emotional state, and using that data to identify an optimized style.

[0371] A "database" is an information processing system that systematically stores information necessary for style recommendations and allows for quick access when needed.

[0372] "Emotional feedback" refers to information collected through an emotional engine, which reflects the user's reaction to recommended styles and is used to improve style selection in the future.

[0373] "Real-time reflection" refers to the process of instantly overlaying the analyzed style onto the user's video, allowing for immediate confirmation.

[0374] This invention is a system that provides users with emotion-based style suggestions. This system primarily consists of a terminal, a server, and various analysis engines.

[0375] terminal

[0376] The device is a piece of equipment that includes a camera and a display. The device's camera captures the user's face and acquires image data. This image data is sent in real time to the emotion engine, which analyzes the user's micro-expressions and skin condition. Image processing technologies such as the OpenCV library are used for the analysis.

[0377] server

[0378] The server receives emotion data and facial feature data sent from the terminal and selects the optimal style based on this data. The database stores past user history and various style information, and the server refers to this database to recommend styles while utilizing emotion information. Specifically, databases such as MongoDB and MySQL can be used.

[0379] Emotion engine and virtual simulation

[0380] The emotion engine analyzes the user's emotional state in real time and plays a role in obtaining emotional feedback. This feedback information is reflected in future style recommendations. The selected style is virtually overlaid on the user's face using AR technology on the device's display. Available platforms include Unity and Vuforia.

[0381] Concrete examples of user experience

[0382] For example, consider a scenario where a user wants to try a new makeup style and uses the system. The device immediately captures the user's face with its camera and analyzes it using an emotion engine. If the user is relaxed, the server selects a suitable makeup style from its database and displays it virtually on the device. All of these actions occur in real time.

[0383] An example of a prompt message would be, "Recommend the optimal style in real time based on the user's facial features and emotion data." This allows the user experience to be highly personalized to the individual's emotions.

[0384] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0385] Step 1:

[0386] The device captures the user's face with its camera. The user's face image is acquired as input and sent to the emotion engine. The image data is processed based on detected facial features to generate micro-expression data. This process utilizes image recognition technology and tools such as the OpenCV library.

[0387] Step 2:

[0388] The device uses an emotion engine to analyze the user's emotions from acquired image data. It receives facial feature data as input and generates emotion data as output through analysis. This analysis process uses algorithms that analyze microexpressions and skin condition in detail, allowing the user's emotional state (e.g., joy, surprise) to be identified in real time.

[0389] Step 3:

[0390] The device sends the analyzed emotion data to the server. Here, the emotion data is received from the emotion engine and sent to the server for comparison with the database. To ensure security, the data is encrypted before transmission. The input is emotion data, and the output is a database lookup request on the server side.

[0391] Step 4:

[0392] The server selects the optimal style from a database based on the user's emotional data and facial features. It takes emotional data and facial feature data as input and generates recommended styles as output. In this step, it refers to similar user patterns and past style selection history, and uses a database management system (e.g., MongoDB) to select the optimal makeup and hairstyle.

[0393] Step 5:

[0394] The server sends the selected style information to the terminal. Here, the recommended style is retrieved from the server and sent to the user's display device in real time. The recommended style data is received from the server as input, and style information is provided to the terminal as output. Through this process, the user can see a virtual simulation of the recommended style.

[0395] Step 6:

[0396] The device overlays the selected style onto the user's face and obtains emotional feedback again. It receives recommended style data as input and virtually overlays it onto the user's face. The output generates a virtual simulation display and emotional feedback data. The emotion engine re-evaluates the feedback, and the feedback is used for future recommendations.

[0397] Step 7:

[0398] The device provides the user with the final style and related product information, and supports their purchase. It receives style feedback as input and presents relevant product information. It generates purchase instructions as output. In this step, the user can confidently select and purchase products while understanding the reasons for the recommendations.

[0399] (Application Example 2)

[0400] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0401] Traditional style recommendation systems determine styles based solely on the user's facial features, which limits their ability to provide personalized recommendations that take into account the user's emotional state. Furthermore, there is a lack of style evaluation and product recommendations that utilize real-time emotional responses, highlighting the need for improved user experience.

[0402] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0403] In this invention, the server includes means for recognizing the user's facial features and emotional state, and analyzing a style based on those features and state; means for reflecting the analyzed style on the user's image in real time and evaluating the emotional response; means for linking with a personal information database for recommending a style suitable for the user based on the evaluation results; and means for displaying product information related to the style selected by the user, presenting recommendation reasons based on emotions, and facilitating the ordering process. This makes it possible to provide highly accurate style recommendations to users that take emotions into consideration, and a highly satisfying purchasing experience.

[0404] "Facial features" refer to the unique shape and arrangement information present in a user's face, and are data used for style analysis.

[0405] "Emotional state" refers to information that indicates the inner feelings and psychological state obtained from the user's facial expressions and actions.

[0406] "Style" refers to external elements such as hairstyle, makeup, and clothing applied to a user, and is selected based on individual preferences and feelings.

[0407] "Real-time" means that information processing and the reflection of results are performed instantly without delay.

[0408] "Evaluation results" are conclusions regarding the appropriateness and suitability of a particular style or product, derived from user response data analyzed by the emotion engine.

[0409] A "personal information database" is a source of information that stores data on users' facial features, emotional history, and style preferences, and is used for style recommendations.

[0410] "Product information" refers to detailed information about the product name, features, price, and purchase process related to the style the user has shown interest in.

[0411] The "order process" refers to the series of steps required when a user purchases a selected product, including confirmation, payment, and shipping.

[0412] This invention is a system that uses a smart mirror installed in stores or homes to recommend styles to users. The server captures the user's facial features and emotional state through a built-in camera and analyzes their style based on this data. The analysis uses a facial recognition library (such as OpenCV) and an emotion analysis engine. The emotion engine evaluates the user's micro-expressions in real time, and the server stores the evaluation results. As a result, the most suitable fashion items and related products are selected from the database and displayed on the screen as a virtual try-on experience. The display follows the user's movements and smoothly provides try-on results and product information. The user can check product information related to the styles they are interested in and, if necessary, proceed with the ordering process on the smart mirror.

[0413] As a concrete example, when a user tries on their favorite style in front of a smart mirror, the system analyzes the user's micro-expressions to detect feelings of happiness and curiosity. As a result, it recommends casual and relaxing clothing to the user, creating a more satisfying shopping experience. An example of a prompt to the generative AI model is, "Design an algorithm for a virtual try-on app that analyzes the user's micro-expressions and suggests casual clothing if they appear happy."

[0414] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0415] Step 1:

[0416] The device uses its built-in camera to capture the user's facial features and emotional state. The input is image data of the user from the front, captured by the camera, and the output is facial feature data and micro-expression data. This data is analyzed using a facial recognition library (such as OpenCV).

[0417] Step 2:

[0418] The server receives facial feature data and microexpression data sent from the terminal. The input is data from the terminal, and the output is the emotion analysis result. The server uses an emotion analysis engine to identify emotions such as joy and surprise from the user's microexpressions.

[0419] Step 3:

[0420] The server initiates style recommendations based on the sentiment analysis results. The input is the sentiment analysis data, and the output is style information suitable for the user. The server interacts with a person information database to search for styles that match the analysis results.

[0421] Step 4:

[0422] The terminal receives style information sent from the server and provides the user with a virtual try-on experience on the display. The input is style information, and the output is a virtual try-on image displayed on the screen. The display follows the user's movements in real time to support the try-on experience.

[0423] Step 5:

[0424] The user reviews the virtual try-on experience and expresses emotional responses to styles that interest them. The input is the user's micro-facial expressions, and the output is the emotional analysis data. The server performs the analysis again via the emotion engine to evaluate the user's satisfaction level.

[0425] Step 6:

[0426] The server provides users with relevant product information based on the evaluation results. The input is sentiment evaluation data, and the output is product information and reasons for recommendation. Based on this, the user proceeds with the order process and completes the purchase on their terminal.

[0427] Step 7:

[0428] User purchase history and sentiment data are stored on the server to be used to improve future style recommendations. The input is purchase and sentiment history data, and the output is an updated database.

[0429] This entire process allows us to optimize the user experience and provide personalized styles and products.

[0430] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0431] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0432] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0433] [Third Embodiment]

[0434] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0435] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0436] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0437] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0438] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0440] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0441] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0442] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0443] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0444] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0445] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0446] This invention presents an embodiment of a system that allows users to easily find makeup and hairstyles that suit them. This system uses facial recognition technology to scan the user's facial features and optimizes the style based on that data. The specific implementation of each element is described below.

[0447] User facial recognition

[0448] The device has a built-in high-resolution camera that automatically activates when the user stands in front of the smart mirror. The device uses an advanced facial recognition algorithm to capture the user's facial features in real time and analyze the contours, eyes, nose, and mouth position and shape. This data is sent to a server and used for further analysis.

[0449] Style recommendations and displays

[0450] The server uses the received facial feature data to match it against a large style database. This allows it to suggest multiple optimal makeup and hairstyle styles for the user. The suggested styles are displayed on the device's screen, and the user can select and try different styles using touch controls.

[0451] Virtual Simulation

[0452] The device overlays the selected style onto the user's face in real time, visualizing virtual makeup and hairstyles. This simulation follows the user's movements, allowing for viewing from different angles. The system utilizes GPU-driven image processing to achieve this.

[0453] Product recommendation and purchase processing

[0454] Products matching the user's selected style are fed back to the server and displayed as a list on the device. If the user wishes to purchase a product, they can proceed with the online order process through the device, and payment information is sent to the server via a secure communication method. The server records this information and stores the purchase history in a database.

[0455] As a concrete example, consider a scenario where User A stands in front of a smart mirror and tries to change their hairstyle. User A's face is recognized, and several hairstyle styles are recommended from the server. User A selects their favorite hairstyle, and that hairstyle is virtually simulated. During this time, User A can check whether it suits their face from various angles, and if they like it, they can purchase related products on the spot.

[0456] Through this process, users can easily try out new styles and purchase related products.

[0457] The following describes the processing flow.

[0458] Step 1:

[0459] The device activates its camera and captures a video of the user's face. When the user stands in front of the smart mirror, it automatically recognizes the face area and extracts feature data.

[0460] Step 2:

[0461] The server receives the user's facial feature data sent from the terminal. Based on this, it compares it with data in a pre-built style database and selects a style that suits the user.

[0462] Step 3:

[0463] The server generates a list of selected makeup and hairstyle styles and sends this information to the terminal. The terminal displays these recommended styles to the user and prompts them to make a selection.

[0464] Step 4:

[0465] The user selects a style they are interested in, and the device overlays the selected style onto the user's face in real time. This allows the user to see how the style will look on their own face.

[0466] Step 5:

[0467] The server analyzes the user's selections and creates an optimized list of product recommendations. This product information is sent to the terminal and presented to the user.

[0468] Step 6:

[0469] The user selects a product they like and expresses their intention to purchase it via their device. The device then uses secure communication with the server to process the order and verify payment information.

[0470] Step 7:

[0471] The server processes purchase information and stores it in the database as purchase history. This allows for more personalized suggestions to the user the next time they use the service.

[0472] (Example 1)

[0473] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0474] There is a problem in that it is difficult for individual users to easily and effectively find the style that is best suited to them. Furthermore, there is a lack of a process for users to easily purchase products related to their chosen style, as they cannot see how the style will actually look in real time.

[0475] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0476] In this invention, the server includes means for identifying the features of the user's face and analyzing expressions based on those features, means for reflecting the analyzed expressions on the user's video in real time, and means for coordinating with a data storage device for recommending expressions suitable for the user. This enables the user to quickly find the style that is best suited to them, confirm that style in real time, and smoothly purchase related products.

[0477] A "user" refers to an individual who uses this system to select and experiment with styles.

[0478] "Facial features" refer to data about the position and shape of an individual's eyes, nose, mouth, and facial contours.

[0479] "Expression" refers to styles and designs that change a user's appearance, such as makeup and hairstyles.

[0480] "Identification" refers to the process of extracting and recognizing specific features from images or data using a particular algorithm.

[0481] "Analysis" is the process of thoroughly examining collected data, extracting information, and understanding it.

[0482] A "generative AI model" refers to an algorithm or program that uses AI technology to automatically create new designs and styles.

[0483] A "data storage device" refers to a system for systematically storing and managing multiple styles and product information.

[0484] "Real-time" refers to the processing and reflection of information occurring almost instantaneously or in a very short time.

[0485] "This invention" refers to a system designed to enable users to select and experience styles that are optimized for them.

[0486] A description of embodiments for carrying out the present invention will be provided.

[0487] This system is designed to help users easily find styles that suit them. The device incorporates a high-resolution camera and display, and when a user stands in front of the device, it automatically captures the user's face and identifies their facial features. This utilizes advanced facial recognition algorithms to analyze the contours, eyes, nose, and mouth position and shape in real time.

[0488] This analysis data is sent to a server, which uses a generative AI model to generate multiple styles based on the data. The server then compares these styles with a large-scale data storage system and recommends the style best suited to the user. This recommendation is personalized based on the user's individual characteristics.

[0489] The device displays recommended styles on its screen, and the user can try out the styles using touch controls. The device simulates the selected style on the user's face in real time. This process is made possible by the device's GPU technology, allowing the user to view the styles from various angles while moving.

[0490] Furthermore, the server retrieves product information related to the selected style from a data storage device and displays it as a list on the terminal. The user can then purchase related products based on this information. The purchase process is completed online through the terminal, payment information is securely transmitted to the server, and the purchase history is recorded in the database.

[0491] For example, if user A wants to try a new hairstyle using this system, the hairstyle style will be simulated on their face in real time, and if they like it, the corresponding care products will be immediately displayed and available for purchase.

[0492] An example of a prompt message would be: "Use facial recognition technology to suggest makeup and hairstyle styles that suit me. Please simulate them virtually in real time and also display a list of related products."

[0493] In this way, users can easily and efficiently try out styles that suit them, and purchase related products seamlessly.

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] The device automatically activates its high-resolution camera when a user stands in front of it. The camera captures the user's face and acquires image data. Based on this input data, the device uses a facial recognition algorithm to identify the user's facial features, determining the contours and the position and shape of the eyes, nose, and mouth. As a result, analyzed feature data is output.

[0497] Step 2:

[0498] The server receives facial feature data transmitted from the device and inputs this data into a generating AI model. The server uses the data to generate the optimal style for the user and outputs several possible styles. The server then creates a unique style and prepares to make optimal suggestions.

[0499] Step 3:

[0500] The server compares the generated styles with style data stored in the data storage device. Through this process, the server identifies the style best suited to the user and outputs prioritized style information.

[0501] Step 4:

[0502] Upon receiving style information from the server, the terminal displays these styles on its screen. Based on this display, the user selects the style they wish to try via the touchscreen. This selection process then passes the selected style data to the next stage.

[0503] Step 5:

[0504] The device overlays the selected style onto the user's face in real time, displaying it for the user to see. The GPU is used to perform style simulations, dynamically adjusting the style to different angles and lighting conditions. The output is a real-time visual simulation.

[0505] Step 6:

[0506] Based on the user's interests, the server retrieves relevant product information from the data storage device and sends it to the terminal. The output provides a list of products related to the desired style, allowing the user to make a purchase selection.

[0507] Step 7:

[0508] If a user wishes to purchase a selected product, the terminal initiates the ordering process. Payment information is sent to the server via secure communication, which receives this information and records it in the purchase history database. As a result, the online purchase is completed, and the purchase history is updated.

[0509] (Application Example 1)

[0510] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0511] Traditional beauty experiences require users to spend a lot of time and effort trying out new looks, and the process of actually purchasing the products associated with the chosen look is often cumbersome. Therefore, there is a need for an efficient system that allows users to easily try out new looks and purchase related products on the spot.

[0512] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0513] In this invention, the server includes means for recognizing the user's facial features and analyzing their appearance based on those features; means for reflecting the analyzed appearance on the user's video in real time; and means for displaying product information related to the appearance selected by the user and facilitating the purchase process. This makes it possible for users to interactively try out new appearances in physical stores and efficiently purchase related products on the spot.

[0514] A "user" is an individual who operates the system and tries out different appearances to find what suits them best.

[0515] "Facial features" refers to the physical characteristics of a user's face, such as its shape, contours, eyes, nose, and mouth.

[0516] "Appearance" refers to styles such as makeup and hairstyles that users can try out.

[0517] An "information collection" is a database used to provide users with recommended appearance options.

[0518] "Product information" refers to information about products and services related to the appearance selected by the user.

[0519] The "purchase process" refers to the series of steps involved in a user ordering a selected product online and completing the payment.

[0520] "Physical stores" refer to shopping facilities and beauty salons that exist in a physical location.

[0521] An "interactive experience" is a process in which users can interact with the system in both directions, try out the appearance in real time, and make a purchase.

[0522] The system for implementing this invention mainly consists of interactions between a server, a terminal, and a user.

[0523] The server analyzes the user's facial features using advanced facial recognition algorithms. Specifically, it processes the user's facial image captured by a high-resolution camera, identifying the contours and positions of the eyes, nose, and mouth. Based on this data, the server works in conjunction with an information database to generate a selection of options to suggest the most suitable appearance for the user.

[0524] The terminal incorporates a mirror-type display and a camera. The terminal overlays suggested appearances received from the server onto the user's video in real time, displaying them interactively. Through the display, users can try out different styles and instantly view product information related to their selected appearance.

[0525] When a user selects a specific appearance, the system displays relevant product information on the terminal, allowing them to proceed with the purchase immediately. Once the user completes the payment, the information is securely transmitted to the server, and the purchase history is recorded.

[0526] As a concrete example, imagine a customer experiencing this interactive system for the first time in a department store's cosmetics counter. The customer stands in front of a smart mirror, tries out several makeup styles, and by selecting one, can order related cosmetics on the spot. This process takes only a few minutes from start to finish, providing the customer with an efficient and intuitive shopping experience.

[0527] Furthermore, the generative AI model associated with this system can generate new styles using prompt statements like the following.

[0528] "Design an interactive system that uses facial recognition technology to suggest recommended makeup and hairstyles based on the user's facial features in real time, and allows users to purchase related products."

[0529] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0530] Step 1:

[0531] The server receives a user's facial image sent from the terminal. Based on this input image, it uses a facial recognition algorithm to analyze facial features such as the eyes, nose, and mouth, and outputs their coordinate data. This facial data is basic information necessary for subsequent style recommendation.

[0532] Step 2:

[0533] The server uses facial feature data to match it with a database and creates a list of appearances suitable for the user. The input consists of facial feature data and user preference data, which are then used to output candidate appearance styles, which are sent to the terminal.

[0534] Step 3:

[0535] The terminal displays style candidates received from the server, overlaid on the user's video in real time. The input consists of style data and the user's video, and multiple styles are applied to the displayed video, allowing the user to visually try them out.

[0536] Step 4:

[0537] Users select their preferred appearance style through the device's interface. This selection information is sent to a server, where product information related to that appearance is collected. The selected appearance serves as an indicator in the purchase process.

[0538] Step 5:

[0539] The server sends product information related to the selected appearance to the terminal, which then displays this information to the user. At this stage, the product price, detailed information, and purchase availability are provided as output.

[0540] Step 6:

[0541] The user selects the product they wish to purchase and completes the purchase process via their device. Based on the user's selection, the purchase information is securely transmitted to the server, and the purchase history is updated. This is the final transaction processing step.

[0542] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0543] This invention not only provides a style that suits the user, but also includes a system configuration that incorporates user emotion recognition. This system operates by combining a terminal including a camera, a server that manages style and emotion information, and an emotion engine that appropriately determines reactions.

[0544] emotion recognition

[0545] When the device scans the user's face, it analyzes not only facial feature data but also the user's micro-expressions and skin condition using an emotion analysis algorithm. The emotion engine responsible for this process identifies various emotional states, such as joy, surprise, and anxiety, in real time based on the characteristics of the user's facial expressions. This data is sent to a server and used in the style recommendation process.

[0546] Recommended Style

[0547] The server considers the user's facial features and emotional data to select the most suitable makeup and hairstyle from the database. For example, if the user has a tense expression, it can recommend styles and makeup that are expected to have a relaxing effect. This makes the user experience more emotionally responsive and provides a more personalized service.

[0548] Virtual simulation and emotional feedback

[0549] The device virtually displays the selected style over the user's face. At this point, the emotion engine operates again to evaluate the user's reaction. The evaluation results are used to re-select styles and recommend products. For example, styles that the user expressed pleasure with will be given priority in subsequent recommendations.

[0550] Product information provision and purchase support

[0551] Product information related to the style selected by the user is displayed, including recommendation reasons that match their emotional state. Users can then proceed with the online purchase process based on the product information. The server stores purchase history data and uses it to improve recommendation accuracy in the future.

[0552] As a concrete example, when user B changes to an interesting hairstyle while using the system, the terminal instantly reflects the change, and the emotion engine analyzes user B's micro-expressions. If user B shows a happy reaction at this time, products related to that hairstyle are recommended in more detail. This allows user B to confidently select and purchase styles and products that fit their emotions.

[0553] Thus, the present invention provides a concrete example of a system that provides users with more satisfying personalized services by combining the analysis of emotional information.

[0554] The following describes the processing flow.

[0555] Step 1:

[0556] The device activates its built-in camera and captures the user's face in real time. It analyzes not only facial features but also micro-expressions to recognize emotions.

[0557] Step 2:

[0558] The device sends extracted facial feature data and emotion data to the server. The server receives this data, activates the emotion engine, and identifies the user's current emotional state.

[0559] Step 3:

[0560] The server accesses the database and recommends the style that best matches the user's facial features and emotional state. The recommended style takes the user's emotional state into consideration, and the one that best suits a specific emotion is selected.

[0561] Step 4:

[0562] The server sends back the selected style information to the terminal. The terminal receives this information and virtually applies the style to the user's face in real time, visualizing it.

[0563] Step 5:

[0564] The emotion engine is restarted on the device to analyze how the user is reacting to the virtual simulation. Emotional feedback is sent to the server and used to improve the recommended style.

[0565] Step 6:

[0566] The server selects the most suitable product information based on the user's emotional feedback and purchase history, and sends it to the device. The product information includes the reasons for the recommendation and is presented visually to the user.

[0567] Step 7:

[0568] The user selects their preferred product and completes the purchase process via their device. The server securely processes the purchase information, stores it in a database as a history, and uses it to improve recommendations for future purchases.

[0569] (Example 2)

[0570] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] Traditional style recommendation systems performed style analysis based on the user's facial features, but they could not take into account the user's emotional state, making it difficult to provide the optimal style for each individual user. Furthermore, they could not fully utilize the user's selection history or reactions, resulting in limited information for providing a personalized experience.

[0572] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0573] In this invention, the server includes means for analyzing the style based on the user's facial features and emotional state, means for acquiring emotional feedback and reflecting it in the user's image in real time, and means for utilizing emotional information and linking with a database. This makes it possible to take the user's emotional state into consideration and provide a personalized style that is optimal for each individual user.

[0574] "Facial features" refer to the specific patterns and shapes that make up a user's face, and are fundamental data used in selecting a style.

[0575] "Emotional state" refers to the psychological state obtained from the user's facial expressions and behavior, and includes specific emotional responses such as joy, surprise, and anxiety.

[0576] "Style" refers to the aesthetic or functional appearance recommended to the user, and includes choices of appearance such as hairstyle and makeup.

[0577] "Analysis" is the process of collecting data from the user's facial features and emotional state, and using that data to identify an optimized style.

[0578] A "database" is an information processing system that systematically stores information necessary for style recommendations and allows for quick access when needed.

[0579] "Emotional feedback" refers to information collected through an emotional engine, which reflects the user's reaction to recommended styles and is used to improve style selection in the future.

[0580] "Real-time reflection" refers to the process of instantly overlaying the analyzed style onto the user's video, allowing for immediate confirmation.

[0581] This invention is a system that provides users with emotion-based style suggestions. This system primarily consists of a terminal, a server, and various analysis engines.

[0582] terminal

[0583] The device is a piece of equipment that includes a camera and a display. The device's camera captures the user's face and acquires image data. This image data is sent in real time to the emotion engine, which analyzes the user's micro-expressions and skin condition. Image processing technologies such as the OpenCV library are used for the analysis.

[0584] server

[0585] The server receives emotion data and facial feature data sent from the terminal and selects the optimal style based on this data. The database stores past user history and various style information, and the server refers to this database to recommend styles while utilizing emotion information. Specifically, databases such as MongoDB and MySQL can be used.

[0586] Emotion engine and virtual simulation

[0587] The emotion engine analyzes the user's emotional state in real time and plays a role in obtaining emotional feedback. This feedback information is reflected in future style recommendations. The selected style is virtually overlaid on the user's face using AR technology on the device's display. Available platforms include Unity and Vuforia.

[0588] Concrete examples of user experience

[0589] For example, consider a scenario where a user wants to try a new makeup style and uses the system. The device immediately captures the user's face with its camera and analyzes it using an emotion engine. If the user is relaxed, the server selects a suitable makeup style from its database and displays it virtually on the device. All of these actions occur in real time.

[0590] An example of a prompt message would be, "Recommend the optimal style in real time based on the user's facial features and emotion data." This allows the user experience to be highly personalized to the individual's emotions.

[0591] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0592] Step 1:

[0593] The device captures the user's face with its camera. The user's face image is acquired as input and sent to the emotion engine. The image data is processed based on detected facial features to generate micro-expression data. This process utilizes image recognition technology and tools such as the OpenCV library.

[0594] Step 2:

[0595] The device uses an emotion engine to analyze the user's emotions from acquired image data. It receives facial feature data as input and generates emotion data as output through analysis. This analysis process uses algorithms that analyze microexpressions and skin condition in detail, allowing the user's emotional state (e.g., joy, surprise) to be identified in real time.

[0596] Step 3:

[0597] The device sends the analyzed emotion data to the server. Here, the emotion data is received from the emotion engine and sent to the server for comparison with the database. To ensure security, the data is encrypted before transmission. The input is emotion data, and the output is a database lookup request on the server side.

[0598] Step 4:

[0599] The server selects the optimal style from a database based on the user's emotional data and facial features. It takes emotional data and facial feature data as input and generates recommended styles as output. In this step, it refers to similar user patterns and past style selection history, and uses a database management system (e.g., MongoDB) to select the optimal makeup and hairstyle.

[0600] Step 5:

[0601] The server sends the selected style information to the terminal. Here, the recommended style is retrieved from the server and sent to the user's display device in real time. The recommended style data is received from the server as input, and style information is provided to the terminal as output. Through this process, the user can see a virtual simulation of the recommended style.

[0602] Step 6:

[0603] The device overlays the selected style onto the user's face and obtains emotional feedback again. It receives recommended style data as input and virtually overlays it onto the user's face. The output generates a virtual simulation display and emotional feedback data. The emotion engine re-evaluates the feedback, and the feedback is used for future recommendations.

[0604] Step 7:

[0605] The device provides the user with the final style and related product information, and supports their purchase. It receives style feedback as input and presents relevant product information. It generates purchase instructions as output. In this step, the user can confidently select and purchase products while understanding the reasons for the recommendations.

[0606] (Application Example 2)

[0607] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0608] Traditional style recommendation systems determine styles based solely on the user's facial features, which limits their ability to provide personalized recommendations that take into account the user's emotional state. Furthermore, there is a lack of style evaluation and product recommendations that utilize real-time emotional responses, highlighting the need for improved user experience.

[0609] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0610] In this invention, the server includes means for recognizing the user's facial features and emotional state, and analyzing a style based on those features and state; means for reflecting the analyzed style on the user's image in real time and evaluating the emotional response; means for linking with a personal information database for recommending a style suitable for the user based on the evaluation results; and means for displaying product information related to the style selected by the user, presenting recommendation reasons based on emotions, and facilitating the ordering process. This makes it possible to provide highly accurate style recommendations to users that take emotions into consideration, and a highly satisfying purchasing experience.

[0611] "Facial features" refer to the unique shape and arrangement information present in a user's face, and are data used for style analysis.

[0612] "Emotional state" refers to information that indicates the inner feelings and psychological state obtained from the user's facial expressions and actions.

[0613] "Style" refers to external elements such as hairstyle, makeup, and clothing applied to a user, and is selected based on individual preferences and feelings.

[0614] "Real-time" means that information processing and the reflection of results are performed instantly without delay.

[0615] "Evaluation results" are conclusions regarding the appropriateness and suitability of a particular style or product, derived from user response data analyzed by the emotion engine.

[0616] A "personal information database" is a source of information that stores data on users' facial features, emotional history, and style preferences, and is used for style recommendations.

[0617] "Product information" refers to detailed information about the product name, features, price, and purchase process related to the style the user has shown interest in.

[0618] The "order process" refers to the series of steps required when a user purchases a selected product, including confirmation, payment, and shipping.

[0619] This invention is a system that uses a smart mirror installed in stores or homes to recommend styles to users. The server captures the user's facial features and emotional state through a built-in camera and analyzes their style based on this data. The analysis uses a facial recognition library (such as OpenCV) and an emotion analysis engine. The emotion engine evaluates the user's micro-expressions in real time, and the server stores the evaluation results. As a result, the most suitable fashion items and related products are selected from the database and displayed on the screen as a virtual try-on experience. The display follows the user's movements and smoothly provides try-on results and product information. The user can check product information related to the styles they are interested in and, if necessary, proceed with the ordering process on the smart mirror.

[0620] As a concrete example, when a user tries on their favorite style in front of a smart mirror, the system analyzes the user's micro-expressions to detect feelings of happiness and curiosity. As a result, it recommends casual and relaxing clothing to the user, creating a more satisfying shopping experience. An example of a prompt to the generative AI model is, "Design an algorithm for a virtual try-on app that analyzes the user's micro-expressions and suggests casual clothing if they appear happy."

[0621] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0622] Step 1:

[0623] The device uses its built-in camera to capture the user's facial features and emotional state. The input is image data of the user from the front, captured by the camera, and the output is facial feature data and micro-expression data. This data is analyzed using a facial recognition library (such as OpenCV).

[0624] Step 2:

[0625] The server receives facial feature data and microexpression data sent from the terminal. The input is data from the terminal, and the output is the emotion analysis result. The server uses an emotion analysis engine to identify emotions such as joy and surprise from the user's microexpressions.

[0626] Step 3:

[0627] The server initiates style recommendations based on the sentiment analysis results. The input is the sentiment analysis data, and the output is style information suitable for the user. The server interacts with a person information database to search for styles that match the analysis results.

[0628] Step 4:

[0629] The terminal receives style information sent from the server and provides the user with a virtual try-on experience on the display. The input is style information, and the output is a virtual try-on image displayed on the screen. The display follows the user's movements in real time to support the try-on experience.

[0630] Step 5:

[0631] The user reviews the virtual try-on experience and expresses emotional responses to styles that interest them. The input is the user's micro-facial expressions, and the output is the emotional analysis data. The server performs the analysis again via the emotion engine to evaluate the user's satisfaction level.

[0632] Step 6:

[0633] The server provides users with relevant product information based on the evaluation results. The input is sentiment evaluation data, and the output is product information and reasons for recommendation. Based on this, the user proceeds with the order process and completes the purchase on their terminal.

[0634] Step 7:

[0635] User purchase history and sentiment data are stored on the server to be used to improve future style recommendations. The input is purchase and sentiment history data, and the output is an updated database.

[0636] This entire process allows us to optimize the user experience and provide personalized styles and products.

[0637] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0638] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0639] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0640] [Fourth Embodiment]

[0641] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0642] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0643] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0644] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0645] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0646] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0647] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0648] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0649] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0650] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0651] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0652] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0653] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0654] This invention presents an embodiment of a system that allows users to easily find makeup and hairstyles that suit them. This system uses facial recognition technology to scan the user's facial features and optimizes the style based on that data. The specific implementation of each element is described below.

[0655] User facial recognition

[0656] The device has a built-in high-resolution camera that automatically activates when the user stands in front of the smart mirror. The device uses an advanced facial recognition algorithm to capture the user's facial features in real time and analyze the contours, eyes, nose, and mouth position and shape. This data is sent to a server and used for further analysis.

[0657] Style recommendations and displays

[0658] The server uses the received facial feature data to match it against a large style database. This allows it to suggest multiple optimal makeup and hairstyle styles for the user. The suggested styles are displayed on the device's screen, and the user can select and try different styles using touch controls.

[0659] Virtual Simulation

[0660] The device overlays the selected style onto the user's face in real time, visualizing virtual makeup and hairstyles. This simulation follows the user's movements, allowing for viewing from different angles. The system utilizes GPU-driven image processing to achieve this.

[0661] Product recommendation and purchase processing

[0662] Products matching the user's selected style are fed back to the server and displayed as a list on the device. If the user wishes to purchase a product, they can proceed with the online order process through the device, and payment information is sent to the server via a secure communication method. The server records this information and stores the purchase history in a database.

[0663] As a concrete example, consider a scenario where User A stands in front of a smart mirror and tries to change their hairstyle. User A's face is recognized, and several hairstyle styles are recommended from the server. User A selects their favorite hairstyle, and that hairstyle is virtually simulated. During this time, User A can check whether it suits their face from various angles, and if they like it, they can purchase related products on the spot.

[0664] Through this process, users can easily try out new styles and purchase related products.

[0665] The following describes the processing flow.

[0666] Step 1:

[0667] The device activates its camera and captures a video of the user's face. When the user stands in front of the smart mirror, it automatically recognizes the face area and extracts feature data.

[0668] Step 2:

[0669] The server receives the user's facial feature data sent from the terminal. Based on this, it compares it with data in a pre-built style database and selects a style that suits the user.

[0670] Step 3:

[0671] The server generates a list of selected makeup and hairstyle styles and sends this information to the terminal. The terminal displays these recommended styles to the user and prompts them to make a selection.

[0672] Step 4:

[0673] The user selects a style they are interested in, and the device overlays the selected style onto the user's face in real time. This allows the user to see how the style will look on their own face.

[0674] Step 5:

[0675] The server analyzes the user's selections and creates an optimized list of product recommendations. This product information is sent to the terminal and presented to the user.

[0676] Step 6:

[0677] The user selects a product they like and expresses their intention to purchase it via their device. The device then uses secure communication with the server to process the order and verify payment information.

[0678] Step 7:

[0679] The server processes purchase information and stores it in the database as purchase history. This allows for more personalized suggestions to the user the next time they use the service.

[0680] (Example 1)

[0681] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0682] There is a problem in that it is difficult for individual users to easily and effectively find the style that is best suited to them. Furthermore, there is a lack of a process for users to easily purchase products related to their chosen style, as they cannot see how the style will actually look in real time.

[0683] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0684] In this invention, the server includes means for identifying the features of the user's face and analyzing expressions based on those features, means for reflecting the analyzed expressions on the user's video in real time, and means for coordinating with a data storage device for recommending expressions suitable for the user. This enables the user to quickly find the style that is best suited to them, confirm that style in real time, and smoothly purchase related products.

[0685] A "user" refers to an individual who uses this system to select and experiment with styles.

[0686] "Facial features" refer to data about the position and shape of an individual's eyes, nose, mouth, and facial contours.

[0687] "Expression" refers to styles and designs that change a user's appearance, such as makeup and hairstyles.

[0688] "Identification" refers to the process of extracting and recognizing specific features from images or data using a particular algorithm.

[0689] "Analysis" is the process of thoroughly examining collected data, extracting information, and understanding it.

[0690] A "generative AI model" refers to an algorithm or program that uses AI technology to automatically create new designs and styles.

[0691] A "data storage device" refers to a system for systematically storing and managing multiple styles and product information.

[0692] "Real-time" refers to the processing and reflection of information occurring almost instantaneously or in a very short time.

[0693] "This invention" refers to a system designed to enable users to select and experience styles that are optimized for them.

[0694] A description of embodiments for carrying out the present invention will be provided.

[0695] This system is designed to help users easily find styles that suit them. The device incorporates a high-resolution camera and display, and when a user stands in front of the device, it automatically captures the user's face and identifies their facial features. This utilizes advanced facial recognition algorithms to analyze the contours, eyes, nose, and mouth position and shape in real time.

[0696] This analysis data is sent to a server, which uses a generative AI model to generate multiple styles based on the data. The server then compares these styles with a large-scale data storage system and recommends the style best suited to the user. This recommendation is personalized based on the user's individual characteristics.

[0697] The device displays recommended styles on its screen, and the user can try out the styles using touch controls. The device simulates the selected style on the user's face in real time. This process is made possible by the device's GPU technology, allowing the user to view the styles from various angles while moving.

[0698] Furthermore, the server retrieves product information related to the selected style from a data storage device and displays it as a list on the terminal. The user can then purchase related products based on this information. The purchase process is completed online through the terminal, payment information is securely transmitted to the server, and the purchase history is recorded in the database.

[0699] For example, if user A wants to try a new hairstyle using this system, the hairstyle style will be simulated on their face in real time, and if they like it, the corresponding care products will be immediately displayed and available for purchase.

[0700] An example of a prompt message would be: "Use facial recognition technology to suggest makeup and hairstyle styles that suit me. Please simulate them virtually in real time and also display a list of related products."

[0701] In this way, users can easily and efficiently try out styles that suit them, and purchase related products seamlessly.

[0702] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0703] Step 1:

[0704] The device automatically activates its high-resolution camera when a user stands in front of it. The camera captures the user's face and acquires image data. Based on this input data, the device uses a facial recognition algorithm to identify the user's facial features, determining the contours and the position and shape of the eyes, nose, and mouth. As a result, analyzed feature data is output.

[0705] Step 2:

[0706] The server receives facial feature data transmitted from the device and inputs this data into a generating AI model. The server uses the data to generate the optimal style for the user and outputs several possible styles. The server then creates a unique style and prepares to make optimal suggestions.

[0707] Step 3:

[0708] The server compares the generated styles with style data stored in the data storage device. Through this process, the server identifies the style best suited to the user and outputs prioritized style information.

[0709] Step 4:

[0710] Upon receiving style information from the server, the terminal displays these styles on its screen. Based on this display, the user selects the style they wish to try via the touchscreen. This selection process then passes the selected style data to the next stage.

[0711] Step 5:

[0712] The device overlays the selected style onto the user's face in real time, displaying it for the user to see. The GPU is used to perform style simulations, dynamically adjusting the style to different angles and lighting conditions. The output is a real-time visual simulation.

[0713] Step 6:

[0714] Based on the user's interests, the server retrieves relevant product information from the data storage device and sends it to the terminal. The output provides a list of products related to the desired style, allowing the user to make a purchase selection.

[0715] Step 7:

[0716] If a user wishes to purchase a selected product, the terminal initiates the ordering process. Payment information is sent to the server via secure communication, which receives this information and records it in the purchase history database. As a result, the online purchase is completed, and the purchase history is updated.

[0717] (Application Example 1)

[0718] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0719] Traditional beauty experiences require users to spend a lot of time and effort trying out new looks, and the process of actually purchasing the products associated with the chosen look is often cumbersome. Therefore, there is a need for an efficient system that allows users to easily try out new looks and purchase related products on the spot.

[0720] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0721] In this invention, the server includes means for recognizing the user's facial features and analyzing their appearance based on those features; means for reflecting the analyzed appearance on the user's video in real time; and means for displaying product information related to the appearance selected by the user and facilitating the purchase process. This makes it possible for users to interactively try out new appearances in physical stores and efficiently purchase related products on the spot.

[0722] A "user" is an individual who operates the system and tries out different appearances to find what suits them best.

[0723] "Facial features" refers to the physical characteristics of a user's face, such as its shape, contours, eyes, nose, and mouth.

[0724] "Appearance" refers to styles such as makeup and hairstyles that users can try out.

[0725] An "information collection" is a database used to provide users with recommended appearance options.

[0726] "Product information" refers to information about products and services related to the appearance selected by the user.

[0727] The "purchase process" refers to the series of steps involved in a user ordering a selected product online and completing the payment.

[0728] "Physical stores" refer to shopping facilities and beauty salons that exist in a physical location.

[0729] An "interactive experience" is a process in which users can interact with the system in both directions, try out the appearance in real time, and make a purchase.

[0730] The system for implementing this invention mainly consists of interactions between a server, a terminal, and a user.

[0731] The server analyzes the user's facial features using advanced facial recognition algorithms. Specifically, it processes the user's facial image captured by a high-resolution camera, identifying the contours and positions of the eyes, nose, and mouth. Based on this data, the server works in conjunction with an information database to generate a selection of options to suggest the most suitable appearance for the user.

[0732] The terminal incorporates a mirror-type display and a camera. The terminal overlays suggested appearances received from the server onto the user's video in real time, displaying them interactively. Through the display, users can try out different styles and instantly view product information related to their selected appearance.

[0733] When a user selects a specific appearance, the system displays relevant product information on the terminal, allowing them to proceed with the purchase immediately. Once the user completes the payment, the information is securely transmitted to the server, and the purchase history is recorded.

[0734] As a concrete example, imagine a customer experiencing this interactive system for the first time in a department store's cosmetics counter. The customer stands in front of a smart mirror, tries out several makeup styles, and by selecting one, can order related cosmetics on the spot. This process takes only a few minutes from start to finish, providing the customer with an efficient and intuitive shopping experience.

[0735] Furthermore, the generative AI model associated with this system can generate new styles using prompt statements like the following.

[0736] "Design an interactive system that uses facial recognition technology to suggest recommended makeup and hairstyles based on the user's facial features in real time, and allows users to purchase related products."

[0737] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0738] Step 1:

[0739] The server receives a user's facial image sent from the terminal. Based on this input image, it uses a facial recognition algorithm to analyze facial features such as the eyes, nose, and mouth, and outputs their coordinate data. This facial data is basic information necessary for subsequent style recommendation.

[0740] Step 2:

[0741] The server uses facial feature data to match it with a database and creates a list of appearances suitable for the user. The input consists of facial feature data and user preference data, which are then used to output candidate appearance styles, which are sent to the terminal.

[0742] Step 3:

[0743] The terminal displays style candidates received from the server, overlaid on the user's video in real time. The input consists of style data and the user's video, and multiple styles are applied to the displayed video, allowing the user to visually try them out.

[0744] Step 4:

[0745] Users select their preferred appearance style through the device's interface. This selection information is sent to a server, where product information related to that appearance is collected. The selected appearance serves as an indicator in the purchase process.

[0746] Step 5:

[0747] The server sends product information related to the selected appearance to the terminal, which then displays this information to the user. At this stage, the product price, detailed information, and purchase availability are provided as output.

[0748] Step 6:

[0749] The user selects the product they wish to purchase and completes the purchase process via their device. Based on the user's selection, the purchase information is securely transmitted to the server, and the purchase history is updated. This is the final transaction processing step.

[0750] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0751] This invention not only provides a style that suits the user, but also includes a system configuration that incorporates user emotion recognition. This system operates by combining a terminal including a camera, a server that manages style and emotion information, and an emotion engine that appropriately determines reactions.

[0752] emotion recognition

[0753] When the device scans the user's face, it analyzes not only facial feature data but also the user's micro-expressions and skin condition using an emotion analysis algorithm. The emotion engine responsible for this process identifies various emotional states, such as joy, surprise, and anxiety, in real time based on the characteristics of the user's facial expressions. This data is sent to a server and used in the style recommendation process.

[0754] Recommended Style

[0755] The server considers the user's facial features and emotional data to select the most suitable makeup and hairstyle from the database. For example, if the user has a tense expression, it can recommend styles and makeup that are expected to have a relaxing effect. This makes the user experience more emotionally responsive and provides a more personalized service.

[0756] Virtual simulation and emotional feedback

[0757] The device virtually displays the selected style over the user's face. At this point, the emotion engine operates again to evaluate the user's reaction. The evaluation results are used to re-select styles and recommend products. For example, styles that the user expressed pleasure with will be given priority in subsequent recommendations.

[0758] Product information provision and purchase support

[0759] Product information related to the style selected by the user is displayed, including recommendation reasons that match their emotional state. Users can then proceed with the online purchase process based on the product information. The server stores purchase history data and uses it to improve recommendation accuracy in the future.

[0760] As a concrete example, when user B changes to an interesting hairstyle while using the system, the terminal instantly reflects the change, and the emotion engine analyzes user B's micro-expressions. If user B shows a happy reaction at this time, products related to that hairstyle are recommended in more detail. This allows user B to confidently select and purchase styles and products that fit their emotions.

[0761] Thus, the present invention provides a concrete example of a system that provides users with more satisfying personalized services by combining the analysis of emotional information.

[0762] The following describes the processing flow.

[0763] Step 1:

[0764] The device activates its built-in camera and captures the user's face in real time. It analyzes not only facial features but also micro-expressions to recognize emotions.

[0765] Step 2:

[0766] The device sends extracted facial feature data and emotion data to the server. The server receives this data, activates the emotion engine, and identifies the user's current emotional state.

[0767] Step 3:

[0768] The server accesses the database and recommends the style that best matches the user's facial features and emotional state. The recommended style takes the user's emotional state into consideration, and the one that best suits a specific emotion is selected.

[0769] Step 4:

[0770] The server sends back the selected style information to the terminal. The terminal receives this information and virtually applies the style to the user's face in real time, visualizing it.

[0771] Step 5:

[0772] The emotion engine is restarted on the device to analyze how the user is reacting to the virtual simulation. Emotional feedback is sent to the server and used to improve the recommended style.

[0773] Step 6:

[0774] The server selects the most suitable product information based on the user's emotional feedback and purchase history, and sends it to the device. The product information includes the reasons for the recommendation and is presented visually to the user.

[0775] Step 7:

[0776] The user selects their preferred product and completes the purchase process via their device. The server securely processes the purchase information, stores it in a database as a history, and uses it to improve recommendations for future purchases.

[0777] (Example 2)

[0778] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] Traditional style recommendation systems performed style analysis based on the user's facial features, but they could not take into account the user's emotional state, making it difficult to provide the optimal style for each individual user. Furthermore, they could not fully utilize the user's selection history or reactions, resulting in limited information for providing a personalized experience.

[0780] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0781] In this invention, the server includes means for analyzing the style based on the user's facial features and emotional state, means for acquiring emotional feedback and reflecting it in the user's image in real time, and means for utilizing emotional information and linking with a database. This makes it possible to take the user's emotional state into consideration and provide a personalized style that is optimal for each individual user.

[0782] "Facial features" refer to the specific patterns and shapes that make up a user's face, and are fundamental data used in selecting a style.

[0783] "Emotional state" refers to the psychological state obtained from the user's facial expressions and behavior, and includes specific emotional responses such as joy, surprise, and anxiety.

[0784] "Style" refers to the aesthetic or functional appearance recommended to the user, and includes choices of appearance such as hairstyle and makeup.

[0785] "Analysis" is the process of collecting data from the user's facial features and emotional state, and using that data to identify an optimized style.

[0786] A "database" is an information processing system that systematically stores information necessary for style recommendations and allows for quick access when needed.

[0787] "Emotional feedback" refers to information collected through an emotional engine, which reflects the user's reaction to recommended styles and is used to improve style selection in the future.

[0788] "Real-time reflection" refers to the process of instantly overlaying the analyzed style onto the user's video, allowing for immediate confirmation.

[0789] This invention is a system that provides users with emotion-based style suggestions. This system primarily consists of a terminal, a server, and various analysis engines.

[0790] terminal

[0791] The device is a piece of equipment that includes a camera and a display. The device's camera captures the user's face and acquires image data. This image data is sent in real time to the emotion engine, which analyzes the user's micro-expressions and skin condition. Image processing technologies such as the OpenCV library are used for the analysis.

[0792] server

[0793] The server receives emotion data and facial feature data sent from the terminal and selects the optimal style based on this data. The database stores past user history and various style information, and the server refers to this database to recommend styles while utilizing emotion information. Specifically, databases such as MongoDB and MySQL can be used.

[0794] Emotion engine and virtual simulation

[0795] The emotion engine analyzes the user's emotional state in real time and plays a role in obtaining emotional feedback. This feedback information is reflected in future style recommendations. The selected style is virtually overlaid on the user's face using AR technology on the device's display. Available platforms include Unity and Vuforia.

[0796] Concrete examples of user experience

[0797] For example, consider a scenario where a user wants to try a new makeup style and uses the system. The device immediately captures the user's face with its camera and analyzes it using an emotion engine. If the user is relaxed, the server selects a suitable makeup style from its database and displays it virtually on the device. All of these actions occur in real time.

[0798] An example of a prompt message would be, "Recommend the optimal style in real time based on the user's facial features and emotion data." This allows the user experience to be highly personalized to the individual's emotions.

[0799] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0800] Step 1:

[0801] The device captures the user's face with its camera. The user's face image is acquired as input and sent to the emotion engine. The image data is processed based on detected facial features to generate micro-expression data. This process utilizes image recognition technology and tools such as the OpenCV library.

[0802] Step 2:

[0803] The device uses an emotion engine to analyze the user's emotions from acquired image data. It receives facial feature data as input and generates emotion data as output through analysis. This analysis process uses algorithms that analyze microexpressions and skin condition in detail, allowing the user's emotional state (e.g., joy, surprise) to be identified in real time.

[0804] Step 3:

[0805] The device sends the analyzed emotion data to the server. Here, the emotion data is received from the emotion engine and sent to the server for comparison with the database. To ensure security, the data is encrypted before transmission. The input is emotion data, and the output is a database lookup request on the server side.

[0806] Step 4:

[0807] The server selects the optimal style from a database based on the user's emotional data and facial features. It takes emotional data and facial feature data as input and generates recommended styles as output. In this step, it refers to similar user patterns and past style selection history, and uses a database management system (e.g., MongoDB) to select the optimal makeup and hairstyle.

[0808] Step 5:

[0809] The server sends the selected style information to the terminal. Here, the recommended style is retrieved from the server and sent to the user's display device in real time. The recommended style data is received from the server as input, and style information is provided to the terminal as output. Through this process, the user can see a virtual simulation of the recommended style.

[0810] Step 6:

[0811] The device overlays the selected style onto the user's face and obtains emotional feedback again. It receives recommended style data as input and virtually overlays it onto the user's face. The output generates a virtual simulation display and emotional feedback data. The emotion engine re-evaluates the feedback, and the feedback is used for future recommendations.

[0812] Step 7:

[0813] The device provides the user with the final style and related product information, and supports their purchase. It receives style feedback as input and presents relevant product information. It generates purchase instructions as output. In this step, the user can confidently select and purchase products while understanding the reasons for the recommendations.

[0814] (Application Example 2)

[0815] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0816] Traditional style recommendation systems determine styles based solely on the user's facial features, which limits their ability to provide personalized recommendations that take into account the user's emotional state. Furthermore, there is a lack of style evaluation and product recommendations that utilize real-time emotional responses, highlighting the need for improved user experience.

[0817] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0818] In this invention, the server includes means for recognizing the user's facial features and emotional state, and analyzing a style based on those features and state; means for reflecting the analyzed style on the user's image in real time and evaluating the emotional response; means for linking with a personal information database for recommending a style suitable for the user based on the evaluation results; and means for displaying product information related to the style selected by the user, presenting recommendation reasons based on emotions, and facilitating the ordering process. This makes it possible to provide highly accurate style recommendations to users that take emotions into consideration, and a highly satisfying purchasing experience.

[0819] "Facial features" refer to the unique shape and arrangement information present in a user's face, and are data used for style analysis.

[0820] "Emotional state" refers to information that indicates the inner feelings and psychological state obtained from the user's facial expressions and actions.

[0821] "Style" refers to external elements such as hairstyle, makeup, and clothing applied to a user, and is selected based on individual preferences and feelings.

[0822] "Real-time" means that information processing and the reflection of results are performed instantly without delay.

[0823] "Evaluation results" are conclusions regarding the appropriateness and suitability of a particular style or product, derived from user response data analyzed by the emotion engine.

[0824] A "personal information database" is a source of information that stores data on users' facial features, emotional history, and style preferences, and is used for style recommendations.

[0825] "Product information" refers to detailed information about the product name, features, price, and purchase process related to the style the user has shown interest in.

[0826] The "order process" refers to the series of steps required when a user purchases a selected product, including confirmation, payment, and shipping.

[0827] This invention is a system that uses a smart mirror installed in stores or homes to recommend styles to users. The server captures the user's facial features and emotional state through a built-in camera and analyzes their style based on this data. The analysis uses a facial recognition library (such as OpenCV) and an emotion analysis engine. The emotion engine evaluates the user's micro-expressions in real time, and the server stores the evaluation results. As a result, the most suitable fashion items and related products are selected from the database and displayed on the screen as a virtual try-on experience. The display follows the user's movements and smoothly provides try-on results and product information. The user can check product information related to the styles they are interested in and, if necessary, proceed with the ordering process on the smart mirror.

[0828] As a concrete example, when a user tries on their favorite style in front of a smart mirror, the system analyzes the user's micro-expressions to detect feelings of happiness and curiosity. As a result, it recommends casual and relaxing clothing to the user, creating a more satisfying shopping experience. An example of a prompt to the generative AI model is, "Design an algorithm for a virtual try-on app that analyzes the user's micro-expressions and suggests casual clothing if they appear happy."

[0829] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0830] Step 1:

[0831] The device uses its built-in camera to capture the user's facial features and emotional state. The input is image data of the user from the front, captured by the camera, and the output is facial feature data and micro-expression data. This data is analyzed using a facial recognition library (such as OpenCV).

[0832] Step 2:

[0833] The server receives facial feature data and microexpression data sent from the terminal. The input is data from the terminal, and the output is the emotion analysis result. The server uses an emotion analysis engine to identify emotions such as joy and surprise from the user's microexpressions.

[0834] Step 3:

[0835] The server initiates style recommendations based on the sentiment analysis results. The input is the sentiment analysis data, and the output is style information suitable for the user. The server interacts with a person information database to search for styles that match the analysis results.

[0836] Step 4:

[0837] The terminal receives style information sent from the server and provides the user with a virtual try-on experience on the display. The input is style information, and the output is a virtual try-on image displayed on the screen. The display follows the user's movements in real time to support the try-on experience.

[0838] Step 5:

[0839] The user reviews the virtual try-on experience and expresses emotional responses to styles that interest them. The input is the user's micro-facial expressions, and the output is the emotional analysis data. The server performs the analysis again via the emotion engine to evaluate the user's satisfaction level.

[0840] Step 6:

[0841] The server provides users with relevant product information based on the evaluation results. The input is sentiment evaluation data, and the output is product information and reasons for recommendation. Based on this, the user proceeds with the order process and completes the purchase on their terminal.

[0842] Step 7:

[0843] User purchase history and sentiment data are stored on the server to be used to improve future style recommendations. The input is purchase and sentiment history data, and the output is an updated database.

[0844] This entire process allows us to optimize the user experience and provide personalized styles and products.

[0845] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0846] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0847] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0848] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0849] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0850] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0851] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0852] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0853] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0854] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0855] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0856] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0857] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0858] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0859] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0860] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0861] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0862] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0863] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0864] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0865] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0866] The following is further disclosed regarding the embodiments described above.

[0867] (Claim 1)

[0868] A means for recognizing the features of a user's face and analyzing their style based on those features,

[0869] A means of reflecting the analyzed style onto the user's image in real time,

[0870] A means of linking with a database to recommend a style suitable for the user,

[0871] A means of displaying product information related to the style selected by the user and proceeding with the order,

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, comprising an algorithm for storing user history and preference information and optimizing style recommendations based on said information.

[0875] (Claim 3)

[0876] The system according to claim 1, comprising a mirror-type display device incorporating a camera and a display, and including means for tracking the user's movements in real time.

[0877] "Example 1"

[0878] (Claim 1)

[0879] A means for identifying the features of a user's face and analyzing the representation based on those features,

[0880] A means of reflecting the analyzed expression in the user's video in real time,

[0881] A means of collaborating with a data storage device to recommend expressions suitable for the user,

[0882] A means for displaying product information related to the expression selected by the user and for processing an order,

[0883] A method for generating numerous user-optimized styles using a generative AI model,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, comprising computer processing for storing user history and preference information and optimizing the recommendation of representations based on said information.

[0887] (Claim 3)

[0888] The system according to claim 1, comprising a reflective display device incorporating an imaging device and a display device, and including means for tracking the user's movements in real time.

[0889] "Application Example 1"

[0890] (Claim 1)

[0891] A means for recognizing the user's facial features and analyzing their appearance based on those features,

[0892] A means of reflecting the analyzed appearance onto the user's video in real time,

[0893] A means of linking with a collection of information to recommend an appearance suitable for the user,

[0894] A means of displaying product information related to the appearance selected by the user and proceeding with the purchase,

[0895] A means of providing an interactive experience in physical stores where users can try out new looks and purchase products on the spot,

[0896] A system that includes this.

[0897] (Claim 2)

[0898] The system according to claim 1, comprising a calculation formula for storing user history and preference information and optimizing appearance recommendations based on said information.

[0899] (Claim 3)

[0900] The system according to claim 1, comprising a mirror-type display means incorporating a shooting device and a display device, and including means for tracking the user's movements in real time.

[0901] "Example 2 of combining an emotion engine"

[0902] (Claim 1)

[0903] A means for recognizing the facial features of a user and analyzing their style based on those features and the user's emotional state,

[0904] A means of reflecting the analyzed style in the user's image in real time and obtaining emotional feedback,

[0905] It integrates with a database to recommend styles suitable for the user and utilizes emotional information.

[0906] A means of displaying product information related to the style selected by the user, and carrying out the ordering and purchase process,

[0907] A system that includes this.

[0908] (Claim 2)

[0909] The system according to claim 1, comprising an algorithm for storing past emotional responses in addition to the user's history and preference information, and for optimizing style recommendations based on said information.

[0910] (Claim 3)

[0911] The system according to claim 1, comprising a mirror-type display mechanism incorporating a camera and a display device, and including means for tracking the user and changes in their emotions in real time.

[0912] "Application example 2 when combining with an emotional engine"

[0913] (Claim 1)

[0914] A means for recognizing the user's facial features and emotional state, and analyzing the style based on those features and state,

[0915] The analyzed style is reflected in the user's profile in real time, and a means is provided to evaluate emotional responses.

[0916] A means of linking with a personal information database to recommend a style suitable for the user based on evaluation results,

[0917] A means of displaying product information related to the style selected by the user, presenting emotion-based recommendation reasons, and guiding the user through the ordering process.

[0918] A system that includes this.

[0919] (Claim 2)

[0920] The system according to claim 1, comprising a calculation means for storing user history, preference information, and emotional responses, and for optimizing style recommendations based on said information.

[0921] (Claim 3)

[0922] The system according to claim 1, comprising a reflective device incorporating a camera and a display device, and including means for tracking the user's movements and emotions in real time. [Explanation of symbols]

[0923] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for recognizing the features of a user's face and analyzing their style based on those features, A means of reflecting the analyzed style onto the user's image in real time, A means of linking with a database to recommend a style suitable for the user, A means of displaying product information related to the style selected by the user and proceeding with the order, A system that includes this.

2. The system according to claim 1, comprising an algorithm for storing user history and preference information and optimizing style recommendations based on said information.

3. The system according to claim 1, comprising a mirror-type display device incorporating a camera and a display, and including means for tracking the user's movements in real time.

Images