System

Abstract

A system is provided.SOLUTION: A system comprising: image input means; means for receiving image data acquired by the image input means; image analysis means for extracting component information of a product from the received image data; component collation means for collating the extracted component information with a database; component evaluation means for evaluating a balance and a bias of components based on the component information; substitute proposal means for searching for and proposing a substitute item based on the component balance and the bias evaluated by the component evaluation means; and information display means for displaying detailed information of the substitute item.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The modern skincare market is home to a vast number of products, each with different ingredients and effects. However, it can be extremely difficult for consumers to understand the ingredient information of all of these products and select the product that best suits their skin condition and goals. Furthermore, combining multiple skincare products can result in excess or deficiency of ingredients, leading to an imbalance in skincare. Therefore, it is desirable to provide a system that analyzes the ingredients of skincare products in users' hands, evaluates the balance of ingredients, and then suggests optimal alternative products. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system including an image input device, a device for receiving image data acquired by the image input device, an image analysis device for extracting product ingredient information from the received image data, an ingredient comparison device for comparing the extracted ingredient information with a database, an ingredient evaluation device for evaluating the balance and imbalance of ingredients based on the ingredient information, a substitute suggestion device for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation device, and an information display device for displaying detailed information about the alternative items. This allows users to photograph and analyze the ingredients of their skin care products and receive suggestions for optimal alternative items based on the ingredient imbalance and balance. As a result, it is possible to maximize the effects of skin care and maintain good skin condition.

[0006] "Image input means" refers to a device or function that allows a user to capture an image of a skin care product, and specifically refers to a smartphone camera or the like.

[0007] The "means for receiving image data" is a system or protocol for receiving image data acquired by the image input means and passing it on to the next processing step.

[0008] "Image analysis means" refers to technologies and algorithms for recognizing and extracting product ingredient information from received image data, and specifically includes optical character recognition (OCR) technology.

[0009] The "ingredient collating means" is a technology for collating ingredient information extracted by the image analyzing means with a database to identify detailed ingredient data and their effects.

[0010] The "ingredient evaluation means" is a system for evaluating the balance and bias of ingredients based on ingredient information and making a judgment that takes into consideration the user's skin condition and skin care goals.

[0011] The "substitute product suggestion means" is a function for searching the database for optimal substitute skin care items based on the results of the ingredient evaluation means and suggesting them to the user.

[0012] The "information display means" is an interface that displays detailed information such as ingredients, effects, reviews, and price ranges of alternative items to the user, making it easier for the user to select the most suitable skin care product. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[0035] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0036] Based on this detailed ingredient information, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below the standard value, it will evaluate that the user's skin is dry. Next, the server searches the database for alternative items with high moisturizing effects and suggests the most suitable item using an alternative suggestion means. Specifically, it may suggest "Brand B's highly moisturizing lotion."

[0037] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that best suits them.

[0038] This system also supports the use of multiple skincare products, and can comprehensively recommend items to supplement missing or excess ingredients while taking into account the overall balance of ingredients. This allows users to choose the skincare products that are best suited to their skin condition, helping to maintain good skin condition.

[0039] The processing flow will be explained below.

[0040] Step 1:

[0041] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[0042] Specifically, take care to photograph skin care product labels and packaging clearly.

[0043] Step 2:

[0044] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[0045] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[0046] Step 3:

[0047] The terminal transmits the image data to the server.

[0048] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[0049] Step 4:

[0050] The server receives the image data.

[0051] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[0052] Step 5:

[0053] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[0054] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[0055] Step 6:

[0056] The server uses the component matching means to match the extracted component information with an internal database.

[0057] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[0058] Step 7:

[0059] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[0060] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[0061] Step 8:

[0062] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[0063] Specifically, the database will be used to create a list of products that supplement missing ingredients and products that help to achieve overall balance.

[0064] Step 9:

[0065] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[0066] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[0067] Step 10:

[0068] The terminal displays detailed information of the substitute item received from the server to the user.

[0069] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[0070] Step 11:

[0071] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[0072] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[0073] This detailed processing step provides users with specific and intuitive assistance in choosing the best skin care products for their skin condition.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] Conventional skin care product recommendation systems have difficulty automatically selecting the optimal skin care product for a user's skin condition. Furthermore, systems that accurately grasp the product's ingredient information and then recommend optimal alternative products based on that information are insufficient. As a result, users end up purchasing products at their own discretion, which can prevent them from maximizing their effectiveness.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes an image input means for a user to take an image of a skin care product, a means for receiving image data acquired by the image input means, a means for compressing and transmitting the received image data, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, and an information display means for displaying detailed information about the alternative items. This allows users to easily select and purchase skin care products that are best suited to their skin condition.

[0079] "Image input means" refers to a function or device for taking an image of a skin care product, and includes the camera of a mobile terminal used by the user.

[0080] The "receiving means" is a function or device for taking the image data acquired by the image input means into the server.

[0081] The "means for compressing and transmitting" is a function or device for reducing the data size of acquired image data before transmitting it.

[0082] "Image analysis means" means a function or device for extracting product ingredient information from received image data, including using optical character recognition technology.

[0083] The "ingredient collating means" is a function or device for comparing extracted ingredient information with a database and collating it.

[0084] The "component evaluation means" is a function or device for evaluating the balance and bias of components based on component information.

[0085] The "substitute product suggestion means" is a function or device for searching for and suggesting substitute items based on the evaluated ingredient balance and bias.

[0086] The "information display means" is a function or device for visually displaying detailed information about substitute items to the user.

[0087] The skin care product recommendation system of the present invention is designed to enable users to easily select the skin care product that is best suited to their skin condition. To realize this system, the following hardware and software are used.

[0088] The user takes a photo of the label of the skin care product they are using using the camera function of their smartphone. The smartphone then uses a camera app to capture the image data, and the entire system operates based on this image data.

[0089] The device receives the captured image data and compresses it in a format such as JPEG. The compressed image data is then sent to a server via the HTTPS protocol. This communication uses a network device (e.g., a Wi-Fi router).

[0090] The server uses the Google Cloud Vision API to analyze the received image data. This analysis extracts the ingredient information on the label from the image in text format using optical character recognition (OCR). For example, ingredients such as "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a lotion.

[0091] The server then compares the extracted ingredient information with an internal database (e.g., MySQL). Detailed information about each ingredient's effects and recommended dosages are obtained. An internal algorithm is used to evaluate the balance and bias of the ingredients. For example, if the moisturizing ingredients are below the standard value, the server evaluates the user's skin as dry.

[0092] The server then uses a recommendation engine to search for suitable alternative skin care products from the database. Based on the results of the ingredient evaluation, it suggests the best alternative. For example, it may suggest a "highly moisturizing lotion from a specific brand" that has a high moisturizing effect.

[0093] The results of the suggestions are sent from the server to the device and visually displayed to the user through a dedicated application or web browser. Detailed information about the suggested alternative skin care products (ingredients, effects, user reviews, price range, etc.) is displayed. The user can confirm this information and tap the purchase button to be redirected to an online shopping site and purchase the product.

[0094] As a concrete example, a user can take a photo of their lotion label with their smartphone camera and send the image to a server. The server can then use optical character recognition (OCR) technology to extract the ingredient information, compare it with a database, and determine if the moisturizing ingredients are lacking. Based on this, the server can suggest alternative skin care products with higher moisturizing effects and display the information on the device.

[0095] Below is an example of a prompt sentence to input to the generative AI model.

[0096] (Example) Prompt statement:

[0097] The user takes a photo of their own lotion with their smartphone camera and sends the image to the server. The server uses optical character recognition (OCR) technology to extract ingredient information, compares it with a database, and evaluates the balance of ingredients. As a result, it recommends a specific brand of highly moisturizing lotion with a high moisturizing effect. The user can then check this information on their smartphone and select and purchase the most suitable skin care product.

[0098] In this way, the system of the present invention has the function of suggesting optimal skin care products to the user through cooperation between the user, the terminal, and the server.

[0099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0100] Step 1:

[0101] The user takes a picture of a skin care product. Using the smartphone camera app, the user takes a picture so that the label of the skin care product is clearly visible. For example, the user places a lotion bottle on a flat surface and holds the camera horizontally. In this step, the input is the actual skin care product and the smartphone camera, and the output is a JPEG image file.

[0102] Step 2:

[0103] The device compresses the image data and sends it to the server. The device compresses the captured image data into JPEG format and uploads it to the server using the HTTPS protocol. This is done using a network device (e.g., a Wi-Fi router). The input is a JPEG image file, and the compressed image data is sent to the server as the output.

[0104] Step 3:

[0105] The server analyzes the image data. The server calls the Google Cloud Vision API and extracts the ingredient information on the label from the received image data using optical character recognition (OCR) technology. The input is compressed image data, and the output is text data of the extracted ingredient information. Specifically, ingredients such as "hyaluronic acid, glycerin, aloe vera" listed on the label are extracted as text.

[0106] Step 4:

[0107] The server compares the ingredient information with an internal database. The server then compares the extracted ingredient information with a MySQL database to obtain the detailed effects and recommended usage amounts of each ingredient. The extracted ingredient information is text data as input, and the detailed effects of the ingredients and recommended usage amounts are obtained as output. Specifically, it executes an SQL query to obtain the moisturizing effect and recommended concentration of "hyaluronic acid."

[0108] Step 5:

[0109] The server evaluates the balance of ingredients. Using an internal algorithm, the server evaluates the balance and imbalance of ingredients based on the acquired ingredient information. The input is detailed ingredient effects and recommended usage amounts, and the output is the evaluation result of the ingredient balance. Specifically, it detects that moisturizing ingredients are below the standard value and evaluates that the user's skin is dry.

[0110] Step 6:

[0111] The server proposes alternative products. Based on the results of the ingredient evaluation, the server searches its internal database for the optimal alternative skin care product. The input is the evaluation result of the ingredient balance, and the output is information on the proposed alternative skin care product. Specifically, it proposes a "highly moisturizing lotion from a specific brand" with a high moisturizing effect.

[0112] Step 7:

[0113] The device displays the recommendation results. The device receives detailed information about the recommendation results (ingredients, effects, user reviews, price range, etc.) from the server and displays it in a dedicated application or web browser. The input is information about the suggested alternative skin care products, and the output is visually displayed to the user. Specifically, the information is displayed on the smartphone screen and the user views it.

[0114] Step 8:

[0115] The user checks the suggested results and makes a selection / purchase. The user operates the UI of the dedicated app to check detailed information about the suggested alternative skincare products. The information displayed on the smartphone screen is the input, and the decision to select / purchase is the output. The specific operation is that the user taps the purchase button, is redirected to an online shopping site, and purchases the product.

[0116] (Application example 1)

[0117] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0118] The present invention relates to a system that quickly and accurately analyzes the ingredient information of skin care products brought in by users and suggests optimal alternative products based on the user's skin condition and skin care goals. In particular, the objective is to realize this in a way that users can easily use in physical stores, and to streamline users' skin care product selection by providing visual information via smartphones or in-store terminals.

[0119] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0120] In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, a substitute suggestion means for searching for and suggesting substitute items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an information display means for displaying detailed information of the substitute items, and a means for the information display means to visually display information on a physical store terminal, thereby enabling users to visually check skin care products in the physical store and select the optimal substitute product.

[0121] The "image input means" is a device for acquiring an image of a product brought in by a user.

[0122] The "receiving means" is a device that receives the image data acquired by the image input means.

[0123] "Image analysis means" is a device that extracts product ingredient information from received image data.

[0124] The "component collating means" is a device that collates extracted component information with a database.

[0125] The "component evaluation means" is a device that evaluates the balance and bias of components based on component information.

[0126] The "substitute product suggestion means" is a device that searches for and suggests substitute items based on the component balance and bias evaluated by the component evaluation means.

[0127] The "information display means" is a device that visually displays detailed information about a substitute item.

[0128] A "physical store terminal" is a device installed in a physical store that has the function of acquiring images of products brought in by users and analyzing their ingredient information.

[0129] "Optical character recognition technology" is a technology that extracts text information from images.

[0130] An embodiment of the present invention is described below.

[0131] 1. Image input means

[0132] Users take pictures of their skin care products using their smartphone camera or a camera installed in a brick-and-mortar terminal, and the image capture method captures detailed product label information.

[0133] 2. Receiving Method

[0134] The smartphone or the physical store terminal compresses the acquired image data and sends it over the network to the server, which receives the image data.

[0135] 3. Image analysis methods

[0136] The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. Specifically, for example, it extracts ingredients such as "hyaluronic acid, glycerin, and aloe vera" from the label of a lotion.

[0137] 4. Component Matching Method

[0138] The server compares the extracted ingredient information with an internal database containing information on various skin care ingredients to obtain detailed information on each ingredient's effects and recommended dosage.

[0139] 5. Ingredient evaluation methods

[0140] The server evaluates the balance and bias of the ingredients based on the ingredient information. For example, if the moisturizing ingredient is below a standard value, it evaluates the user's skin as dry.

[0141] 6. Means of suggesting alternative products

[0142] Based on the results of the ingredient evaluation, the server searches the database for the most suitable alternative skin care item and suggests it. For example, it may suggest "Brand B's highly moisturizing lotion."

[0143] 7. Information display means

[0144] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range) is sent from the server to a smartphone or a brick-and-mortar store terminal and visually displayed to the user.

[0145] Specific examples

[0146] The system scans the label of a "lotion" brought in by the user with a smartphone camera and extracts "hyaluronic acid, glycerin, aloe vera" from the ingredient information. This information is sent to a server, which then compares it with a database and suggests the most suitable alternative product. For example, based on the results of the ingredient evaluation, "Brand B's highly moisturizing lotion" may be suggested.

[0147] Prompt Sentence Examples

[0148] "Evaluate products containing ingredients such as hyaluronic acid, glycerin, and aloe vera, and suggest alternative products with higher moisturizing benefits."

[0149] This system allows users to efficiently select skin care products in physical stores, enabling them to choose the products that best suit their skin condition.

[0150] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0151] Step 1:

[0152] A user takes an image of their own skin care product using a smartphone camera or a camera on a physical store terminal. The input is image data of the skin care product, and the output is the captured image data. The user launches a camera app to capture an image and saves the image data on the terminal.

[0153] Step 2:

[0154] The terminal compresses the acquired image data and sends it to the server via the network. The input is the captured image data, and the output is the transmission of compressed image data. Specifically, the terminal converts the image data into JPEG format, compresses it, and then uploads it to the server.

[0155] Step 3:

[0156] The server receives image data sent from the terminal. The input is compressed image data, and the output is decompressed image data. The server receives the image data using a network protocol, decompresses it, and stores it in its internal memory.

[0157] Step 4:

[0158] The server analyzes the received image data and extracts the component information in text format using optical character recognition (OCR) technology. The input is the decompressed image data, and the output is the extracted text information. The server uses the OCR engine to analyze the characters in the image and extract it as text information.

[0159] Step 5:

[0160] The server compares the extracted ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage. The input is the extracted text information, and the output is detailed information about the corresponding ingredients. Specifically, the server searches for ingredient information using an SQL query and obtains detailed ingredient information from the database.

[0161] Step 6:

[0162] The server evaluates the balance and bias of ingredients based on the ingredient information. The input is detailed information about the ingredients, and the output is the ingredient balance evaluation result. The server compares the results with the standard values ​​for each ingredient and executes an algorithm to evaluate the balance.

[0163] Step 7:

[0164] The server searches the database for optimal alternative skin care items based on the evaluation results and proposes them. The input is the ingredient balance evaluation results, and the output is a list of optimal alternative products. The server selects the optimal products using an evaluation algorithm and a matching engine.

[0165] Step 8:

[0166] The server sends detailed information about the suggested alternative items to a smartphone or a terminal in a physical store and displays it visually to the user. The input is a list of optimal alternative products, and the output is the information to be displayed. Specifically, the server generates detailed information in HTML format or similar and sends the data to the terminal. The terminal then displays the received HTML data in a browser or similar.

[0167] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0168] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[0169] The present invention also incorporates an emotion engine that recognizes the user's emotional state. By analyzing the user's voice and facial expression data, the emotion engine can recognize the user's emotional state and evaluate their emotional response to the suggested items. This allows the system to understand which suggestions the user is satisfied or dissatisfied with, and reflects this feedback in the next suggestions.

[0170] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0171] Based on this detailed ingredient information, the server uses an ingredient evaluation tool to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Next, the server searches the database for alternative items with high moisturizing effects and uses an alternative suggestion tool to suggest the most suitable item. Specifically, it will suggest a "lotion with high moisturizing power" or suggest the addition of a vitamin C serum, taking into account the overall balance.

[0172] In addition to this suggestion process, the emotion engine senses the user's reactions. If the user responds positively to a suggested product through voice or facial expression, the emotion engine recognizes this and ensures the suggestion is appropriate. Conversely, if a negative reaction is observed, the emotion engine captures that feedback and adjusts the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will suggest a moisturizing lotion with different ingredients or brand.

[0173] Detailed information about the suggested alternative items (ingredients, effects, reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that is best suited to them.

[0174] The emotion engine analyzes the user's emotional state in real time and dynamically adjusts the suggestions based on the user's reaction, thereby increasing user satisfaction and providing the most suitable skin care products. This system allows users to always choose the skin care products that best suit their skin condition and emotional state, thereby maintaining good skin condition.

[0175] The processing flow will be explained below.

[0176] Step 1:

[0177] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[0178] Specifically, take care to photograph skin care product labels and packaging clearly.

[0179] Step 2:

[0180] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[0181] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[0182] Step 3:

[0183] The terminal transmits the image data to the server.

[0184] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[0185] Step 4:

[0186] The server receives the image data.

[0187] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[0188] Step 5:

[0189] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[0190] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[0191] Step 6:

[0192] The server uses the component matching means to match the extracted component information with an internal database.

[0193] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[0194] Step 7:

[0195] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[0196] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[0197] Step 8:

[0198] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[0199] Specifically, the system will create a list of products from the database that will supplement the missing ingredients or balance the skin overall. For example, if the skin lacks moisturizing ingredients, the system will search for a moisturizing lotion.

[0200] Step 9:

[0201] The server uses an emotion engine to recognize the user's emotional state when making alternative suggestions.

[0202] Specifically, the system analyzes the user's voice data and facial expression data to detect positive or negative reactions.

[0203] Step 10:

[0204] The server reflects the emotion recognition results from the emotion engine in proposing alternative products.

[0205] Specifically, the system prioritizes ingredients and products to which the user previously had a positive reaction, and avoids ingredients and products to which the user previously had a negative reaction.

[0206] Step 11:

[0207] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[0208] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[0209] Step 12:

[0210] The terminal displays detailed information of the substitute item received from the server to the user.

[0211] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[0212] Step 13:

[0213] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[0214] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[0215] This detailed processing step provides users with specific and intuitive assistance in choosing the skin care products that best suit their skin and emotional state.

[0216] Example 2

[0217] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] Current skincare product recommendation systems lack the ability to easily obtain ingredient information about products currently in use and then suggest appropriate alternatives based on that information. Furthermore, they do not optimize recommendations based on the user's emotional responses, leaving insufficient means to increase user satisfaction. Furthermore, there is no mechanism for incorporating the user's emotional feedback on a suggested product into future recommendations. To address these issues, an advanced recommendation system is needed that incorporates not only image analysis and ingredient evaluation, but also the user's emotional responses.

[0219] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image analysis means for extracting product ingredient information from image data, an ingredient comparison means for comparing the ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the extracted ingredient information, an alternative product suggestion means for searching for and suggesting alternative products, an emotion analysis means for analyzing the user's emotional response to the suggested alternative items, and an information display means for displaying detailed information about the alternative items. This makes it possible to efficiently extract and evaluate ingredient information for skin care products currently used by the user and suggest appropriate alternative products based on the user's skin condition. Furthermore, by analyzing the user's emotional response to the suggested products and reflecting this in the next suggestion, user satisfaction can be increased.

[0220] The "image input means" is a device that acquires image data of the product being used by the user, and in many cases refers to the camera of a mobile terminal.

[0221] The "means for receiving image data" refers to a device or function for receiving image data acquired by the image input means.

[0222] "Image analysis means" refers to technology or equipment for extracting product ingredient information from received image data, and specifically, optical character recognition technology is often used.

[0223] "Ingredient matching means" refers to a technology or device for matching ingredient information extracted by the image analysis means with a database and obtaining information such as detailed effects and recommended usage amounts.

[0224] "Ingredient evaluation means" refers to the technology and equipment used to evaluate the balance and bias of ingredients based on extracted and collated ingredient information.

[0225] The "substitute product suggestion means" refers to a technology or device for searching for and suggesting the most suitable substitute product based on the results of evaluation by the component evaluation means.

[0226] "Emotion analysis means" refers to technology or devices for analyzing a user's emotional response to a proposed substitute item, and includes functions for analyzing voice and facial expression data.

[0227] "Information display means" refers to a device or technology for visually displaying detailed information about the proposed substitute item to the user.

[0228] "Ingredient information" refers to detailed data such as the names, effects, and recommended usage amounts of various ingredients contained in skin care products.

[0229] "Database" refers to a system that stores information used to collate and evaluate ingredient information.

[0230] The skin care product recommendation system of this invention starts when a user takes a picture of a skin care product they own with a smartphone camera and sends the image to a server. The terminal receives the image data and uploads it to the server via a network. At this stage, the terminal is a mobile terminal such as a smartphone.

[0231] The server uses optical character recognition (OCR) technology to analyze the received image data. For example, the server extracts the ingredients "hyaluronic acid, glycerin, aloe vera" from the label of a lotion that has been photographed. The extracted ingredient information is compared with the server's internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0232] Next, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Based on this evaluation result, the server searches the database for suitable alternative items and suggests them to the user using an alternative suggestion means. Specifically, it will suggest items such as "highly moisturizing lotion" and "vitamin C serum."

[0233] Furthermore, this system incorporates an emotion engine. The emotion engine on the server analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user responds "I like it" to a suggested product through voice or facial expression, the emotion engine recognizes this as a positive reaction and confirms that the suggestion was appropriate. Conversely, if a negative reaction is observed, that feedback is reflected in the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will be a moisturizing lotion with different ingredients or brand.

[0234] Detailed information on the suggested alternative items is sent from the server to the terminal, and the user can visually check it through the terminal's information display means. The information includes ingredients, effects, reviews, price range, etc. Based on this information, the user can select and purchase the most suitable item.

[0235] Specific examples

[0236] For example, if a user is using a lotion containing "hyaluronic acid, glycerin, and aloe vera," the server compares the ingredient information with the database to obtain detailed effects and recommended usage amounts. The server then evaluates that the moisturizing effect is below the standard value and suggests a "lotion with high moisturizing power." If the user responds to this suggestion by saying "like," the emotion engine recognizes this and determines that the suggestion was appropriate.

[0237] Prompt Sentence Examples

[0238] "Please tell me the ingredients of the lotion you are currently using."

[0239] "Please suggest skin care products with high moisturizing effects."

[0240] Please tell me an alternative to this lotion.

[0241] In this way, by combining advanced image analysis technology with emotion analysis functions, this system can increase user satisfaction and provide optimal skin care products.

[0242] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0243] Step 1:

[0244] The user uses the camera function of their smartphone to take a photo of their skin care product. Specifically, the user opens the camera app and takes a photo of the label of the skin care product.

[0245] Input: Skincare product image

[0246] Output: Captured image data

[0247] Step 2:

[0248] The device compresses the captured image data and uploads it to a server via the network, for example by converting it to JPEG format to reduce the file size.

[0249] Input: Photographed image data

[0250] Output: Compressed image data

[0251] Step 3:

[0252] The server uses optical character recognition (OCR) technology to analyze the received image data. The server inputs the image data into an OCR engine and extracts the component information as text.

[0253] Input: Compressed image data

[0254] Output: Extracted ingredient information (text format)

[0255] Step 4:

[0256] The server compares the extracted ingredient information with its internal database, specifically by executing a database query using the ingredient name as a key to obtain detailed information about each ingredient's effects and recommended dosage.

[0257] Input: Extracted ingredient information

[0258] Output: Detailed ingredient information (effects, recommended dosage, etc.)

[0259] Step 5:

[0260] The server uses the component evaluation means to evaluate the balance and bias of the components. For example, if the amount of a specific moisturizing component is determined to be below a reference value, the server determines the user's skin condition as dry.

[0261] Input: Detailed ingredient information

[0262] Output: Evaluation results (ingredients balance, skin condition)

[0263] Step 6:

[0264] The server searches for and suggests alternative products from a database based on the evaluation results.The server uses the alternative product suggestion means to list and suggest suitable skin care products (e.g., moisturizing lotion).

[0265] Input: Evaluation result

[0266] Output: A list of suggested replacements

[0267] Step 7:

[0268] The server's emotion engine analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, it uses voice and facial expression data obtained from the user's remote camera or microphone as input and applies an emotion analysis algorithm to determine whether the reaction is positive or negative.

[0269] Input: Voice data, facial expression data

[0270] Output: Emotional response analysis results

[0271] Step 8:

[0272] The server sends detailed information about the proposed substitute items to the terminal and displays it to the user through the terminal's information display means. The user can then check this information and select the most suitable item.

[0273] Input: List of suggested replacements, emotional response analysis results

[0274] Output: Detailed information displayed on the terminal

[0275] Step 9:

[0276] The user checks the detailed information of the suggested items displayed on the device, selects the item that best suits them, and then purchases it. Specifically, the user clicks on a link to access the online shopping site and completes the purchase procedure.

[0277] Input: Detailed information (ingredients, effects, reviews, price range)

[0278] Output: Purchase order for selected skin care products

[0279] (Application example 2)

[0280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] In recent years, with the diversification of skin care products, users have found it difficult to select the product that best suits their skin. Furthermore, there are few feedback systems based on the user's emotional state, making it a challenge to improve user satisfaction. There is a need to solve these issues and provide a system that recommends more effective and satisfying skin care products.

[0282] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an emotion analysis means for analyzing the user's emotional state, a feedback means for evaluating the user's emotional response using the emotion analysis means and adjusting the next suggestion, and an information display means for displaying detailed information about the alternative items. This enables the user to select optimal skin care products based on their skin condition and emotional state.

[0283] "Image input means" refers to a device or function for acquiring image data captured by a user.

[0284] The "receiving means" refers to a device or function for receiving image data transmitted from the image input means.

[0285] "Image analysis means" refers to a device or function for extracting product ingredient information from received image data.

[0286] The "ingredient collating means" refers to a device or function for collating extracted ingredient information with a database.

[0287] The "component evaluation means" refers to a device or function for evaluating the balance and bias of components based on component information.

[0288] The "substitute product suggestion means" refers to a device or function for searching for and suggesting substitute items based on the component balance and bias evaluated by the component evaluation means.

[0289] "Information display means" refers to a device or function for displaying detailed information about a substitute item.

[0290] "Emotion analysis means" refers to a device or function for analyzing the user's emotional state.

[0291] "Feedback means" refers to a device or function that evaluates the user's emotional response using emotion analysis means and adjusts the content of the next suggestion.

[0292] In order to implement the present invention, it is necessary to construct the following system.

[0293] System Overview

[0294] The system includes an image input unit, a receiving unit, an image analysis unit, an ingredient matching unit, an ingredient evaluation unit, an alternative product suggestion unit, an information display unit, an emotion analysis unit, and a feedback unit. By linking these units, the system can suggest optimal skin care products to users.

[0295] Hardware and Software

[0296] Hardware:

[0297] Smartphone: Used as an image input and user interface

[0298] Servers: Data processing and storage

[0299] Camera: Built-in camera on smartphone

[0300] Microphone: A device for voice input.

[0301] software:

[0302] Image Analysis: OpenCV and Google Tesseract OCR

[0303] Server communication: requests module (Python)

[0304] Emotion analysis: emotion_recognition library (Python)

[0305] Programming: Python language used

[0306] Processing steps

[0307] 1. Image Acquisition:

[0308] Users take a photo of a skin care product label using their smartphone camera, and the image data is temporarily stored in the smartphone's memory and then sent to the server.

[0309] 2. Image Analysis:

[0310] The server analyzes the received image data and extracts component information from the image using Google Tesseract OCR.

[0311] 3. Ingredient Matching:

[0312] The extracted ingredient information is compared with a database on the server, and detailed information about the ingredients, their effects, recommended usage, and other information is retrieved from the database.

[0313] 4. Ingredients evaluation:

[0314] Based on the acquired ingredient information, the server evaluates the balance and imbalance of ingredients, particularly determining whether moisturizing ingredients and nutritional supplement ingredients are appropriate.

[0315] 5. Substitute suggestions:

[0316] Based on the results of the ingredient evaluation, the system proposes alternative skin care products that are best suited to the user's skin condition. The system searches for optimal products from a database in the server via an alternative product suggestion means and proposes them to the user.

[0317] 6. Information display:

[0318] Users can view detailed information about suggested alternative products via their smartphone, including ingredients, effects, other users' reviews, and prices.

[0319] 7. Emotion analysis:

[0320] The emotion analysis means analyzes the user's voice and facial expression data to evaluate their emotional reaction to the proposed product. If a positive reaction is obtained, it is fed back to the server and used to improve the next proposal.

[0321] 8. Feedback and Suggested Adjustments:

[0322] The next recommendation will be optimized based on the feedback information obtained through sentiment analysis. For example, if there is a negative reaction, the next recommendation will be a product with different ingredients or brand.

[0323] Specific examples

[0324] Consider a scenario where a user takes a photo of the label of a "lotion containing hyaluronic acid" with their smartphone camera in the skincare section of a physical store. The program captures the image and uses Tesseract OCR to extract ingredient information such as "hyaluronic acid, glycerin, aloe vera." The data sent to the server is analyzed, and the result is returned indicating that the moisturizing ingredients are highly rated.

[0325] Furthermore, based on the user's feedback, the company analyzed their sentiment and found that they responded positively to the question, "Is this lotion effective for dry skin?", so it decided to continue recommending skin care products with high moisturizing power from next time onwards. These results are displayed on the AI ​​assistant's display in the physical store.

[0326] Prompt Sentence Examples

[0327] "How can we develop an application that analyzes the ingredients of skin care products taken by customers in physical stores and suggests the best alternatives based on their skin condition and emotional state?"

[0328] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0329] Program processing steps

[0330] Step 1:

[0331] A user takes a photo of a skin care product label using a smartphone at the skin care section of a physical store. The image is captured using the smartphone's camera function, and the captured image data is saved on the smartphone. The input is the image data of the label, and the output is the saved image file.

[0332] Step 2:

[0333] The device sends the captured image data to a server. Specifically, the image file on the smartphone is compressed and uploaded to the server via the Internet. The input is the image file, and the output is the image data sent to the server.

[0334] Step 3:

[0335] The server analyzes the received image data. It uses Google Tesseract OCR to extract component information from the image in text format. The input is image data, and the output is component information in text format.

[0336] Step 4:

[0337] The server compares the extracted ingredient information with an internal database. Using the ingredient comparison means, detailed information, effects, and recommended usage of the relevant ingredients are obtained from the ingredient database. The input is ingredient information in text format, and the output is detailed information about the ingredients.

[0338] Step 5:

[0339] The server uses the ingredient evaluation means to evaluate the balance and imbalance of ingredients based on ingredient information. In particular, it checks the balance of moisturizing ingredients and nutritional supplement ingredients. The input is detailed information about the ingredients, and the output is the evaluation result of the ingredient balance. Specifically, if the moisturizing ingredients are below the standard value, the evaluation result will show "insufficient moisturizing ingredients."

[0340] Step 6:

[0341] The server uses the substitute suggestion means based on the ingredient evaluation results to search for and suggest alternative skin care products that are best suited to the user's skin condition. Products that meet the criteria are searched for from the skin care product database within the server. The input is the ingredient evaluation results, and the output is a list of suggested alternative skin care products.

[0342] Step 7:

[0343] The server presents detailed information about the suggested alternative items to the user through an information display means. The smartphone display shows the product's ingredients, effects, reviews from other users, price, etc. The input is a list of alternative skin care products, and the output is detailed information displayed on the smartphone screen.

[0344] Step 8:

[0345] The user's voice and facial expressions are collected through a camera and microphone and sent from the device to a server. The input is the user's voice and facial expression data, and the output is the emotional data sent to the server.

[0346] Step 9:

[0347] The server analyzes the user's emotional state using emotion analysis. Specifically, it uses the emotion_recognition library to identify emotions from voice and facial expression data and determine whether they are positive or negative. The input is the user's voice and facial expression data, and the output is the result of the emotion analysis.

[0348] Step 10:

[0349] The server uses a feedback mechanism to adjust the next recommendation based on the sentiment analysis results. For example, if a negative reaction is shown, it may suggest a product with different ingredients or brand the next time. The input is the sentiment analysis results, and the output is the next recommendation.

[0350] The above processing steps allow the user to select the most suitable skin care product based on their skin condition and emotional state.

[0351] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0352] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0353] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0354] [Second embodiment]

[0355] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0356] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0357] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0358] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0359] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0360] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0361] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0362] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0365] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0366] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0367] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[0368] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0369] Based on this detailed ingredient information, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below the standard value, it will evaluate that the user's skin is dry. Next, the server searches the database for alternative items with high moisturizing effects and suggests the most suitable item using an alternative suggestion means. Specifically, it may suggest "Brand B's highly moisturizing lotion."

[0370] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that best suits them.

[0371] This system also supports the use of multiple skincare products, and can comprehensively recommend items to supplement missing or excess ingredients while taking into account the overall balance of ingredients. This allows users to choose the skincare products that are best suited to their skin condition, helping to maintain good skin condition.

[0372] The processing flow will be explained below.

[0373] Step 1:

[0374] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[0375] Specifically, take care to photograph skin care product labels and packaging clearly.

[0376] Step 2:

[0377] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[0378] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[0379] Step 3:

[0380] The terminal transmits the image data to the server.

[0381] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[0382] Step 4:

[0383] The server receives the image data.

[0384] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[0385] Step 5:

[0386] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[0387] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[0388] Step 6:

[0389] The server uses the component matching means to match the extracted component information with an internal database.

[0390] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[0391] Step 7:

[0392] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[0393] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[0394] Step 8:

[0395] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[0396] Specifically, the database will be used to create a list of products that supplement missing ingredients and products that help to achieve overall balance.

[0397] Step 9:

[0398] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[0399] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[0400] Step 10:

[0401] The terminal displays detailed information of the substitute item received from the server to the user.

[0402] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[0403] Step 11:

[0404] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[0405] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[0406] This detailed processing step provides users with specific and intuitive assistance in choosing the best skin care products for their skin condition.

[0407] Example 1

[0408] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0409] Conventional skin care product recommendation systems have difficulty automatically selecting the optimal skin care product for a user's skin condition. Furthermore, systems that accurately grasp the product's ingredient information and then recommend optimal alternative products based on that information are insufficient. As a result, users end up purchasing products at their own discretion, which can prevent them from maximizing their effectiveness.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0411] In this invention, the server includes an image input means for a user to take an image of a skin care product, a means for receiving image data acquired by the image input means, a means for compressing and transmitting the received image data, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, and an information display means for displaying detailed information about the alternative items. This allows users to easily select and purchase skin care products that are best suited to their skin condition.

[0412] "Image input means" refers to a function or device for taking an image of a skin care product, and includes the camera of a mobile terminal used by the user.

[0413] The "receiving means" is a function or device for taking the image data acquired by the image input means into the server.

[0414] The "means for compressing and transmitting" is a function or device for reducing the data size of acquired image data before transmitting it.

[0415] "Image analysis means" means a function or device for extracting product ingredient information from received image data, including using optical character recognition technology.

[0416] The "ingredient collating means" is a function or device for comparing extracted ingredient information with a database and collating it.

[0417] The "component evaluation means" is a function or device for evaluating the balance and bias of components based on component information.

[0418] The "substitute product suggestion means" is a function or device for searching for and suggesting substitute items based on the evaluated ingredient balance and bias.

[0419] The "information display means" is a function or device for visually displaying detailed information about substitute items to the user.

[0420] The skin care product recommendation system of the present invention is designed to enable users to easily select the skin care product that is best suited to their skin condition. To realize this system, the following hardware and software are used.

[0421] The user takes a photo of the label of the skin care product they are using using the camera function of their smartphone. The smartphone then uses a camera app to capture the image data, and the entire system operates based on this image data.

[0422] The device receives the captured image data and compresses it in a format such as JPEG. The compressed image data is then sent to a server via the HTTPS protocol. This communication uses a network device (e.g., a Wi-Fi router).

[0423] The server uses the Google Cloud Vision API to analyze the received image data. This analysis extracts the ingredient information on the label from the image in text format using optical character recognition (OCR). For example, ingredients such as "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a lotion.

[0424] The server then compares the extracted ingredient information with an internal database (e.g., MySQL). Detailed information about each ingredient's effects and recommended dosages are obtained. An internal algorithm is used to evaluate the balance and bias of the ingredients. For example, if the moisturizing ingredients are below the standard value, the server evaluates the user's skin as dry.

[0425] The server then uses a recommendation engine to search for suitable alternative skin care products from the database. Based on the results of the ingredient evaluation, it suggests the best alternative. For example, it may suggest a "highly moisturizing lotion from a specific brand" that has a high moisturizing effect.

[0426] The results of the suggestions are sent from the server to the device and visually displayed to the user through a dedicated application or web browser. Detailed information about the suggested alternative skin care products (ingredients, effects, user reviews, price range, etc.) is displayed. The user can confirm this information and tap the purchase button to be redirected to an online shopping site and purchase the product.

[0427] As a concrete example, a user can take a photo of their lotion label with their smartphone camera and send the image to a server. The server can then use optical character recognition (OCR) technology to extract the ingredient information, compare it with a database, and determine if the moisturizing ingredients are lacking. Based on this, the server can suggest alternative skin care products with higher moisturizing effects and display the information on the device.

[0428] Below is an example of a prompt sentence to input to the generative AI model.

[0429] (Example) Prompt statement:

[0430] The user takes a photo of their own lotion with their smartphone camera and sends the image to the server. The server uses optical character recognition (OCR) technology to extract ingredient information, compares it with a database, and evaluates the balance of ingredients. As a result, it recommends a specific brand of highly moisturizing lotion with a high moisturizing effect. The user can then check this information on their smartphone and select and purchase the most suitable skin care product.

[0431] In this way, the system of the present invention has the function of suggesting optimal skin care products to the user through cooperation between the user, the terminal, and the server.

[0432] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0433] Step 1:

[0434] The user takes a picture of a skin care product. Using the smartphone camera app, the user takes a picture so that the label of the skin care product is clearly visible. For example, the user places a lotion bottle on a flat surface and holds the camera horizontally. In this step, the input is the actual skin care product and the smartphone camera, and the output is a JPEG image file.

[0435] Step 2:

[0436] The device compresses the image data and sends it to the server. The device compresses the captured image data into JPEG format and uploads it to the server using the HTTPS protocol. This is done using a network device (e.g., a Wi-Fi router). The input is a JPEG image file, and the compressed image data is sent to the server as the output.

[0437] Step 3:

[0438] The server analyzes the image data. The server calls the Google Cloud Vision API and extracts the ingredient information on the label from the received image data using optical character recognition (OCR) technology. The input is compressed image data, and the output is text data of the extracted ingredient information. Specifically, ingredients such as "hyaluronic acid, glycerin, aloe vera" listed on the label are extracted as text.

[0439] Step 4:

[0440] The server compares the ingredient information with an internal database. The server then compares the extracted ingredient information with a MySQL database to obtain the detailed effects and recommended usage amounts of each ingredient. The extracted ingredient information is text data as input, and the detailed effects of the ingredients and recommended usage amounts are obtained as output. Specifically, it executes an SQL query to obtain the moisturizing effect and recommended concentration of "hyaluronic acid."

[0441] Step 5:

[0442] The server evaluates the balance of ingredients. Using an internal algorithm, the server evaluates the balance and imbalance of ingredients based on the acquired ingredient information. The input is detailed ingredient effects and recommended usage amounts, and the output is the evaluation result of the ingredient balance. Specifically, it detects that moisturizing ingredients are below the standard value and evaluates that the user's skin is dry.

[0443] Step 6:

[0444] The server proposes alternative products. Based on the results of the ingredient evaluation, the server searches its internal database for the optimal alternative skin care product. The input is the evaluation result of the ingredient balance, and the output is information on the proposed alternative skin care product. Specifically, it proposes a "highly moisturizing lotion from a specific brand" with a high moisturizing effect.

[0445] Step 7:

[0446] The device displays the recommendation results. The device receives detailed information about the recommendation results (ingredients, effects, user reviews, price range, etc.) from the server and displays it in a dedicated application or web browser. The input is information about the suggested alternative skin care products, and the output is visually displayed to the user. Specifically, the information is displayed on the smartphone screen and the user views it.

[0447] Step 8:

[0448] The user checks the suggested results and makes a selection / purchase. The user operates the UI of the dedicated app to check detailed information about the suggested alternative skincare products. The information displayed on the smartphone screen is the input, and the decision to select / purchase is the output. The specific operation is that the user taps the purchase button, is redirected to an online shopping site, and purchases the product.

[0449] (Application example 1)

[0450] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0451] The present invention relates to a system that quickly and accurately analyzes the ingredient information of skin care products brought in by users and suggests optimal alternative products based on the user's skin condition and skin care goals. In particular, the objective is to realize this in a way that users can easily use in physical stores, and to streamline users' skin care product selection by providing visual information via smartphones or in-store terminals.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0453] In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, a substitute suggestion means for searching for and suggesting substitute items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an information display means for displaying detailed information of the substitute items, and a means for the information display means to visually display information on a physical store terminal, thereby enabling users to visually check skin care products in the physical store and select the optimal substitute product.

[0454] The "image input means" is a device for acquiring an image of a product brought in by a user.

[0455] The "receiving means" is a device that receives the image data acquired by the image input means.

[0456] "Image analysis means" is a device that extracts product ingredient information from received image data.

[0457] The "component collating means" is a device that collates extracted component information with a database.

[0458] The "component evaluation means" is a device that evaluates the balance and bias of components based on component information.

[0459] The "substitute product suggestion means" is a device that searches for and suggests substitute items based on the component balance and bias evaluated by the component evaluation means.

[0460] The "information display means" is a device that visually displays detailed information about a substitute item.

[0461] A "physical store terminal" is a device installed in a physical store that has the function of acquiring images of products brought in by users and analyzing their ingredient information.

[0462] "Optical character recognition technology" is a technology that extracts text information from images.

[0463] An embodiment of the present invention is described below.

[0464] 1. Image input means

[0465] Users take pictures of their skin care products using their smartphone camera or a camera installed in a brick-and-mortar terminal, and the image capture method captures detailed product label information.

[0466] 2. Receiving Method

[0467] The smartphone or the physical store terminal compresses the acquired image data and sends it over the network to the server, which receives the image data.

[0468] 3. Image analysis methods

[0469] The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. Specifically, for example, it extracts ingredients such as "hyaluronic acid, glycerin, and aloe vera" from the label of a lotion.

[0470] 4. Component Matching Method

[0471] The server compares the extracted ingredient information with an internal database containing information on various skin care ingredients to obtain detailed information on each ingredient's effects and recommended dosage.

[0472] 5. Ingredient evaluation methods

[0473] The server evaluates the balance and bias of the ingredients based on the ingredient information. For example, if the moisturizing ingredient is below a standard value, it evaluates the user's skin as dry.

[0474] 6. Means of suggesting alternative products

[0475] Based on the results of the ingredient evaluation, the server searches the database for the most suitable alternative skin care item and suggests it. For example, it may suggest "Brand B's highly moisturizing lotion."

[0476] 7. Information display means

[0477] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range) is sent from the server to a smartphone or a brick-and-mortar store terminal and visually displayed to the user.

[0478] Specific examples

[0479] The system scans the label of a "lotion" brought in by the user with a smartphone camera and extracts "hyaluronic acid, glycerin, aloe vera" from the ingredient information. This information is sent to a server, which then compares it with a database and suggests the most suitable alternative product. For example, based on the results of the ingredient evaluation, "Brand B's highly moisturizing lotion" may be suggested.

[0480] Prompt Sentence Examples

[0481] "Evaluate products containing ingredients such as hyaluronic acid, glycerin, and aloe vera, and suggest alternative products with higher moisturizing benefits."

[0482] This system allows users to efficiently select skin care products in physical stores, enabling them to choose the products that best suit their skin condition.

[0483] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0484] Step 1:

[0485] A user takes an image of their own skin care product using a smartphone camera or a camera on a physical store terminal. The input is image data of the skin care product, and the output is the captured image data. The user launches a camera app to capture an image and saves the image data on the terminal.

[0486] Step 2:

[0487] The terminal compresses the acquired image data and sends it to the server via the network. The input is the captured image data, and the output is the transmission of compressed image data. Specifically, the terminal converts the image data into JPEG format, compresses it, and then uploads it to the server.

[0488] Step 3:

[0489] The server receives image data sent from the terminal. The input is compressed image data, and the output is decompressed image data. The server receives the image data using a network protocol, decompresses it, and stores it in its internal memory.

[0490] Step 4:

[0491] The server analyzes the received image data and extracts the component information in text format using optical character recognition (OCR) technology. The input is the decompressed image data, and the output is the extracted text information. The server uses the OCR engine to analyze the characters in the image and extract it as text information.

[0492] Step 5:

[0493] The server compares the extracted ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage. The input is the extracted text information, and the output is detailed information about the corresponding ingredients. Specifically, the server searches for ingredient information using an SQL query and obtains detailed ingredient information from the database.

[0494] Step 6:

[0495] The server evaluates the balance and bias of ingredients based on the ingredient information. The input is detailed information about the ingredients, and the output is the ingredient balance evaluation result. The server compares the results with the standard values ​​for each ingredient and executes an algorithm to evaluate the balance.

[0496] Step 7:

[0497] The server searches the database for optimal alternative skin care items based on the evaluation results and proposes them. The input is the ingredient balance evaluation results, and the output is a list of optimal alternative products. The server selects the optimal products using an evaluation algorithm and a matching engine.

[0498] Step 8:

[0499] The server sends detailed information about the suggested alternative items to a smartphone or a terminal in a physical store and displays it visually to the user. The input is a list of optimal alternative products, and the output is the information to be displayed. Specifically, the server generates detailed information in HTML format or similar and sends the data to the terminal. The terminal then displays the received HTML data in a browser or similar.

[0500] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0501] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[0502] The present invention also incorporates an emotion engine that recognizes the user's emotional state. By analyzing the user's voice and facial expression data, the emotion engine can recognize the user's emotional state and evaluate their emotional response to the suggested items. This allows the system to understand which suggestions the user is satisfied or dissatisfied with, and reflects this feedback in the next suggestions.

[0503] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0504] Based on this detailed ingredient information, the server uses an ingredient evaluation tool to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Next, the server searches the database for alternative items with high moisturizing effects and uses an alternative suggestion tool to suggest the most suitable item. Specifically, it will suggest a "lotion with high moisturizing power" or suggest the addition of a vitamin C serum, taking into account the overall balance.

[0505] In addition to this suggestion process, the emotion engine senses the user's reactions. If the user responds positively to a suggested product through voice or facial expression, the emotion engine recognizes this and ensures the suggestion is appropriate. Conversely, if a negative reaction is observed, the emotion engine captures that feedback and adjusts the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will suggest a moisturizing lotion with different ingredients or brand.

[0506] Detailed information about the suggested alternative items (ingredients, effects, reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that is best suited to them.

[0507] The emotion engine analyzes the user's emotional state in real time and dynamically adjusts the suggestions based on the user's reaction, thereby increasing user satisfaction and providing the most suitable skin care products. This system allows users to always choose the skin care products that best suit their skin condition and emotional state, thereby maintaining good skin condition.

[0508] The processing flow will be explained below.

[0509] Step 1:

[0510] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[0511] Specifically, take care to photograph skin care product labels and packaging clearly.

[0512] Step 2:

[0513] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[0514] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[0515] Step 3:

[0516] The terminal transmits the image data to the server.

[0517] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[0518] Step 4:

[0519] The server receives the image data.

[0520] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[0521] Step 5:

[0522] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[0523] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[0524] Step 6:

[0525] The server uses the component matching means to match the extracted component information with an internal database.

[0526] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[0527] Step 7:

[0528] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[0529] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[0530] Step 8:

[0531] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[0532] Specifically, the system will create a list of products from the database that will supplement the missing ingredients or balance the skin overall. For example, if the skin lacks moisturizing ingredients, the system will search for a moisturizing lotion.

[0533] Step 9:

[0534] The server uses an emotion engine to recognize the user's emotional state when making alternative suggestions.

[0535] Specifically, the system analyzes the user's voice data and facial expression data to detect positive or negative reactions.

[0536] Step 10:

[0537] The server reflects the emotion recognition results from the emotion engine in proposing alternative products.

[0538] Specifically, the system prioritizes ingredients and products to which the user previously had a positive reaction, and avoids ingredients and products to which the user previously had a negative reaction.

[0539] Step 11:

[0540] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[0541] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[0542] Step 12:

[0543] The terminal displays detailed information of the substitute item received from the server to the user.

[0544] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[0545] Step 13:

[0546] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[0547] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[0548] This detailed processing step provides users with specific and intuitive assistance in choosing the skin care products that best suit their skin and emotional state.

[0549] Example 2

[0550] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0551] Current skincare product recommendation systems lack the ability to easily obtain ingredient information about products currently in use and then suggest appropriate alternatives based on that information. Furthermore, they do not optimize recommendations based on the user's emotional responses, leaving insufficient means to increase user satisfaction. Furthermore, there is no mechanism for incorporating the user's emotional feedback on a suggested product into future recommendations. To address these issues, an advanced recommendation system is needed that incorporates not only image analysis and ingredient evaluation, but also the user's emotional responses.

[0552] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image analysis means for extracting product ingredient information from image data, an ingredient comparison means for comparing the ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the extracted ingredient information, an alternative product suggestion means for searching for and suggesting alternative products, an emotion analysis means for analyzing the user's emotional response to the suggested alternative items, and an information display means for displaying detailed information about the alternative items. This makes it possible to efficiently extract and evaluate ingredient information for skin care products currently used by the user and suggest appropriate alternative products based on the user's skin condition. Furthermore, by analyzing the user's emotional response to the suggested products and reflecting this in the next suggestion, user satisfaction can be increased.

[0553] The "image input means" is a device that acquires image data of the product being used by the user, and in many cases refers to the camera of a mobile terminal.

[0554] The "means for receiving image data" refers to a device or function for receiving image data acquired by the image input means.

[0555] "Image analysis means" refers to technology or equipment for extracting product ingredient information from received image data, and specifically, optical character recognition technology is often used.

[0556] "Ingredient matching means" refers to a technology or device for matching ingredient information extracted by the image analysis means with a database and obtaining information such as detailed effects and recommended usage amounts.

[0557] "Ingredient evaluation means" refers to the technology and equipment used to evaluate the balance and bias of ingredients based on extracted and collated ingredient information.

[0558] The "substitute product suggestion means" refers to a technology or device for searching for and suggesting the most suitable substitute product based on the results of evaluation by the component evaluation means.

[0559] "Emotion analysis means" refers to technology or devices for analyzing a user's emotional response to a proposed substitute item, and includes functions for analyzing voice and facial expression data.

[0560] "Information display means" refers to a device or technology for visually displaying detailed information about the proposed substitute item to the user.

[0561] "Ingredient information" refers to detailed data such as the names, effects, and recommended usage amounts of various ingredients contained in skin care products.

[0562] "Database" refers to a system that stores information used to collate and evaluate ingredient information.

[0563] The skin care product recommendation system of this invention starts when a user takes a picture of a skin care product they own with a smartphone camera and sends the image to a server. The terminal receives the image data and uploads it to the server via a network. At this stage, the terminal is a mobile terminal such as a smartphone.

[0564] The server uses optical character recognition (OCR) technology to analyze the received image data. For example, the server extracts the ingredients "hyaluronic acid, glycerin, aloe vera" from the label of a lotion that has been photographed. The extracted ingredient information is compared with the server's internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0565] Next, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Based on this evaluation result, the server searches the database for suitable alternative items and suggests them to the user using an alternative suggestion means. Specifically, it will suggest items such as "highly moisturizing lotion" and "vitamin C serum."

[0566] Furthermore, this system incorporates an emotion engine. The emotion engine on the server analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user responds "I like it" to a suggested product through voice or facial expression, the emotion engine recognizes this as a positive reaction and confirms that the suggestion was appropriate. Conversely, if a negative reaction is observed, that feedback is reflected in the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will be a moisturizing lotion with different ingredients or brand.

[0567] Detailed information on the suggested alternative items is sent from the server to the terminal, and the user can visually check it through the terminal's information display means. The information includes ingredients, effects, reviews, price range, etc. Based on this information, the user can select and purchase the most suitable item.

[0568] Specific examples

[0569] For example, if a user is using a lotion containing "hyaluronic acid, glycerin, and aloe vera," the server compares the ingredient information with the database to obtain detailed effects and recommended usage amounts. The server then evaluates that the moisturizing effect is below the standard value and suggests a "lotion with high moisturizing power." If the user responds to this suggestion by saying "like," the emotion engine recognizes this and determines that the suggestion was appropriate.

[0570] Prompt Sentence Examples

[0571] "Please tell me the ingredients of the lotion you are currently using."

[0572] "Please suggest skin care products with high moisturizing effects."

[0573] Please tell me an alternative to this lotion.

[0574] In this way, by combining advanced image analysis technology with emotion analysis functions, this system can increase user satisfaction and provide optimal skin care products.

[0575] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0576] Step 1:

[0577] The user uses the camera function of their smartphone to take a photo of their skin care product. Specifically, the user opens the camera app and takes a photo of the label of the skin care product.

[0578] Input: Skincare product image

[0579] Output: Captured image data

[0580] Step 2:

[0581] The device compresses the captured image data and uploads it to a server via the network, for example by converting it to JPEG format to reduce the file size.

[0582] Input: Photographed image data

[0583] Output: Compressed image data

[0584] Step 3:

[0585] The server uses optical character recognition (OCR) technology to analyze the received image data. The server inputs the image data into an OCR engine and extracts the component information as text.

[0586] Input: Compressed image data

[0587] Output: Extracted ingredient information (text format)

[0588] Step 4:

[0589] The server compares the extracted ingredient information with its internal database, specifically by executing a database query using the ingredient name as a key to obtain detailed information about each ingredient's effects and recommended dosage.

[0590] Input: Extracted ingredient information

[0591] Output: Detailed ingredient information (effects, recommended dosage, etc.)

[0592] Step 5:

[0593] The server uses the component evaluation means to evaluate the balance and bias of the components. For example, if the amount of a specific moisturizing component is determined to be below a reference value, the server determines the user's skin condition as dry.

[0594] Input: Detailed ingredient information

[0595] Output: Evaluation results (ingredients balance, skin condition)

[0596] Step 6:

[0597] The server searches for and suggests alternative products from a database based on the evaluation results.The server uses the alternative product suggestion means to list and suggest suitable skin care products (e.g., moisturizing lotion).

[0598] Input: Evaluation result

[0599] Output: A list of suggested replacements

[0600] Step 7:

[0601] The server's emotion engine analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, it uses voice and facial expression data obtained from the user's remote camera or microphone as input and applies an emotion analysis algorithm to determine whether the reaction is positive or negative.

[0602] Input: Voice data, facial expression data

[0603] Output: Emotional response analysis results

[0604] Step 8:

[0605] The server sends detailed information about the proposed substitute items to the terminal and displays it to the user through the terminal's information display means. The user can then check this information and select the most suitable item.

[0606] Input: List of suggested replacements, emotional response analysis results

[0607] Output: Detailed information displayed on the terminal

[0608] Step 9:

[0609] The user checks the detailed information of the suggested items displayed on the device, selects the item that best suits them, and then purchases it. Specifically, the user clicks on a link to access the online shopping site and completes the purchase procedure.

[0610] Input: Detailed information (ingredients, effects, reviews, price range)

[0611] Output: Purchase order for selected skin care products

[0612] (Application example 2)

[0613] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] In recent years, with the diversification of skin care products, users have found it difficult to select the product that best suits their skin. Furthermore, there are few feedback systems based on the user's emotional state, making it a challenge to improve user satisfaction. There is a need to solve these issues and provide a system that recommends more effective and satisfying skin care products.

[0615] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an emotion analysis means for analyzing the user's emotional state, a feedback means for evaluating the user's emotional response using the emotion analysis means and adjusting the next suggestion, and an information display means for displaying detailed information about the alternative items. This enables the user to select optimal skin care products based on their skin condition and emotional state.

[0616] "Image input means" refers to a device or function for acquiring image data captured by a user.

[0617] The "receiving means" refers to a device or function for receiving image data transmitted from the image input means.

[0618] "Image analysis means" refers to a device or function for extracting product ingredient information from received image data.

[0619] The "ingredient collating means" refers to a device or function for collating extracted ingredient information with a database.

[0620] The "component evaluation means" refers to a device or function for evaluating the balance and bias of components based on component information.

[0621] The "substitute product suggestion means" refers to a device or function for searching for and suggesting substitute items based on the component balance and bias evaluated by the component evaluation means.

[0622] "Information display means" refers to a device or function for displaying detailed information about a substitute item.

[0623] "Emotion analysis means" refers to a device or function for analyzing the user's emotional state.

[0624] "Feedback means" refers to a device or function that evaluates the user's emotional response using emotion analysis means and adjusts the content of the next suggestion.

[0625] In order to implement the present invention, it is necessary to construct the following system.

[0626] System Overview

[0627] The system includes an image input unit, a receiving unit, an image analysis unit, an ingredient matching unit, an ingredient evaluation unit, an alternative product suggestion unit, an information display unit, an emotion analysis unit, and a feedback unit. By linking these units, the system can suggest optimal skin care products to users.

[0628] Hardware and Software

[0629] Hardware:

[0630] Smartphone: Used as an image input and user interface

[0631] Servers: Data processing and storage

[0632] Camera: Built-in camera on smartphone

[0633] Microphone: A device for voice input.

[0634] software:

[0635] Image Analysis: OpenCV and Google Tesseract OCR

[0636] Server communication: requests module (Python)

[0637] Emotion analysis: emotion_recognition library (Python)

[0638] Programming: Python language used

[0639] Processing steps

[0640] 1. Image Acquisition:

[0641] Users take a photo of a skin care product label using their smartphone camera, and the image data is temporarily stored in the smartphone's memory and then sent to the server.

[0642] 2. Image Analysis:

[0643] The server analyzes the received image data and extracts component information from the image using Google Tesseract OCR.

[0644] 3. Ingredient Matching:

[0645] The extracted ingredient information is compared with a database on the server, and detailed information about the ingredients, their effects, recommended usage, and other information is retrieved from the database.

[0646] 4. Ingredients evaluation:

[0647] Based on the acquired ingredient information, the server evaluates the balance and imbalance of ingredients, particularly determining whether moisturizing ingredients and nutritional supplement ingredients are appropriate.

[0648] 5. Substitute suggestions:

[0649] Based on the results of the ingredient evaluation, the system proposes alternative skin care products that are best suited to the user's skin condition. The system searches for optimal products from a database in the server via an alternative product suggestion means and proposes them to the user.

[0650] 6. Information display:

[0651] Users can view detailed information about suggested alternative products via their smartphone, including ingredients, effects, other users' reviews, and prices.

[0652] 7. Emotion analysis:

[0653] The emotion analysis means analyzes the user's voice and facial expression data to evaluate their emotional reaction to the proposed product. If a positive reaction is obtained, it is fed back to the server and used to improve the next proposal.

[0654] 8. Feedback and Suggested Adjustments:

[0655] The next recommendation will be optimized based on the feedback information obtained through sentiment analysis. For example, if there is a negative reaction, the next recommendation will be a product with different ingredients or brand.

[0656] Specific examples

[0657] Consider a scenario where a user takes a photo of the label of a "lotion containing hyaluronic acid" with their smartphone camera in the skincare section of a physical store. The program captures the image and uses Tesseract OCR to extract ingredient information such as "hyaluronic acid, glycerin, aloe vera." The data sent to the server is analyzed, and the result is returned indicating that the moisturizing ingredients are highly rated.

[0658] Furthermore, based on the user's feedback, the company analyzed their sentiment and found that they responded positively to the question, "Is this lotion effective for dry skin?", so it decided to continue recommending skin care products with high moisturizing power from next time onwards. These results are displayed on the AI ​​assistant's display in the physical store.

[0659] Prompt Sentence Examples

[0660] "How can we develop an application that analyzes the ingredients of skin care products taken by customers in physical stores and suggests the best alternatives based on their skin condition and emotional state?"

[0661] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0662] Program processing steps

[0663] Step 1:

[0664] A user takes a photo of a skin care product label using a smartphone at the skin care section of a physical store. The image is captured using the smartphone's camera function, and the captured image data is saved on the smartphone. The input is the image data of the label, and the output is the saved image file.

[0665] Step 2:

[0666] The device sends the captured image data to a server. Specifically, the image file on the smartphone is compressed and uploaded to the server via the Internet. The input is the image file, and the output is the image data sent to the server.

[0667] Step 3:

[0668] The server analyzes the received image data. It uses Google Tesseract OCR to extract component information from the image in text format. The input is image data, and the output is component information in text format.

[0669] Step 4:

[0670] The server compares the extracted ingredient information with an internal database. Using the ingredient comparison means, detailed information, effects, and recommended usage of the relevant ingredients are obtained from the ingredient database. The input is ingredient information in text format, and the output is detailed information about the ingredients.

[0671] Step 5:

[0672] The server uses the ingredient evaluation means to evaluate the balance and imbalance of ingredients based on ingredient information. In particular, it checks the balance of moisturizing ingredients and nutritional supplement ingredients. The input is detailed information about the ingredients, and the output is the evaluation result of the ingredient balance. Specifically, if the moisturizing ingredients are below the standard value, the evaluation result will show "insufficient moisturizing ingredients."

[0673] Step 6:

[0674] The server uses the substitute suggestion means based on the ingredient evaluation results to search for and suggest alternative skin care products that are best suited to the user's skin condition. Products that meet the criteria are searched for from the skin care product database within the server. The input is the ingredient evaluation results, and the output is a list of suggested alternative skin care products.

[0675] Step 7:

[0676] The server presents detailed information about the suggested alternative items to the user through an information display means. The smartphone display shows the product's ingredients, effects, reviews from other users, price, etc. The input is a list of alternative skin care products, and the output is detailed information displayed on the smartphone screen.

[0677] Step 8:

[0678] The user's voice and facial expressions are collected through a camera and microphone and sent from the device to a server. The input is the user's voice and facial expression data, and the output is the emotional data sent to the server.

[0679] Step 9:

[0680] The server analyzes the user's emotional state using emotion analysis. Specifically, it uses the emotion_recognition library to identify emotions from voice and facial expression data and determine whether they are positive or negative. The input is the user's voice and facial expression data, and the output is the result of the emotion analysis.

[0681] Step 10:

[0682] The server uses a feedback mechanism to adjust the next recommendation based on the sentiment analysis results. For example, if a negative reaction is shown, it may suggest a product with different ingredients or brand the next time. The input is the sentiment analysis results, and the output is the next recommendation.

[0683] The above processing steps allow the user to select the most suitable skin care product based on their skin condition and emotional state.

[0684] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0685] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0686] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0687] [Third embodiment]

[0688] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0689] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0690] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0691] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0692] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0693] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0694] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0695] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0696] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0697] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0698] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0699] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0700] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[0701] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0702] Based on this detailed ingredient information, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below the standard value, it will evaluate that the user's skin is dry. Next, the server searches the database for alternative items with high moisturizing effects and suggests the most suitable item using an alternative suggestion means. Specifically, it may suggest "Brand B's highly moisturizing lotion."

[0703] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that best suits them.

[0704] This system also supports the use of multiple skincare products, and can comprehensively recommend items to supplement missing or excess ingredients while taking into account the overall balance of ingredients. This allows users to choose the skincare products that are best suited to their skin condition, helping to maintain good skin condition.

[0705] The processing flow will be explained below.

[0706] Step 1:

[0707] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[0708] Specifically, take care to photograph skin care product labels and packaging clearly.

[0709] Step 2:

[0710] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[0711] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[0712] Step 3:

[0713] The terminal transmits the image data to the server.

[0714] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[0715] Step 4:

[0716] The server receives the image data.

[0717] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[0718] Step 5:

[0719] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[0720] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[0721] Step 6:

[0722] The server uses the component matching means to match the extracted component information with an internal database.

[0723] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[0724] Step 7:

[0725] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[0726] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[0727] Step 8:

[0728] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[0729] Specifically, the database will be used to create a list of products that supplement missing ingredients and products that help to achieve overall balance.

[0730] Step 9:

[0731] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[0732] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[0733] Step 10:

[0734] The terminal displays detailed information of the substitute item received from the server to the user.

[0735] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[0736] Step 11:

[0737] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[0738] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[0739] This detailed processing step provides users with specific and intuitive assistance in choosing the best skin care products for their skin condition.

[0740] Example 1

[0741] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0742] Conventional skin care product recommendation systems have difficulty automatically selecting the optimal skin care product for a user's skin condition. Furthermore, systems that accurately grasp the product's ingredient information and then recommend optimal alternative products based on that information are insufficient. As a result, users end up purchasing products at their own discretion, which can prevent them from maximizing their effectiveness.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0744] In this invention, the server includes an image input means for a user to take an image of a skin care product, a means for receiving image data acquired by the image input means, a means for compressing and transmitting the received image data, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, and an information display means for displaying detailed information about the alternative items. This allows users to easily select and purchase skin care products that are best suited to their skin condition.

[0745] "Image input means" refers to a function or device for taking an image of a skin care product, and includes the camera of a mobile terminal used by the user.

[0746] The "receiving means" is a function or device for taking the image data acquired by the image input means into the server.

[0747] The "means for compressing and transmitting" is a function or device for reducing the data size of acquired image data before transmitting it.

[0748] "Image analysis means" means a function or device for extracting product ingredient information from received image data, including using optical character recognition technology.

[0749] The "ingredient collating means" is a function or device for comparing extracted ingredient information with a database and collating it.

[0750] The "component evaluation means" is a function or device for evaluating the balance and bias of components based on component information.

[0751] The "substitute product suggestion means" is a function or device for searching for and suggesting substitute items based on the evaluated ingredient balance and bias.

[0752] The "information display means" is a function or device for visually displaying detailed information about substitute items to the user.

[0753] The skin care product recommendation system of the present invention is designed to enable users to easily select the skin care product that is best suited to their skin condition. To realize this system, the following hardware and software are used.

[0754] The user takes a photo of the label of the skin care product they are using using the camera function of their smartphone. The smartphone then uses a camera app to capture the image data, and the entire system operates based on this image data.

[0755] The device receives the captured image data and compresses it in a format such as JPEG. The compressed image data is then sent to a server via the HTTPS protocol. This communication uses a network device (e.g., a Wi-Fi router).

[0756] The server uses the Google Cloud Vision API to analyze the received image data. This analysis extracts the ingredient information on the label from the image in text format using optical character recognition (OCR). For example, ingredients such as "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a lotion.

[0757] The server then compares the extracted ingredient information with an internal database (e.g., MySQL). Detailed information about each ingredient's effects and recommended dosages are obtained. An internal algorithm is used to evaluate the balance and bias of the ingredients. For example, if the moisturizing ingredients are below the standard value, the server evaluates the user's skin as dry.

[0758] The server then uses a recommendation engine to search for suitable alternative skin care products from the database. Based on the results of the ingredient evaluation, it suggests the best alternative. For example, it may suggest a "highly moisturizing lotion from a specific brand" that has a high moisturizing effect.

[0759] The results of the suggestions are sent from the server to the device and visually displayed to the user through a dedicated application or web browser. Detailed information about the suggested alternative skin care products (ingredients, effects, user reviews, price range, etc.) is displayed. The user can confirm this information and tap the purchase button to be redirected to an online shopping site and purchase the product.

[0760] As a concrete example, a user can take a photo of their lotion label with their smartphone camera and send the image to a server. The server can then use optical character recognition (OCR) technology to extract the ingredient information, compare it with a database, and determine if the moisturizing ingredients are lacking. Based on this, the server can suggest alternative skin care products with higher moisturizing effects and display the information on the device.

[0761] Below is an example of a prompt sentence to input to the generative AI model.

[0762] (Example) Prompt statement:

[0763] The user takes a photo of their own lotion with their smartphone camera and sends the image to the server. The server uses optical character recognition (OCR) technology to extract ingredient information, compares it with a database, and evaluates the balance of ingredients. As a result, it recommends a specific brand of highly moisturizing lotion with a high moisturizing effect. The user can then check this information on their smartphone and select and purchase the most suitable skin care product.

[0764] In this way, the system of the present invention has the function of suggesting optimal skin care products to the user through cooperation between the user, the terminal, and the server.

[0765] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0766] Step 1:

[0767] The user takes a picture of a skin care product. Using the smartphone camera app, the user takes a picture so that the label of the skin care product is clearly visible. For example, the user places a lotion bottle on a flat surface and holds the camera horizontally. In this step, the input is the actual skin care product and the smartphone camera, and the output is a JPEG image file.

[0768] Step 2:

[0769] The device compresses the image data and sends it to the server. The device compresses the captured image data into JPEG format and uploads it to the server using the HTTPS protocol. This is done using a network device (e.g., a Wi-Fi router). The input is a JPEG image file, and the compressed image data is sent to the server as the output.

[0770] Step 3:

[0771] The server analyzes the image data. The server calls the Google Cloud Vision API and extracts the ingredient information on the label from the received image data using optical character recognition (OCR) technology. The input is compressed image data, and the output is text data of the extracted ingredient information. Specifically, ingredients such as "hyaluronic acid, glycerin, aloe vera" listed on the label are extracted as text.

[0772] Step 4:

[0773] The server compares the ingredient information with an internal database. The server then compares the extracted ingredient information with a MySQL database to obtain the detailed effects and recommended usage amounts of each ingredient. The extracted ingredient information is text data as input, and the detailed effects of the ingredients and recommended usage amounts are obtained as output. Specifically, it executes an SQL query to obtain the moisturizing effect and recommended concentration of "hyaluronic acid."

[0774] Step 5:

[0775] The server evaluates the balance of ingredients. Using an internal algorithm, the server evaluates the balance and imbalance of ingredients based on the acquired ingredient information. The input is detailed ingredient effects and recommended usage amounts, and the output is the evaluation result of the ingredient balance. Specifically, it detects that moisturizing ingredients are below the standard value and evaluates that the user's skin is dry.

[0776] Step 6:

[0777] The server proposes alternative products. Based on the results of the ingredient evaluation, the server searches its internal database for the optimal alternative skin care product. The input is the evaluation result of the ingredient balance, and the output is information on the proposed alternative skin care product. Specifically, it proposes a "highly moisturizing lotion from a specific brand" with a high moisturizing effect.

[0778] Step 7:

[0779] The device displays the recommendation results. The device receives detailed information about the recommendation results (ingredients, effects, user reviews, price range, etc.) from the server and displays it in a dedicated application or web browser. The input is information about the suggested alternative skin care products, and the output is visually displayed to the user. Specifically, the information is displayed on the smartphone screen and the user views it.

[0780] Step 8:

[0781] The user checks the suggested results and makes a selection / purchase. The user operates the UI of the dedicated app to check detailed information about the suggested alternative skincare products. The information displayed on the smartphone screen is the input, and the decision to select / purchase is the output. The specific operation is that the user taps the purchase button, is redirected to an online shopping site, and purchases the product.

[0782] (Application example 1)

[0783] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0784] The present invention relates to a system that quickly and accurately analyzes the ingredient information of skin care products brought in by users and suggests optimal alternative products based on the user's skin condition and skin care goals. In particular, the objective is to realize this in a way that users can easily use in physical stores, and to streamline users' skin care product selection by providing visual information via smartphones or in-store terminals.

[0785] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0786] In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, a substitute suggestion means for searching for and suggesting substitute items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an information display means for displaying detailed information of the substitute items, and a means for the information display means to visually display information on a physical store terminal, thereby enabling users to visually check skin care products in the physical store and select the optimal substitute product.

[0787] The "image input means" is a device for acquiring an image of a product brought in by a user.

[0788] The "receiving means" is a device that receives the image data acquired by the image input means.

[0789] "Image analysis means" is a device that extracts product ingredient information from received image data.

[0790] The "component collating means" is a device that collates extracted component information with a database.

[0791] The "component evaluation means" is a device that evaluates the balance and bias of components based on component information.

[0792] The "substitute product suggestion means" is a device that searches for and suggests substitute items based on the component balance and bias evaluated by the component evaluation means.

[0793] The "information display means" is a device that visually displays detailed information about a substitute item.

[0794] A "physical store terminal" is a device installed in a physical store that has the function of acquiring images of products brought in by users and analyzing their ingredient information.

[0795] "Optical character recognition technology" is a technology that extracts text information from images.

[0796] An embodiment of the present invention is described below.

[0797] 1. Image input means

[0798] Users take pictures of their skin care products using their smartphone camera or a camera installed in a brick-and-mortar terminal, and the image capture method captures detailed product label information.

[0799] 2. Receiving Method

[0800] The smartphone or the physical store terminal compresses the acquired image data and sends it over the network to the server, which receives the image data.

[0801] 3. Image analysis methods

[0802] The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. Specifically, for example, it extracts ingredients such as "hyaluronic acid, glycerin, and aloe vera" from the label of a lotion.

[0803] 4. Component Matching Method

[0804] The server compares the extracted ingredient information with an internal database containing information on various skin care ingredients to obtain detailed information on each ingredient's effects and recommended dosage.

[0805] 5. Ingredient evaluation methods

[0806] The server evaluates the balance and bias of the ingredients based on the ingredient information. For example, if the moisturizing ingredient is below a standard value, it evaluates the user's skin as dry.

[0807] 6. Means of suggesting alternative products

[0808] Based on the results of the ingredient evaluation, the server searches the database for the most suitable alternative skin care item and suggests it. For example, it may suggest "Brand B's highly moisturizing lotion."

[0809] 7. Information display means

[0810] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range) is sent from the server to a smartphone or a brick-and-mortar store terminal and visually displayed to the user.

[0811] Specific examples

[0812] The system scans the label of a "lotion" brought in by the user with a smartphone camera and extracts "hyaluronic acid, glycerin, aloe vera" from the ingredient information. This information is sent to a server, which then compares it with a database and suggests the most suitable alternative product. For example, based on the results of the ingredient evaluation, "Brand B's highly moisturizing lotion" may be suggested.

[0813] Prompt Sentence Examples

[0814] "Evaluate products containing ingredients such as hyaluronic acid, glycerin, and aloe vera, and suggest alternative products with higher moisturizing benefits."

[0815] This system allows users to efficiently select skin care products in physical stores, enabling them to choose the products that best suit their skin condition.

[0816] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0817] Step 1:

[0818] A user takes an image of their own skin care product using a smartphone camera or a camera on a physical store terminal. The input is image data of the skin care product, and the output is the captured image data. The user launches a camera app to capture an image and saves the image data on the terminal.

[0819] Step 2:

[0820] The terminal compresses the acquired image data and sends it to the server via the network. The input is the captured image data, and the output is the transmission of compressed image data. Specifically, the terminal converts the image data into JPEG format, compresses it, and then uploads it to the server.

[0821] Step 3:

[0822] The server receives image data sent from the terminal. The input is compressed image data, and the output is decompressed image data. The server receives the image data using a network protocol, decompresses it, and stores it in its internal memory.

[0823] Step 4:

[0824] The server analyzes the received image data and extracts the component information in text format using optical character recognition (OCR) technology. The input is the decompressed image data, and the output is the extracted text information. The server uses the OCR engine to analyze the characters in the image and extract it as text information.

[0825] Step 5:

[0826] The server compares the extracted ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage. The input is the extracted text information, and the output is detailed information about the corresponding ingredients. Specifically, the server searches for ingredient information using an SQL query and obtains detailed ingredient information from the database.

[0827] Step 6:

[0828] The server evaluates the balance and bias of ingredients based on the ingredient information. The input is detailed information about the ingredients, and the output is the ingredient balance evaluation result. The server compares the results with the standard values ​​for each ingredient and executes an algorithm to evaluate the balance.

[0829] Step 7:

[0830] The server searches the database for optimal alternative skin care items based on the evaluation results and proposes them. The input is the ingredient balance evaluation results, and the output is a list of optimal alternative products. The server selects the optimal products using an evaluation algorithm and a matching engine.

[0831] Step 8:

[0832] The server sends detailed information about the suggested alternative items to a smartphone or a terminal in a physical store and displays it visually to the user. The input is a list of optimal alternative products, and the output is the information to be displayed. Specifically, the server generates detailed information in HTML format or similar and sends the data to the terminal. The terminal then displays the received HTML data in a browser or similar.

[0833] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0834] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[0835] The present invention also incorporates an emotion engine that recognizes the user's emotional state. By analyzing the user's voice and facial expression data, the emotion engine can recognize the user's emotional state and evaluate their emotional response to the suggested items. This allows the system to understand which suggestions the user is satisfied or dissatisfied with, and reflects this feedback in the next suggestions.

[0836] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0837] Based on this detailed ingredient information, the server uses an ingredient evaluation tool to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Next, the server searches the database for alternative items with high moisturizing effects and uses an alternative suggestion tool to suggest the most suitable item. Specifically, it will suggest a "lotion with high moisturizing power" or suggest the addition of a vitamin C serum, taking into account the overall balance.

[0838] In addition to this suggestion process, the emotion engine senses the user's reactions. If the user responds positively to a suggested product through voice or facial expression, the emotion engine recognizes this and ensures the suggestion is appropriate. Conversely, if a negative reaction is observed, the emotion engine captures that feedback and adjusts the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will suggest a moisturizing lotion with different ingredients or brand.

[0839] Detailed information about the suggested alternative items (ingredients, effects, reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that is best suited to them.

[0840] The emotion engine analyzes the user's emotional state in real time and dynamically adjusts the suggestions based on the user's reaction, thereby increasing user satisfaction and providing the most suitable skin care products. This system allows users to always choose the skin care products that best suit their skin condition and emotional state, thereby maintaining good skin condition.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[0844] Specifically, take care to photograph skin care product labels and packaging clearly.

[0845] Step 2:

[0846] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[0847] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[0848] Step 3:

[0849] The terminal transmits the image data to the server.

[0850] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[0851] Step 4:

[0852] The server receives the image data.

[0853] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[0854] Step 5:

[0855] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[0856] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[0857] Step 6:

[0858] The server uses the component matching means to match the extracted component information with an internal database.

[0859] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[0860] Step 7:

[0861] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[0862] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[0863] Step 8:

[0864] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[0865] Specifically, the system will create a list of products from the database that will supplement the missing ingredients or balance the skin overall. For example, if the skin lacks moisturizing ingredients, the system will search for a moisturizing lotion.

[0866] Step 9:

[0867] The server uses an emotion engine to recognize the user's emotional state when making alternative suggestions.

[0868] Specifically, the system analyzes the user's voice data and facial expression data to detect positive or negative reactions.

[0869] Step 10:

[0870] The server reflects the emotion recognition results from the emotion engine in proposing alternative products.

[0871] Specifically, the system prioritizes ingredients and products to which the user previously had a positive reaction, and avoids ingredients and products to which the user previously had a negative reaction.

[0872] Step 11:

[0873] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[0874] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[0875] Step 12:

[0876] The terminal displays detailed information of the substitute item received from the server to the user.

[0877] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[0878] Step 13:

[0879] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[0880] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[0881] This detailed processing step provides users with specific and intuitive assistance in choosing the skin care products that best suit their skin and emotional state.

[0882] Example 2

[0883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0884] Current skincare product recommendation systems lack the ability to easily obtain ingredient information about products currently in use and then suggest appropriate alternatives based on that information. Furthermore, they do not optimize recommendations based on the user's emotional responses, leaving insufficient means to increase user satisfaction. Furthermore, there is no mechanism for incorporating the user's emotional feedback on a suggested product into future recommendations. To address these issues, an advanced recommendation system is needed that incorporates not only image analysis and ingredient evaluation, but also the user's emotional responses.

[0885] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image analysis means for extracting product ingredient information from image data, an ingredient comparison means for comparing the ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the extracted ingredient information, an alternative product suggestion means for searching for and suggesting alternative products, an emotion analysis means for analyzing the user's emotional response to the suggested alternative items, and an information display means for displaying detailed information about the alternative items. This makes it possible to efficiently extract and evaluate ingredient information for skin care products currently used by the user and suggest appropriate alternative products based on the user's skin condition. Furthermore, by analyzing the user's emotional response to the suggested products and reflecting this in the next suggestion, user satisfaction can be increased.

[0886] The "image input means" is a device that acquires image data of the product being used by the user, and in many cases refers to the camera of a mobile terminal.

[0887] The "means for receiving image data" refers to a device or function for receiving image data acquired by the image input means.

[0888] "Image analysis means" refers to technology or equipment for extracting product ingredient information from received image data, and specifically, optical character recognition technology is often used.

[0889] "Ingredient matching means" refers to a technology or device for matching ingredient information extracted by the image analysis means with a database and obtaining information such as detailed effects and recommended usage amounts.

[0890] "Ingredient evaluation means" refers to the technology and equipment used to evaluate the balance and bias of ingredients based on extracted and collated ingredient information.

[0891] The "substitute product suggestion means" refers to a technology or device for searching for and suggesting the most suitable substitute product based on the results of evaluation by the component evaluation means.

[0892] "Emotion analysis means" refers to technology or devices for analyzing a user's emotional response to a proposed substitute item, and includes functions for analyzing voice and facial expression data.

[0893] "Information display means" refers to a device or technology for visually displaying detailed information about the proposed substitute item to the user.

[0894] "Ingredient information" refers to detailed data such as the names, effects, and recommended usage amounts of various ingredients contained in skin care products.

[0895] "Database" refers to a system that stores information used to collate and evaluate ingredient information.

[0896] The skin care product recommendation system of this invention starts when a user takes a picture of a skin care product they own with a smartphone camera and sends the image to a server. The terminal receives the image data and uploads it to the server via a network. At this stage, the terminal is a mobile terminal such as a smartphone.

[0897] The server uses optical character recognition (OCR) technology to analyze the received image data. For example, the server extracts the ingredients "hyaluronic acid, glycerin, aloe vera" from the label of a lotion that has been photographed. The extracted ingredient information is compared with the server's internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[0898] Next, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Based on this evaluation result, the server searches the database for suitable alternative items and suggests them to the user using an alternative suggestion means. Specifically, it will suggest items such as "highly moisturizing lotion" and "vitamin C serum."

[0899] Furthermore, this system incorporates an emotion engine. The emotion engine on the server analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user responds "I like it" to a suggested product through voice or facial expression, the emotion engine recognizes this as a positive reaction and confirms that the suggestion was appropriate. Conversely, if a negative reaction is observed, that feedback is reflected in the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will be a moisturizing lotion with different ingredients or brand.

[0900] Detailed information on the suggested alternative items is sent from the server to the terminal, and the user can visually check it through the terminal's information display means. The information includes ingredients, effects, reviews, price range, etc. Based on this information, the user can select and purchase the most suitable item.

[0901] Specific examples

[0902] For example, if a user is using a lotion containing "hyaluronic acid, glycerin, and aloe vera," the server compares the ingredient information with the database to obtain detailed effects and recommended usage amounts. The server then evaluates that the moisturizing effect is below the standard value and suggests a "lotion with high moisturizing power." If the user responds to this suggestion by saying "like," the emotion engine recognizes this and determines that the suggestion was appropriate.

[0903] Prompt Sentence Examples

[0904] "Please tell me the ingredients of the lotion you are currently using."

[0905] "Please suggest skin care products with high moisturizing effects."

[0906] Please tell me an alternative to this lotion.

[0907] In this way, by combining advanced image analysis technology with emotion analysis functions, this system can increase user satisfaction and provide optimal skin care products.

[0908] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0909] Step 1:

[0910] The user uses the camera function of their smartphone to take a photo of their skin care product. Specifically, the user opens the camera app and takes a photo of the label of the skin care product.

[0911] Input: Skincare product image

[0912] Output: Captured image data

[0913] Step 2:

[0914] The device compresses the captured image data and uploads it to a server via the network, for example by converting it to JPEG format to reduce the file size.

[0915] Input: Photographed image data

[0916] Output: Compressed image data

[0917] Step 3:

[0918] The server uses optical character recognition (OCR) technology to analyze the received image data. The server inputs the image data into an OCR engine and extracts the component information as text.

[0919] Input: Compressed image data

[0920] Output: Extracted ingredient information (text format)

[0921] Step 4:

[0922] The server compares the extracted ingredient information with its internal database, specifically by executing a database query using the ingredient name as a key to obtain detailed information about each ingredient's effects and recommended dosage.

[0923] Input: Extracted ingredient information

[0924] Output: Detailed ingredient information (effects, recommended dosage, etc.)

[0925] Step 5:

[0926] The server uses the component evaluation means to evaluate the balance and bias of the components. For example, if the amount of a specific moisturizing component is determined to be below a reference value, the server determines the user's skin condition as dry.

[0927] Input: Detailed ingredient information

[0928] Output: Evaluation results (ingredients balance, skin condition)

[0929] Step 6:

[0930] The server searches for and suggests alternative products from a database based on the evaluation results.The server uses the alternative product suggestion means to list and suggest suitable skin care products (e.g., moisturizing lotion).

[0931] Input: Evaluation result

[0932] Output: A list of suggested replacements

[0933] Step 7:

[0934] The server's emotion engine analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, it uses voice and facial expression data obtained from the user's remote camera or microphone as input and applies an emotion analysis algorithm to determine whether the reaction is positive or negative.

[0935] Input: Voice data, facial expression data

[0936] Output: Emotional response analysis results

[0937] Step 8:

[0938] The server sends detailed information about the proposed substitute items to the terminal and displays it to the user through the terminal's information display means. The user can then check this information and select the most suitable item.

[0939] Input: List of suggested replacements, emotional response analysis results

[0940] Output: Detailed information displayed on the terminal

[0941] Step 9:

[0942] The user checks the detailed information of the suggested items displayed on the device, selects the item that best suits them, and then purchases it. Specifically, the user clicks on a link to access the online shopping site and completes the purchase procedure.

[0943] Input: Detailed information (ingredients, effects, reviews, price range)

[0944] Output: Purchase order for selected skin care products

[0945] (Application example 2)

[0946] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0947] In recent years, with the diversification of skin care products, users have found it difficult to select the product that best suits their skin. Furthermore, there are few feedback systems based on the user's emotional state, making it a challenge to improve user satisfaction. There is a need to solve these issues and provide a system that recommends more effective and satisfying skin care products.

[0948] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an emotion analysis means for analyzing the user's emotional state, a feedback means for evaluating the user's emotional response using the emotion analysis means and adjusting the next suggestion, and an information display means for displaying detailed information about the alternative items. This enables the user to select optimal skin care products based on their skin condition and emotional state.

[0949] "Image input means" refers to a device or function for acquiring image data captured by a user.

[0950] The "receiving means" refers to a device or function for receiving image data transmitted from the image input means.

[0951] "Image analysis means" refers to a device or function for extracting product ingredient information from received image data.

[0952] The "ingredient collating means" refers to a device or function for collating extracted ingredient information with a database.

[0953] The "component evaluation means" refers to a device or function for evaluating the balance and bias of components based on component information.

[0954] The "substitute product suggestion means" refers to a device or function for searching for and suggesting substitute items based on the component balance and bias evaluated by the component evaluation means.

[0955] "Information display means" refers to a device or function for displaying detailed information about a substitute item.

[0956] "Emotion analysis means" refers to a device or function for analyzing the user's emotional state.

[0957] "Feedback means" refers to a device or function that evaluates the user's emotional response using emotion analysis means and adjusts the content of the next suggestion.

[0958] In order to implement the present invention, it is necessary to construct the following system.

[0959] System Overview

[0960] The system includes an image input unit, a receiving unit, an image analysis unit, an ingredient matching unit, an ingredient evaluation unit, an alternative product suggestion unit, an information display unit, an emotion analysis unit, and a feedback unit. By linking these units, the system can suggest optimal skin care products to users.

[0961] Hardware and Software

[0962] Hardware:

[0963] Smartphone: Used as an image input and user interface

[0964] Servers: Data processing and storage

[0965] Camera: Built-in camera on smartphone

[0966] Microphone: A device for voice input.

[0967] software:

[0968] Image Analysis: OpenCV and Google Tesseract OCR

[0969] Server communication: requests module (Python)

[0970] Emotion analysis: emotion_recognition library (Python)

[0971] Programming: Python language used

[0972] Processing steps

[0973] 1. Image Acquisition:

[0974] Users take a photo of a skin care product label using their smartphone camera, and the image data is temporarily stored in the smartphone's memory and then sent to the server.

[0975] 2. Image Analysis:

[0976] The server analyzes the received image data and extracts component information from the image using Google Tesseract OCR.

[0977] 3. Ingredient Matching:

[0978] The extracted ingredient information is compared with a database on the server, and detailed information about the ingredients, their effects, recommended usage, and other information is retrieved from the database.

[0979] 4. Ingredients evaluation:

[0980] Based on the acquired ingredient information, the server evaluates the balance and imbalance of ingredients, particularly determining whether moisturizing ingredients and nutritional supplement ingredients are appropriate.

[0981] 5. Substitute suggestions:

[0982] Based on the results of the ingredient evaluation, the system proposes alternative skin care products that are best suited to the user's skin condition. The system searches for optimal products from a database in the server via an alternative product suggestion means and proposes them to the user.

[0983] 6. Information display:

[0984] Users can view detailed information about suggested alternative products via their smartphone, including ingredients, effects, other users' reviews, and prices.

[0985] 7. Emotion analysis:

[0986] The emotion analysis means analyzes the user's voice and facial expression data to evaluate their emotional reaction to the proposed product. If a positive reaction is obtained, it is fed back to the server and used to improve the next proposal.

[0987] 8. Feedback and Suggested Adjustments:

[0988] The next recommendation will be optimized based on the feedback information obtained through sentiment analysis. For example, if there is a negative reaction, the next recommendation will be a product with different ingredients or brand.

[0989] Specific examples

[0990] Consider a scenario where a user takes a photo of the label of a "lotion containing hyaluronic acid" with their smartphone camera in the skincare section of a physical store. The program captures the image and uses Tesseract OCR to extract ingredient information such as "hyaluronic acid, glycerin, aloe vera." The data sent to the server is analyzed, and the result is returned indicating that the moisturizing ingredients are highly rated.

[0991] Furthermore, based on the user's feedback, the company analyzed their sentiment and found that they responded positively to the question, "Is this lotion effective for dry skin?", so it decided to continue recommending skin care products with high moisturizing power from next time onwards. These results are displayed on the AI ​​assistant's display in the physical store.

[0992] Prompt Sentence Examples

[0993] "How can we develop an application that analyzes the ingredients of skin care products taken by customers in physical stores and suggests the best alternatives based on their skin condition and emotional state?"

[0994] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0995] Program processing steps

[0996] Step 1:

[0997] A user takes a photo of a skin care product label using a smartphone at the skin care section of a physical store. The image is captured using the smartphone's camera function, and the captured image data is saved on the smartphone. The input is the image data of the label, and the output is the saved image file.

[0998] Step 2:

[0999] The device sends the captured image data to a server. Specifically, the image file on the smartphone is compressed and uploaded to the server via the Internet. The input is the image file, and the output is the image data sent to the server.

[1000] Step 3:

[1001] The server analyzes the received image data. It uses Google Tesseract OCR to extract component information from the image in text format. The input is image data, and the output is component information in text format.

[1002] Step 4:

[1003] The server compares the extracted ingredient information with an internal database. Using the ingredient comparison means, detailed information, effects, and recommended usage of the relevant ingredients are obtained from the ingredient database. The input is ingredient information in text format, and the output is detailed information about the ingredients.

[1004] Step 5:

[1005] The server uses the ingredient evaluation means to evaluate the balance and imbalance of ingredients based on ingredient information. In particular, it checks the balance of moisturizing ingredients and nutritional supplement ingredients. The input is detailed information about the ingredients, and the output is the evaluation result of the ingredient balance. Specifically, if the moisturizing ingredients are below the standard value, the evaluation result will show "insufficient moisturizing ingredients."

[1006] Step 6:

[1007] The server uses the substitute suggestion means based on the ingredient evaluation results to search for and suggest alternative skin care products that are best suited to the user's skin condition. Products that meet the criteria are searched for from the skin care product database within the server. The input is the ingredient evaluation results, and the output is a list of suggested alternative skin care products.

[1008] Step 7:

[1009] The server presents detailed information about the suggested alternative items to the user through an information display means. The smartphone display shows the product's ingredients, effects, reviews from other users, price, etc. The input is a list of alternative skin care products, and the output is detailed information displayed on the smartphone screen.

[1010] Step 8:

[1011] The user's voice and facial expressions are collected through a camera and microphone and sent from the device to a server. The input is the user's voice and facial expression data, and the output is the emotional data sent to the server.

[1012] Step 9:

[1013] The server analyzes the user's emotional state using emotion analysis. Specifically, it uses the emotion_recognition library to identify emotions from voice and facial expression data and determine whether they are positive or negative. The input is the user's voice and facial expression data, and the output is the result of the emotion analysis.

[1014] Step 10:

[1015] The server uses a feedback mechanism to adjust the next recommendation based on the sentiment analysis results. For example, if a negative reaction is shown, it may suggest a product with different ingredients or brand the next time. The input is the sentiment analysis results, and the output is the next recommendation.

[1016] The above processing steps allow the user to select the most suitable skin care product based on their skin condition and emotional state.

[1017] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1018] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1019] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1020] [Fourth embodiment]

[1021] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1022] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1024] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1025] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1026] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1028] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1029] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1030] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1032] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1034] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[1035] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[1036] Based on this detailed ingredient information, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below the standard value, it will evaluate that the user's skin is dry. Next, the server searches the database for alternative items with high moisturizing effects and suggests the most suitable item using an alternative suggestion means. Specifically, it may suggest "Brand B's highly moisturizing lotion."

[1037] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that best suits them.

[1038] This system also supports the use of multiple skincare products, and can comprehensively recommend items to supplement missing or excess ingredients while taking into account the overall balance of ingredients. This allows users to choose the skincare products that are best suited to their skin condition, helping to maintain good skin condition.

[1039] The processing flow will be explained below.

[1040] Step 1:

[1041] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[1042] Specifically, take care to photograph skin care product labels and packaging clearly.

[1043] Step 2:

[1044] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[1045] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[1046] Step 3:

[1047] The terminal transmits the image data to the server.

[1048] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[1049] Step 4:

[1050] The server receives the image data.

[1051] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[1052] Step 5:

[1053] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[1054] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[1055] Step 6:

[1056] The server uses the component matching means to match the extracted component information with an internal database.

[1057] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[1058] Step 7:

[1059] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[1060] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[1061] Step 8:

[1062] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[1063] Specifically, the database will be used to create a list of products that supplement missing ingredients and products that help to achieve overall balance.

[1064] Step 9:

[1065] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[1066] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[1067] Step 10:

[1068] The terminal displays detailed information of the substitute item received from the server to the user.

[1069] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[1070] Step 11:

[1071] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[1072] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[1073] This detailed processing step provides users with specific and intuitive assistance in choosing the best skin care products for their skin condition.

[1074] Example 1

[1075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1076] Conventional skin care product recommendation systems have difficulty automatically selecting the optimal skin care product for a user's skin condition. Furthermore, systems that accurately grasp the product's ingredient information and then recommend optimal alternative products based on that information are insufficient. As a result, users end up purchasing products at their own discretion, which can prevent them from maximizing their effectiveness.

[1077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1078] In this invention, the server includes an image input means for a user to take an image of a skin care product, a means for receiving image data acquired by the image input means, a means for compressing and transmitting the received image data, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, and an information display means for displaying detailed information about the alternative items. This allows users to easily select and purchase skin care products that are best suited to their skin condition.

[1079] "Image input means" refers to a function or device for taking an image of a skin care product, and includes the camera of a mobile terminal used by the user.

[1080] The "receiving means" is a function or device for taking the image data acquired by the image input means into the server.

[1081] The "means for compressing and transmitting" is a function or device for reducing the data size of acquired image data before transmitting it.

[1082] "Image analysis means" means a function or device for extracting product ingredient information from received image data, including using optical character recognition technology.

[1083] The "ingredient collating means" is a function or device for comparing extracted ingredient information with a database and collating it.

[1084] The "component evaluation means" is a function or device for evaluating the balance and bias of components based on component information.

[1085] The "substitute product suggestion means" is a function or device for searching for and suggesting substitute items based on the evaluated ingredient balance and bias.

[1086] The "information display means" is a function or device for visually displaying detailed information about substitute items to the user.

[1087] The skin care product recommendation system of the present invention is designed to enable users to easily select the skin care product that is best suited to their skin condition. To realize this system, the following hardware and software are used.

[1088] The user takes a photo of the label of the skin care product they are using using the camera function of their smartphone. The smartphone then uses a camera app to capture the image data, and the entire system operates based on this image data.

[1089] The device receives the captured image data and compresses it in a format such as JPEG. The compressed image data is then sent to a server via the HTTPS protocol. This communication uses a network device (e.g., a Wi-Fi router).

[1090] The server uses the Google Cloud Vision API to analyze the received image data. This analysis extracts the ingredient information on the label from the image in text format using optical character recognition (OCR). For example, ingredients such as "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a lotion.

[1091] The server then compares the extracted ingredient information with an internal database (e.g., MySQL). Detailed information about each ingredient's effects and recommended dosages are obtained. An internal algorithm is used to evaluate the balance and bias of the ingredients. For example, if the moisturizing ingredients are below the standard value, the server evaluates the user's skin as dry.

[1092] The server then uses a recommendation engine to search for suitable alternative skin care products from the database. Based on the results of the ingredient evaluation, it suggests the best alternative. For example, it may suggest a "highly moisturizing lotion from a specific brand" that has a high moisturizing effect.

[1093] The results of the suggestions are sent from the server to the device and visually displayed to the user through a dedicated application or web browser. Detailed information about the suggested alternative skin care products (ingredients, effects, user reviews, price range, etc.) is displayed. The user can confirm this information and tap the purchase button to be redirected to an online shopping site and purchase the product.

[1094] As a concrete example, a user can take a photo of their lotion label with their smartphone camera and send the image to a server. The server can then use optical character recognition (OCR) technology to extract the ingredient information, compare it with a database, and determine if the moisturizing ingredients are lacking. Based on this, the server can suggest alternative skin care products with higher moisturizing effects and display the information on the device.

[1095] Below is an example of a prompt sentence to input to the generative AI model.

[1096] (Example) Prompt statement:

[1097] The user takes a photo of their own lotion with their smartphone camera and sends the image to the server. The server uses optical character recognition (OCR) technology to extract ingredient information, compares it with a database, and evaluates the balance of ingredients. As a result, it recommends a specific brand of highly moisturizing lotion with a high moisturizing effect. The user can then check this information on their smartphone and select and purchase the most suitable skin care product.

[1098] In this way, the system of the present invention has the function of suggesting optimal skin care products to the user through cooperation between the user, the terminal, and the server.

[1099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1100] Step 1:

[1101] The user takes a picture of a skin care product. Using the smartphone camera app, the user takes a picture so that the label of the skin care product is clearly visible. For example, the user places a lotion bottle on a flat surface and holds the camera horizontally. In this step, the input is the actual skin care product and the smartphone camera, and the output is a JPEG image file.

[1102] Step 2:

[1103] The device compresses the image data and sends it to the server. The device compresses the captured image data into JPEG format and uploads it to the server using the HTTPS protocol. This is done using a network device (e.g., a Wi-Fi router). The input is a JPEG image file, and the compressed image data is sent to the server as the output.

[1104] Step 3:

[1105] The server analyzes the image data. The server calls the Google Cloud Vision API and extracts the ingredient information on the label from the received image data using optical character recognition (OCR) technology. The input is compressed image data, and the output is text data of the extracted ingredient information. Specifically, ingredients such as "hyaluronic acid, glycerin, aloe vera" listed on the label are extracted as text.

[1106] Step 4:

[1107] The server compares the ingredient information with an internal database. The server then compares the extracted ingredient information with a MySQL database to obtain the detailed effects and recommended usage amounts of each ingredient. The extracted ingredient information is text data as input, and the detailed effects of the ingredients and recommended usage amounts are obtained as output. Specifically, it executes an SQL query to obtain the moisturizing effect and recommended concentration of "hyaluronic acid."

[1108] Step 5:

[1109] The server evaluates the balance of ingredients. Using an internal algorithm, the server evaluates the balance and imbalance of ingredients based on the acquired ingredient information. The input is detailed ingredient effects and recommended usage amounts, and the output is the evaluation result of the ingredient balance. Specifically, it detects that moisturizing ingredients are below the standard value and evaluates that the user's skin is dry.

[1110] Step 6:

[1111] The server proposes alternative products. Based on the results of the ingredient evaluation, the server searches its internal database for the optimal alternative skin care product. The input is the evaluation result of the ingredient balance, and the output is information on the proposed alternative skin care product. Specifically, it proposes a "highly moisturizing lotion from a specific brand" with a high moisturizing effect.

[1112] Step 7:

[1113] The device displays the recommendation results. The device receives detailed information about the recommendation results (ingredients, effects, user reviews, price range, etc.) from the server and displays it in a dedicated application or web browser. The input is information about the suggested alternative skin care products, and the output is visually displayed to the user. Specifically, the information is displayed on the smartphone screen and the user views it.

[1114] Step 8:

[1115] The user checks the suggested results and makes a selection / purchase. The user operates the UI of the dedicated app to check detailed information about the suggested alternative skincare products. The information displayed on the smartphone screen is the input, and the decision to select / purchase is the output. The specific operation is that the user taps the purchase button, is redirected to an online shopping site, and purchases the product.

[1116] (Application example 1)

[1117] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1118] The present invention relates to a system that quickly and accurately analyzes the ingredient information of skin care products brought in by users and suggests optimal alternative products based on the user's skin condition and skin care goals. In particular, the objective is to realize this in a way that users can easily use in physical stores, and to streamline users' skin care product selection by providing visual information via smartphones or in-store terminals.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1120] In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, a substitute suggestion means for searching for and suggesting substitute items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an information display means for displaying detailed information of the substitute items, and a means for the information display means to visually display information on a physical store terminal, thereby enabling a user to visually check skin care products in a physical store and select an optimal substitute product.

[1121] The "image input means" is a device for acquiring an image of a product brought in by a user.

[1122] The "receiving means" is a device that receives the image data acquired by the image input means.

[1123] "Image analysis means" is a device that extracts product ingredient information from received image data.

[1124] The "component collating means" is a device that collates extracted component information with a database.

[1125] The "component evaluation means" is a device that evaluates the balance and bias of components based on component information.

[1126] The "substitute product suggestion means" is a device that searches for and suggests substitute items based on the component balance and bias evaluated by the component evaluation means.

[1127] The "information display means" is a device that visually displays detailed information about a substitute item.

[1128] A "physical store terminal" is a device installed in a physical store that has the function of acquiring images of products brought in by users and analyzing their ingredient information.

[1129] "Optical character recognition technology" is a technology that extracts text information from images.

[1130] An embodiment of the present invention is described below.

[1131] 1. Image input means

[1132] Users take pictures of their skin care products using their smartphone camera or a camera installed in a brick-and-mortar terminal, and the image capture method captures detailed product label information.

[1133] 2. Receiving Method

[1134] The smartphone or the physical store terminal compresses the acquired image data and sends it over the network to the server, which receives the image data.

[1135] 3. Image analysis methods

[1136] The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. Specifically, for example, it extracts ingredients such as "hyaluronic acid, glycerin, and aloe vera" from the label of a lotion.

[1137] 4. Component Matching Method

[1138] The server compares the extracted ingredient information with an internal database containing information on various skin care ingredients to obtain detailed information on each ingredient's effects and recommended dosage.

[1139] 5. Ingredient evaluation methods

[1140] The server evaluates the balance and bias of the ingredients based on the ingredient information. For example, if the moisturizing ingredient is below a standard value, it evaluates the user's skin as dry.

[1141] 6. Means of suggesting alternative products

[1142] Based on the results of the ingredient evaluation, the server searches the database for the most suitable alternative skin care item and suggests it. For example, it may suggest "Brand B's highly moisturizing lotion."

[1143] 7. Information display means

[1144] Detailed information about the suggested alternative items (e.g., ingredients, effects, user reviews, price range) is sent from the server to a smartphone or a brick-and-mortar store terminal and visually displayed to the user.

[1145] Specific examples

[1146] The system scans the label of a "lotion" brought in by the user with a smartphone camera and extracts "hyaluronic acid, glycerin, aloe vera" from the ingredient information. This information is sent to a server, which then compares it with a database and suggests the most suitable alternative product. For example, based on the results of the ingredient evaluation, "Brand B's highly moisturizing lotion" may be suggested.

[1147] Prompt Sentence Examples

[1148] "Evaluate products containing ingredients such as hyaluronic acid, glycerin, and aloe vera, and suggest alternative products with higher moisturizing benefits."

[1149] This system allows users to efficiently select skin care products in physical stores, enabling them to choose the products that best suit their skin condition.

[1150] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1151] Step 1:

[1152] A user takes an image of their own skin care product using a smartphone camera or a camera on a physical store terminal. The input is image data of the skin care product, and the output is the captured image data. The user launches a camera app to capture an image and saves the image data on the terminal.

[1153] Step 2:

[1154] The terminal compresses the acquired image data and sends it to the server via the network. The input is the captured image data, and the output is the transmission of compressed image data. Specifically, the terminal converts the image data into JPEG format, compresses it, and then uploads it to the server.

[1155] Step 3:

[1156] The server receives image data sent from the terminal. The input is compressed image data, and the output is decompressed image data. The server receives the image data using a network protocol, decompresses it, and stores it in its internal memory.

[1157] Step 4:

[1158] The server analyzes the received image data and extracts the component information in text format using optical character recognition (OCR) technology. The input is the decompressed image data, and the output is the extracted text information. The server uses the OCR engine to analyze the characters in the image and extract it as text information.

[1159] Step 5:

[1160] The server compares the extracted ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage. The input is the extracted text information, and the output is detailed information about the corresponding ingredients. Specifically, the server searches for ingredient information using an SQL query and obtains detailed ingredient information from the database.

[1161] Step 6:

[1162] The server evaluates the balance and bias of ingredients based on the ingredient information. The input is detailed information about the ingredients, and the output is the ingredient balance evaluation result. The server compares the results with the standard values ​​for each ingredient and executes an algorithm to evaluate the balance.

[1163] Step 7:

[1164] The server searches the database for optimal alternative skin care items based on the evaluation results and proposes them. The input is the ingredient balance evaluation results, and the output is a list of optimal alternative products. The server selects the optimal products using an evaluation algorithm and a matching engine.

[1165] Step 8:

[1166] The server sends detailed information about the suggested alternative items to a smartphone or a terminal in a physical store and displays it visually to the user. The input is a list of optimal alternative products, and the output is the information to be displayed. Specifically, the server generates detailed information in HTML format or similar and sends the data to the terminal. The terminal then displays the received HTML data in a browser or similar.

[1167] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1168] The skin care product recommendation system of the present invention begins when a user takes a photo of a skin care product they own with a smartphone camera and sends the image to a server. The device receives this image data and sends it to the server. The server uses image analysis means to extract product ingredient information from the image data and compares this ingredient information with an internal database to obtain detailed ingredient data. The server then uses ingredient evaluation means to evaluate the balance and imbalance of ingredients, making an evaluation that takes into account the user's skin condition and skin care goals. The server then uses substitute recommendation means to search for and recommend optimal alternative skin care items based on the ingredient evaluation results. The results are displayed to the user via the device's information display means. The user can then check detailed information about the suggested alternative items and select and purchase the optimal skin care product.

[1169] The present invention also incorporates an emotion engine that recognizes the user's emotional state. By analyzing the user's voice and facial expression data, the emotion engine can recognize the user's emotional state and evaluate their emotional response to the suggested items. This allows the system to understand which suggestions the user is satisfied or dissatisfied with, and reflects this feedback in the next suggestions.

[1170] A user uses the camera function of their smartphone to take a picture of their skin care product (e.g., lotion). The device compresses this image data and uploads it to a server via a network. The server analyzes the received image data and extracts ingredient information in text format using optical character recognition (OCR) technology. For example, the ingredients "hyaluronic acid, glycerin, and aloe vera" can be extracted from the label of a photographed lotion. The server then compares the ingredient information with its internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[1171] Based on this detailed ingredient information, the server uses an ingredient evaluation tool to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Next, the server searches the database for alternative items with high moisturizing effects and uses an alternative suggestion tool to suggest the most suitable item. Specifically, it will suggest a "lotion with high moisturizing power" or suggest the addition of a vitamin C serum, taking into account the overall balance.

[1172] In addition to this suggestion process, the emotion engine senses the user's reactions. If the user responds positively to a suggested product through voice or facial expression, the emotion engine recognizes this and ensures the suggestion is appropriate. Conversely, if a negative reaction is observed, the emotion engine captures that feedback and adjusts the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will suggest a moisturizing lotion with different ingredients or brand.

[1173] Detailed information about the suggested alternative items (ingredients, effects, reviews, price range, etc.) is sent from the server to the terminal and visually displayed to the user by the information display means. The user can check this information and select and purchase the item that is best suited to them.

[1174] The emotion engine analyzes the user's emotional state in real time and dynamically adjusts the suggestions based on the user's reaction, thereby increasing user satisfaction and providing the most suitable skin care products. This system allows users to always choose the skin care products that best suit their skin condition and emotional state, thereby maintaining good skin condition.

[1175] The processing flow will be explained below.

[1176] Step 1:

[1177] A user uses the camera function of a smartphone to take an image of a skin care product in hand.

[1178] Specifically, take care to photograph skin care product labels and packaging clearly.

[1179] Step 2:

[1180] The terminal acquires the photographed image data and temporarily stores the image data in the terminal.

[1181] Specifically, when the user presses the shooting completion button, the image is saved on the terminal.

[1182] Step 3:

[1183] The terminal transmits the image data to the server.

[1184] Specifically, the image data stored in the terminal is compressed and uploaded to a server via a network.

[1185] Step 4:

[1186] The server receives the image data.

[1187] Specifically, the server receives image data at a specified URL or API endpoint and stores it for analysis.

[1188] Step 5:

[1189] The server uses image analysis means to extract ingredient information written on the product label or package from the received image data.

[1190] Specifically, optical character recognition (OCR) technology is used to obtain component information as text data.

[1191] Step 6:

[1192] The server uses the component matching means to match the extracted component information with an internal database.

[1193] Specifically, it compares the information with existing ingredient information in the database and obtains detailed information (effects, recommended usage, etc.) of matching ingredients.

[1194] Step 7:

[1195] The server uses the component evaluation means to evaluate the balance and bias of the components based on the acquired component information.

[1196] Specifically, it evaluates whether there is a deficiency or excess of moisturizing or nutritional ingredients and determines whether it is optimal for the user's skin condition and skin care goals.

[1197] Step 8:

[1198] The server uses the substitute suggestion means to search for the most suitable substitute skin care item based on the component evaluation results.

[1199] Specifically, the system will create a list of products from the database that will supplement the missing ingredients or balance the skin overall. For example, if the skin lacks moisturizing ingredients, the system will search for a moisturizing lotion.

[1200] Step 9:

[1201] The server uses an emotion engine to recognize the user's emotional state when making alternative suggestions.

[1202] Specifically, the system analyzes the user's voice data and facial expression data to detect positive or negative reactions.

[1203] Step 10:

[1204] The server reflects the emotion recognition results from the emotion engine in proposing alternative products.

[1205] Specifically, the system prioritizes ingredients and products to which the user previously had a positive reaction, and avoids ingredients and products to which the user previously had a negative reaction.

[1206] Step 11:

[1207] The server generates a list of suggested alternative products and sends it to the terminal along with detailed information (ingredients, effects, reviews, price range, etc.).

[1208] Specifically, a list is created that includes the characteristics of each proposed product and additional information such as user reviews.

[1209] Step 12:

[1210] The terminal displays detailed information of the substitute item received from the server to the user.

[1211] Specifically, it will be displayed visually as a list, making it easy for users to compare ingredients, effects, reviews, and prices.

[1212] Step 13:

[1213] The user can then refer to the displayed list of alternatives to select and purchase the most suitable skin care product.

[1214] Specifically, users can choose their favorite items from the suggested items and purchase them from the online store.

[1215] This detailed processing step provides users with specific and intuitive assistance in choosing the skin care products that best suit their skin and emotional state.

[1216] Example 2

[1217] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1218] Current skincare product recommendation systems lack the ability to easily obtain ingredient information about products currently in use and then suggest appropriate alternatives based on that information. Furthermore, they do not optimize recommendations based on the user's emotional responses, leaving insufficient means to increase user satisfaction. Furthermore, there is no mechanism for incorporating the user's emotional feedback on a suggested product into future recommendations. To address these issues, an advanced recommendation system is needed that incorporates not only image analysis and ingredient evaluation, but also the user's emotional responses.

[1219] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image analysis means for extracting product ingredient information from image data, an ingredient comparison means for comparing the ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the extracted ingredient information, an alternative product suggestion means for searching for and suggesting alternative products, an emotion analysis means for analyzing the user's emotional response to the suggested alternative items, and an information display means for displaying detailed information about the alternative items. This makes it possible to efficiently extract and evaluate ingredient information for skin care products currently used by the user and suggest appropriate alternative products based on the user's skin condition. Furthermore, by analyzing the user's emotional response to the suggested products and reflecting this in the next suggestion, user satisfaction can be increased.

[1220] The "image input means" is a device that acquires image data of the product being used by the user, and in many cases refers to the camera of a mobile terminal.

[1221] The "means for receiving image data" refers to a device or function for receiving image data acquired by the image input means.

[1222] "Image analysis means" refers to technology or equipment for extracting product ingredient information from received image data, and specifically, optical character recognition technology is often used.

[1223] "Ingredient matching means" refers to a technology or device for matching ingredient information extracted by the image analysis means with a database and obtaining information such as detailed effects and recommended usage amounts.

[1224] "Ingredient evaluation means" refers to the technology and equipment used to evaluate the balance and bias of ingredients based on extracted and collated ingredient information.

[1225] The "substitute product suggestion means" refers to a technology or device for searching for and suggesting the most suitable substitute product based on the results of evaluation by the component evaluation means.

[1226] "Emotion analysis means" refers to technology or devices for analyzing a user's emotional response to a proposed substitute item, and includes functions for analyzing voice and facial expression data.

[1227] "Information display means" refers to a device or technology for visually displaying detailed information about the proposed substitute item to the user.

[1228] "Ingredient information" refers to detailed data such as the names, effects, and recommended usage amounts of various ingredients contained in skin care products.

[1229] "Database" refers to a system that stores information used to collate and evaluate ingredient information.

[1230] The skin care product recommendation system of this invention starts when a user takes a picture of a skin care product they own with a smartphone camera and sends the image to a server. The terminal receives the image data and uploads it to the server via a network. At this stage, the terminal is a mobile terminal such as a smartphone.

[1231] The server uses optical character recognition (OCR) technology to analyze the received image data. For example, the server extracts the ingredients "hyaluronic acid, glycerin, aloe vera" from the label of a lotion that has been photographed. The extracted ingredient information is compared with the server's internal database to obtain detailed information about each ingredient's effects and recommended dosage.

[1232] Next, the server uses an ingredient evaluation means to evaluate the overall ingredient balance of the currently used skin care product. For example, if the moisturizing ingredients are below a standard value, it will evaluate the user's skin as dry. Based on this evaluation result, the server searches the database for suitable alternative items and suggests them to the user using an alternative suggestion means. Specifically, it will suggest items such as "highly moisturizing lotion" and "vitamin C serum."

[1233] Furthermore, this system incorporates an emotion engine. The emotion engine on the server analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user responds "I like it" to a suggested product through voice or facial expression, the emotion engine recognizes this as a positive reaction and confirms that the suggestion was appropriate. Conversely, if a negative reaction is observed, that feedback is reflected in the next suggestion. For example, if the user expresses dissatisfaction with a suggested moisturizing lotion, the next suggestion will be a moisturizing lotion with different ingredients or brand.

[1234] Detailed information on the suggested alternative items is sent from the server to the terminal, and the user can visually check it through the terminal's information display means. The information includes ingredients, effects, reviews, price range, etc. Based on this information, the user can select and purchase the most suitable item.

[1235] Specific examples

[1236] For example, if a user is using a lotion containing "hyaluronic acid, glycerin, and aloe vera," the server compares the ingredient information with the database to obtain detailed effects and recommended usage amounts. The server then evaluates that the moisturizing effect is below the standard value and suggests a "lotion with high moisturizing power." If the user responds to this suggestion by saying "like," the emotion engine recognizes this and determines that the suggestion was appropriate.

[1237] Prompt Sentence Examples

[1238] "Please tell me the ingredients of the lotion you are currently using."

[1239] "Please suggest skin care products with high moisturizing effects."

[1240] Please tell me an alternative to this lotion.

[1241] In this way, by combining advanced image analysis technology with emotion analysis functions, this system can increase user satisfaction and provide optimal skin care products.

[1242] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1243] Step 1:

[1244] The user uses the camera function of their smartphone to take a photo of their skin care product. Specifically, the user opens the camera app and takes a photo of the label of the skin care product.

[1245] Input: Skincare product image

[1246] Output: Captured image data

[1247] Step 2:

[1248] The device compresses the captured image data and uploads it to a server via the network, for example by converting it to JPEG format to reduce the file size.

[1249] Input: Photographed image data

[1250] Output: Compressed image data

[1251] Step 3:

[1252] The server uses optical character recognition (OCR) technology to analyze the received image data. The server inputs the image data into an OCR engine and extracts the component information as text.

[1253] Input: Compressed image data

[1254] Output: Extracted ingredient information (text format)

[1255] Step 4:

[1256] The server compares the extracted ingredient information with its internal database, specifically by executing a database query using the ingredient name as a key to obtain detailed information about each ingredient's effects and recommended dosage.

[1257] Input: Extracted ingredient information

[1258] Output: Detailed ingredient information (effects, recommended dosage, etc.)

[1259] Step 5:

[1260] The server uses the component evaluation means to evaluate the balance and bias of the components. For example, if the amount of a specific moisturizing component is determined to be below a reference value, the server determines the user's skin condition as dry.

[1261] Input: Detailed ingredient information

[1262] Output: Evaluation results (ingredients balance, skin condition)

[1263] Step 6:

[1264] The server searches for and suggests alternative products from a database based on the evaluation results.The server uses the alternative product suggestion means to list and suggest suitable skin care products (e.g., moisturizing lotion).

[1265] Input: Evaluation result

[1266] Output: A list of suggested replacements

[1267] Step 7:

[1268] The server's emotion engine analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, it uses voice and facial expression data obtained from the user's remote camera or microphone as input and applies an emotion analysis algorithm to determine whether the reaction is positive or negative.

[1269] Input: Voice data, facial expression data

[1270] Output: Emotional response analysis results

[1271] Step 8:

[1272] The server sends detailed information about the proposed substitute items to the terminal and displays it to the user through the terminal's information display means. The user can then check this information and select the most suitable item.

[1273] Input: List of suggested replacements, emotional response analysis results

[1274] Output: Detailed information displayed on the terminal

[1275] Step 9:

[1276] The user checks the detailed information of the suggested items displayed on the device, selects the item that best suits them, and then purchases it. Specifically, the user clicks on a link to access the online shopping site and completes the purchase procedure.

[1277] Input: Detailed information (ingredients, effects, reviews, price range)

[1278] Output: Purchase order for selected skin care products

[1279] (Application example 2)

[1280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1281] In recent years, with the diversification of skin care products, users have found it difficult to select the product that best suits their skin. Furthermore, there are few feedback systems based on the user's emotional state, making it a challenge to improve user satisfaction. There is a need to solve these issues and provide a system that recommends more effective and satisfying skin care products.

[1282] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image input means, a means for receiving image data acquired by the image input means, an image analysis means for extracting product ingredient information from the received image data, an ingredient comparison means for comparing the extracted ingredient information with a database, an ingredient evaluation means for evaluating the balance and imbalance of ingredients based on the ingredient information, an alternative product suggestion means for searching for and suggesting alternative items based on the ingredient balance and imbalance evaluated by the ingredient evaluation means, an emotion analysis means for analyzing the user's emotional state, a feedback means for evaluating the user's emotional response using the emotion analysis means and adjusting the next suggestion, and an information display means for displaying detailed information about the alternative items. This enables the user to select optimal skin care products based on their skin condition and emotional state.

[1283] "Image input means" refers to a device or function for acquiring image data captured by a user.

[1284] The "receiving means" refers to a device or function for receiving image data transmitted from the image input means.

[1285] "Image analysis means" refers to a device or function for extracting product ingredient information from received image data.

[1286] The "ingredient collating means" refers to a device or function for collating extracted ingredient information with a database.

[1287] The "component evaluation means" refers to a device or function for evaluating the balance and bias of components based on component information.

[1288] The "substitute product suggestion means" refers to a device or function for searching for and suggesting substitute items based on the component balance and bias evaluated by the component evaluation means.

[1289] "Information display means" refers to a device or function for displaying detailed information about a substitute item.

[1290] "Emotion analysis means" refers to a device or function for analyzing the user's emotional state.

[1291] "Feedback means" refers to a device or function that evaluates the user's emotional response using emotion analysis means and adjusts the content of the next suggestion.

[1292] In order to implement the present invention, it is necessary to construct the following system.

[1293] System Overview

[1294] The system includes an image input unit, a receiving unit, an image analysis unit, an ingredient matching unit, an ingredient evaluation unit, an alternative product suggestion unit, an information display unit, an emotion analysis unit, and a feedback unit. By linking these units, the system can suggest optimal skin care products to users.

[1295] Hardware and Software

[1296] Hardware:

[1297] Smartphone: Used as an image input and user interface

[1298] Servers: Data processing and storage

[1299] Camera: Built-in camera on smartphone

[1300] Microphone: A device for voice input.

[1301] software:

[1302] Image Analysis: OpenCV and Google Tesseract OCR

[1303] Server communication: requests module (Python)

[1304] Emotion analysis: emotion_recognition library (Python)

[1305] Programming: Python language used

[1306] Processing steps

[1307] 1. Image Acquisition:

[1308] Users take a photo of a skin care product label using their smartphone camera, and the image data is temporarily stored in the smartphone's memory and then sent to the server.

[1309] 2. Image Analysis:

[1310] The server analyzes the received image data and extracts component information from the image using Google Tesseract OCR.

[1311] 3. Ingredient Matching:

[1312] The extracted ingredient information is compared with a database on the server, and detailed information about the ingredients, their effects, recommended usage, and other information is retrieved from the database.

[1313] 4. Ingredients evaluation:

[1314] Based on the acquired ingredient information, the server evaluates the balance and imbalance of ingredients, particularly determining whether moisturizing ingredients and nutritional supplement ingredients are appropriate.

[1315] 5. Substitute suggestions:

[1316] Based on the results of the ingredient evaluation, the system proposes alternative skin care products that are best suited to the user's skin condition. The system searches for optimal products from a database in the server via an alternative product suggestion means and proposes them to the user.

[1317] 6. Information display:

[1318] Users can view detailed information about suggested alternative products via their smartphone, including ingredients, effects, other users' reviews, and prices.

[1319] 7. Emotion analysis:

[1320] The emotion analysis means analyzes the user's voice and facial expression data to evaluate their emotional reaction to the proposed product. If a positive reaction is obtained, it is fed back to the server and used to improve the next proposal.

[1321] 8. Feedback and Suggested Adjustments:

[1322] The next recommendation will be optimized based on the feedback information obtained through sentiment analysis. For example, if there is a negative reaction, the next recommendation will be a product with different ingredients or brand.

[1323] Specific examples

[1324] Consider a scenario where a user takes a photo of the label of a "lotion containing hyaluronic acid" with their smartphone camera in the skincare section of a physical store. The program captures the image and uses Tesseract OCR to extract ingredient information such as "hyaluronic acid, glycerin, aloe vera." The data sent to the server is analyzed, and the result is returned indicating that the moisturizing ingredients are highly rated.

[1325] Furthermore, based on the user's feedback, the company analyzed their sentiment and found that they responded positively to the question, "Is this lotion effective for dry skin?", so it decided to continue recommending skin care products with high moisturizing power from next time onwards. These results are displayed on the AI ​​assistant's display in the physical store.

[1326] Prompt Sentence Examples

[1327] "How can we develop an application that analyzes the ingredients of skin care products taken by customers in physical stores and suggests the best alternatives based on their skin condition and emotional state?"

[1328] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1329] Program processing steps

[1330] Step 1:

[1331] A user takes a photo of a skin care product label using a smartphone at the skin care section of a physical store. The image is captured using the smartphone's camera function, and the captured image data is saved on the smartphone. The input is the image data of the label, and the output is the saved image file.

[1332] Step 2:

[1333] The device sends the captured image data to a server. Specifically, the image file on the smartphone is compressed and uploaded to the server via the Internet. The input is the image file, and the output is the image data sent to the server.

[1334] Step 3:

[1335] The server analyzes the received image data. It uses Google Tesseract OCR to extract component information from the image in text format. The input is image data, and the output is component information in text format.

[1336] Step 4:

[1337] The server compares the extracted ingredient information with an internal database. Using the ingredient comparison means, detailed information, effects, and recommended usage of the relevant ingredients are obtained from the ingredient database. The input is ingredient information in text format, and the output is detailed information about the ingredients.

[1338] Step 5:

[1339] The server uses the ingredient evaluation means to evaluate the balance and imbalance of ingredients based on ingredient information. In particular, it checks the balance of moisturizing ingredients and nutritional supplement ingredients. The input is detailed information about the ingredients, and the output is the evaluation result of the ingredient balance. Specifically, if the moisturizing ingredients are below the standard value, the evaluation result will show "insufficient moisturizing ingredients."

[1340] Step 6:

[1341] The server uses the substitute suggestion means based on the ingredient evaluation results to search for and suggest alternative skin care products that are best suited to the user's skin condition. Products that meet the criteria are searched for from the skin care product database within the server. The input is the ingredient evaluation results, and the output is a list of suggested alternative skin care products.

[1342] Step 7:

[1343] The server presents detailed information about the suggested alternative items to the user through an information display means. The smartphone display shows the product's ingredients, effects, reviews from other users, price, etc. The input is a list of alternative skin care products, and the output is detailed information displayed on the smartphone screen.

[1344] Step 8:

[1345] The user's voice and facial expressions are collected through a camera and microphone and sent from the device to a server. The input is the user's voice and facial expression data, and the output is the emotional data sent to the server.

[1346] Step 9:

[1347] The server analyzes the user's emotional state using emotion analysis. Specifically, it uses the emotion_recognition library to identify emotions from voice and facial expression data and determine whether they are positive or negative. The input is the user's voice and facial expression data, and the output is the result of the emotion analysis.

[1348] Step 10:

[1349] The server uses a feedback mechanism to adjust the next recommendation based on the sentiment analysis results. For example, if a negative reaction is shown, it may suggest a product with different ingredients or brand the next time. The input is the sentiment analysis results, and the output is the next recommendation.

[1350] The above processing steps allow the user to select the most suitable skin care product based on their skin condition and emotional state.

[1351] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1352] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1353] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1354] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1355] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1356] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1357] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1358] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1359] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1360] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1361] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1362] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1363] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1364] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1365] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1366] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1367] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1368] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1369] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1370] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1371] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1372] The following is further disclosed regarding the above embodiment.

[1373] (Claim 1)

[1374] Image input means;

[1375] means for receiving image data acquired by the image input means;

[1376] image analysis means for extracting product ingredient information from the received image data;

[1377] a component collating means for collating the extracted component information with a database;

[1378] a component evaluation means for evaluating the balance and bias of components based on the component information;

[1379] an alternative product suggestion means for searching for and suggesting an alternative item based on the component balance and bias evaluated by the component evaluation means;

[1380] an information display means for displaying detailed information about the substitute item;

[1381] A system including:

[1382] (Claim 2)

[1383] 2. The system of claim 1, wherein the image input means is a smartphone camera.

[1384] (Claim 3)

[1385] 10. The system of claim 1, wherein said image analysis means uses optical character recognition techniques.

[1386] (Claim 4)

[1387] The system of claim 1 , wherein the ingredient evaluation means takes into account the user's skin condition and goals.

[1388] (Claim 5)

[1389] The system of claim 1 , wherein the information display means displays ingredients, effects, reviews, and price ranges of alternative items.

[1390] (Claim 6)

[1391] The system according to claim 1, wherein the substitute suggestion means comprehensively evaluates the balance of ingredients in all skin care products and suggests items that supplement missing ingredients.

[1392] "Example 1"

[1393] (Claim 1)

[1394] an image input means for a user to take an image of a skin care product;

[1395] means for receiving image data acquired by the image input means;

[1396] means for compressing and transmitting the received image data;

[1397] image analysis means for extracting product ingredient information from the received image data;

[1398] a component matching means for matching the extracted component information with a database;

[1399] a component evaluation means for evaluating the balance and bias of components based on the component information;

[1400] an alternative product suggestion means for searching for and suggesting an alternative item based on the component balance and bias evaluated by the component evaluation means;

[1401] an information display means for displaying detailed information about the substitute item;

[1402] A system including:

[1403] (Claim 2)

[1404] 2. The system according to claim 1, wherein the image input means is a camera of a mobile terminal.

[1405] (Claim 3)

[1406] 10. The system of claim 1, wherein said image analysis means uses optical character recognition techniques.

[1407] "Application Example 1"

[1408] (Claim 1)

[1409] Image input means;

[1410] means for receiving image data acquired by the image input means;

[1411] image analysis means for extracting product ingredient information from the received image data;

[1412] a component collating means for collating the extracted component information with a database;

[1413] a component evaluation means for evaluating the balance and bias of components based on the component information;

[1414] an alternative product suggestion means for searching for and suggesting an alternative item based on the component balance and bias evaluated by the component evaluation means;

[1415] an information display means for displaying detailed information about the substitute item;

[1416] The information display means visually displays information on a physical store terminal;

[1417] A system including:

[1418] (Claim 2)

[1419] 2. The system according to claim 1, wherein the image input means is a smartphone camera or a camera equipped in a brick-and-mortar store terminal.

[1420] (Claim 3)

[1421] 10. The system of claim 1, wherein said image analysis means uses optical character recognition techniques.

[1422] "Example 2: Combining Emotion Engines"

[1423] (Claim 1)

[1424] Image input means;

[1425] means for receiving image data acquired by the image input means;

[1426] image analysis means for extracting product ingredient information from the received image data;

[1427] a component collating means for collating the extracted component information with a database;

[1428] a component evaluation means for evaluating the balance and bias of components based on the component information;

[1429] an alternative product suggestion means for searching for and suggesting alternative products based on the component balance and bias evaluated by the component evaluation means;

[1430] emotion analysis means for analyzing a user's emotional reaction to the substitute item proposed by the substitute suggestion means;

[1431] an information display means for displaying detailed information about the substitute item;

[1432] A system including:

[1433] (Claim 2)

[1434] 2. The system according to claim 1, wherein the image input means is a camera of a mobile terminal.

[1435] (Claim 3)

[1436] 10. The system of claim 1, wherein said image analysis means uses optical character recognition techniques.

[1437] "Application example 2 when combining emotion engines"

[1438] (Claim 1)

[1439] Image input means;

[1440] means for receiving image data acquired by the image input means;

[1441] image analysis means for extracting product ingredient information from the received image data;

[1442] a component collating means for collating the extracted component information with a database;

[1443] a component evaluation means for evaluating the balance and bias of components based on the component information;

[1444] an alternative product suggestion means for searching for and suggesting an alternative item based on the component balance and bias evaluated by the component evaluation means;

[1445] an information display means for displaying detailed information about the substitute item;

[1446] emotion analysis means for analyzing the emotional state of a user;

[1447] a feedback means for evaluating the emotional response of the user by the emotion analysis means and adjusting the content of the next suggestion;

[1448] A system including:

[1449] (Claim 2)

[1450] 2. The system of claim 1, wherein the image input means is a mobile terminal camera.

[1451] (Claim 3)

[1452] 10. The system of claim 1, wherein said image analysis means uses optical character recognition techniques. [Explanation of symbols]

[1453] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Image input means; means for receiving image data acquired by the image input means; image analysis means for extracting product ingredient information from the received image data; a component collating means for collating the extracted component information with a database; a component evaluation means for evaluating the balance and bias of components based on the component information; an alternative product suggestion means for searching for and suggesting an alternative item based on the component balance and bias evaluated by the component evaluation means; an information display means for displaying detailed information about the substitute item; A system including:

2. The system of claim 1 , wherein the image input means is a smartphone camera.

3. The system of claim 1 , wherein said image analysis means uses optical character recognition techniques.

4. The system of claim 1 , wherein the ingredient evaluation means takes into account the user's skin condition and goals.

5. The system of claim 1 , wherein the information display means displays ingredients, effects, reviews, and price ranges of alternative items.

6. The system according to claim 1 , wherein the substitute suggestion means comprehensively evaluates the balance of ingredients in all skin care products and suggests items that will supplement any missing ingredients.

Images