System

Abstract

A system is provided.SOLUTION: A system, comprising: means for registering information of a detergent possessed by a user; means for capturing an image of dirt using a camera of a terminal; means for analyzing the captured image of the dirt to identify a type of the dirt; means for suggesting an optimal detergent based on the information of the detergent possessed by the user; and means for providing an option to purchase an appropriate detergent from an e-commerce site when the suggested detergent is not in stock of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For people who usually clean themselves, such as those living alone or housewives, the task of finding the best detergent for various stains can be time-consuming. It can also be difficult to choose the right detergent from among those already in stock, which can sometimes require purchasing a new one. The purpose of this invention is to provide a solution that reduces these hassles and allows for efficient cleaning. [Means for solving the problem]

[0005] To solve this problem, we provide a system that includes the following means: means for registering information about detergents owned by a user, means for photographing stains using a camera on a terminal, means for analyzing the photographed image of the stains to identify the type of stain, means for suggesting an optimal detergent based on the information about the detergents owned by the user, and means for providing the user with the option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock.

[0006] "User" refers to any individual or legal entity that uses the System.

[0007] "Detergent" refers to a liquid or powdered chemical product used to remove stains.

[0008] "Information" refers to data such as the name, manufacturer, use, and effects of the registered detergent.

[0009] "Device" refers to a portable electronic device such as a smartphone or tablet.

[0010] "Camera" refers to a device built into a device for capturing images.

[0011] "Photography" refers to the act of generating an image using a camera.

[0012] "Dirt" refers to unwanted substances or stains that adhere to an object.

[0013] "Image" refers to visual data captured by a camera.

[0014] "Analysis" refers to the process of analyzing captured image data and extracting information.

[0015] "Identification" refers to the act of identifying the type of dirt based on the information obtained through analysis.

[0016] "Suggestion" refers to showing the user the most suitable detergent based on the analysis results.

[0017] "E-commerce Site" means a website where goods are bought and sold over the Internet.

[0018] "Options" refers to offering additional choices or features. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that registers information about the detergents a user owns and suggests the most suitable detergent for stains photographed with a smartphone or other device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[0041] 1. Detergent Registration

[0042] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to the server and stored in association with the user's account.

[0043] Examples:

[0044] The user manually enters "Laundry Detergent."

[0045] When a user takes a photo of the "dishwashing detergent" label with the device's camera, the device automatically extracts the information and sends it to the server.

[0046] 2. Photograph the stain and send the image

[0047] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0048] Examples:

[0049] The user takes a picture of a stain on the carpet.

[0050] The device preprocesses the image and sends it to the server.

[0051] 3. AI-based stain analysis

[0052] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[0053] Examples:

[0054] The server analyzes the image and determines that it is a coffee stain.

[0055] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[0056] 4. Suggestions and Recommendations

[0057] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0058] Examples:

[0059] The server sends a suggestion to the device: "Use laundry detergent."

[0060] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[0061] In this way, the system utilizes the detergent information the user has, allowing them to efficiently select the optimal detergent to remove dirt. At each step, image recognition technology and AI analysis are used to minimize the user's effort.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[0065] Step 2:

[0066] The device captures the label image and uses image recognition to extract the necessary information from the label.

[0067] Step 3:

[0068] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[0069] Step 4:

[0070] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[0071] Step 5:

[0072] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[0073] Step 6:

[0074] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[0075] Step 7:

[0076] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[0077] Step 8:

[0078] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[0079] As a specific example, if a user registers "dishwashing detergent" and takes a photo of a coffee stain on a carpet, the server will analyze the image and determine that "laundry detergent" is the best option. If the laundry detergent is not in stock, a purchase link will be provided.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] Conventional detergent selection systems lack the functionality to efficiently manage the detergent information owned by the user and suggest the detergent best suited to a specific stain. Furthermore, when users manually input the information about the detergents they own, it is highly likely to result in time-consuming work and input errors. Furthermore, if the suggested detergent is not in stock, the process of purchasing it can be cumbersome. Therefore, an efficient and accurate system that can solve these issues is needed.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera on the terminal, means for preprocessing the photographed image of the stain and transmitting it to the server, means for analyzing the photographed image of the stain to identify the type of stain, means for suggesting an optimal detergent based on information about the detergents owned by the user, and means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user. This makes it possible to efficiently manage information about detergents owned by the user, quickly suggest an optimal detergent for a specific stain, and easily perform the procedure for purchasing the necessary detergent.

[0085] A "user" is a person who registers detergent information, takes an image of the stain using a terminal, and receives recommendations for the most suitable detergent from the system.

[0086] A "terminal" is a device that has a camera function, takes an image of the stain, performs preprocessing, and transmits the image to the server.

[0087] The "server" is the central computer in the system that stores information about detergents registered by users, analyzes received images of stains, suggests the most suitable detergent, and also provides purchasing options.

[0088] "Means for registering detergent information" refers to a function that allows information such as the name, manufacturer, and purpose of the detergent the user owns to be entered into the application or obtained through image recognition, and then saved on the server.

[0089] The "means for photographing dirt" refers to a function that uses the camera of the terminal to take an image of the dirt that the user needs to clean.

[0090] "Preprocessing" refers to the provision of image processing technology to improve analysis accuracy by performing processes such as color correction and noise removal on captured images.

[0091] "Analysis means" refers to a function that uses an AI model installed on the server to identify the type of dirt in the captured image.

[0092] The "suggestion means" is a function that suggests the most suitable detergent from among the detergents the user owns based on the analyzed type of dirt.

[0093] "Means for providing purchase options" refers to a function that provides a link to an e-commerce site so that the appropriate detergent can be purchased directly if the suggested detergent is not in the user's inventory.

[0094] The "image recognition function" is a technology that extracts information such as the name, manufacturer, and purpose from the detergent label photographed with the device's camera.

[0095] An "AI model" is a model trained using machine learning technology to analyze photographed images of dirt and identify the type of dirt.

[0096] An "e-commerce site" is a website for purchasing goods over the Internet.

[0097] This invention is a system that registers information about the detergents a user owns and suggests the best detergent for stains photographed with a device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[0098] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device then uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to a server and stored in association with the user's account.

[0099] For example, if a user takes a photo of the label of a "laundry detergent" product with their device's camera, the device automatically extracts the information and sends it to the server. This information is then stored in the user's account, allowing them to add other detergents, such as "dish detergent," in the same way.

[0100] Next, if the user finds a stain that needs cleaning, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device, with color adjustment and noise removal performed. The processed image is then sent to the server. For example, if a user takes a photo of a stain on a carpet, the device pre-processes the image and sends it to the server.

[0101] The server has an AI model (e.g., a TensorFlow or PyTorch-based model) that analyzes the received image and identifies the stains in the image. The AI ​​identifies the type of stain (food stain, oil stain, mud stain, etc.) and compares the result with the user's registered information stored in the server's database. If an appropriate detergent is found, it generates a recommendation for the user. For example, if the server analyzes an image and determines that it is a coffee stain, the server will identify from the user's registered information that "laundry detergent" is suitable for coffee stains.

[0102] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so that the user can purchase it directly. For example, if the server sends the suggestion "Use laundry detergent" to the terminal and the user does not have "laundry detergent," a link to Amazon or other sites will also be displayed.

[0103] In this system, examples of prompts to be input to the generative AI model include:

[0104] "A user finds a coffee stain on their carpet and takes a photo of it with their device. The AI ​​should analyze the stain and suggest the best detergent to use. If the suggested detergent is not in stock, the AI ​​should also provide a link to purchase it."

[0105] This prompt allows the system to consistently analyze and make recommendations based on the input data for a specific stain.

[0106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0107] Step 1:

[0108] Users register information about the detergents they own in the application.

[0109] Input: The user manually enters the name, manufacturer, and purpose of the detergent. Alternatively, the user can take a photo of the detergent label using the device's camera.

[0110] Data processing: In the case of manual input, the information entered in the form is verified and saved. In the case of label photography, the device uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, purpose) from the photographed label.

[0111] Output: Extracted detergent information (name, manufacturer, purpose).

[0112] Specific operation: The user manually enters "Laundry detergent." The user takes a photo of the "Dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[0113] Step 2:

[0114] The device sends the extracted detergent information to a server and stores it in the user's account.

[0115] Input: Extracted detergent information (name, manufacturer, purpose).

[0116] Data calculation: The device communicates with the server using a secure protocol such as HTTPS and transmits information.

[0117] Output: User's detergent information stored in the server's database.

[0118] Specific operation: The device sends the information of "dishwashing detergent," "ABC manufacturer," and "dishware" to the server, and the server stores it in the user's account.

[0119] Step 3:

[0120] When a user finds dirt that needs to be cleaned, they take a picture of the dirt with the device's camera.

[0121] Input: A dirty area (e.g., a carpet), and the device's camera to take a photo.

[0122] Data processing: Take an image of the dirt using the device's camera.

[0123] Output: The captured image of the stain.

[0124] Specific operation: The user takes a photo of a stain on the carpet using the device's camera.

[0125] Step 4:

[0126] The captured images are first pre-processed on the device and then sent to the server.

[0127] Input: A photographed image of the dirt.

[0128] Data processing: The terminal performs pre-processing such as color adjustment and noise removal.

[0129] Output: Preprocessed stain image.

[0130] Specific operation: The device performs color correction and noise reduction on the captured image, and then sends the preprocessed image to the server via HTTPS protocol.

[0131] Step 5:

[0132] The server uses an AI model to analyze the received image, which then identifies the stains in the image.

[0133] Input: Preprocessed stain images.

[0134] Data processing: An AI model (e.g., TensorFlow or PyTorch) installed on the server analyzes the image and identifies the type of dirt.

[0135] Output: The type of stain identified (e.g., coffee stain).

[0136] How it works: The server inputs the preprocessed image into the AI ​​model and identifies it as a coffee stain.

[0137] Step 6:

[0138] The server compares the analysis results with the user's registered information and selects the most suitable detergent.

[0139] Input: Identified stain type, user detergent information.

[0140] Data calculation: Matches the type of dirt with the intended use of the detergent the user owns to identify the most suitable detergent.

[0141] Output: Information on the best detergent.

[0142] Specific behavior: The server determines that "laundry detergent" is best for removing coffee stains.

[0143] Step 7:

[0144] The server transmits the suggested detergent information to the terminal and notifies the user.

[0145] Input: Information on the best detergent.

[0146] Data calculation: Sending data for proposal generation and notification.

[0147] Output: Proposal.

[0148] Specific operation: The server sends a suggestion to the device saying "Use laundry detergent."

[0149] Step 8:

[0150] If the user does not have the suggested detergent, the system will provide a link to an e-commerce site where they can purchase it directly.

[0151] Input: detergent inventory information, e-commerce site link information.

[0152] Data calculation: Checking stock information and generating links.

[0153] Output: Purchase link.

[0154] Specific operation: If the server does not have "laundry detergent," it generates a link to Amazon or another site and sends it to the device.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Today's consumers are often confused by the wide variety of cleaning methods available, making it difficult to choose the right cleaning agent. Furthermore, when selecting products in physical stores, there is a lack of a system that instantly suggests the best cleaning agent for each type of dirt and purpose. This creates the risk of using the wrong cleaning agent and damaging the product.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes: means for registering information about detergents owned by the user; means for taking pictures of stains using a camera on the terminal; means for analyzing the captured images of stains and using a generative AI model to identify the type of stain; means for suggesting an optimal detergent based on the information about the detergents owned by the user; means for providing an option to purchase an appropriate detergent from an online sales platform if the suggested detergent is not in the user's inventory; and means for acquiring product information by scanning product barcodes and suggesting the optimal detergent based on the product information. This allows users to instantly find the optimal detergent and apply an appropriate cleaning method when selecting products in a physical store.

[0160] "User" refers to an individual or corporation that uses this system.

[0161] "Cleaning agents" refers to a general range of chemical products used to remove dirt from users' hands.

[0162] "Means for registering information" refers to the function of inputting and saving data on the cleaning agents owned by the user into the system.

[0163] "Device" refers to a smartphone, tablet, or other portable electronic device.

[0164] "Camera" refers to an image capturing device built into the terminal.

[0165] A "generative AI model" refers to an algorithm that uses artificial intelligence to analyze images of dirt and identify its features.

[0166] "Means for suggesting the most suitable cleaning agent" refers to the function of selecting and presenting the appropriate cleaning agent based on the user's registered information and analysis results.

[0167] "Online sales platform" refers to an e-commerce site that allows products to be purchased via the Internet.

[0168] "Barcode scanning function" refers to the function of using a camera to read the barcode attached to the product.

[0169] "Means for analyzing images" refers to technology for processing captured images and identifying their contents.

[0170] "Pre-processing" refers to the initial processing steps performed to improve the quality of a captured image.

[0171] "Server" refers to a computer system for processing, storing, and communicating data.

[0172] This invention is a system that suggests optimal detergents when a user selects products in a physical store. The system communicates between the user, terminal, and server to support the effective use of detergents.

[0173] 1. Cleaning Agent Registration

[0174] First, the user registers information about the detergent they own in the device's application. This information can be obtained by manually entering it or by taking a photo of the detergent label with the device's camera and using image recognition. The device extracts the necessary information (name, manufacturer, and purpose) from the photo of the label and sends it to the server. The server stores this information, linking it to each user's account.

[0175] Specific examples

[0176] The user manually enters "clothing detergent."

[0177] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information and sends it to the server.

[0178] 2. Photograph the stain and send the image

[0179] When a user finds a stain on a product, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device (color adjustment, noise removal, etc.), and then the processed image is sent to the server.

[0180] Specific examples

[0181] The user takes a picture of a stain on the carpet.

[0182] The device preprocesses the image and sends it to the server.

[0183] 3. AI-based stain analysis

[0184] The server analyzes the received image using a generative AI model to identify the type of dirt. The AI ​​identifies the type of dirt (food stains, oil stains, mud stains, etc.) and compares the results with the user's registered information stored in the server's database. If an appropriate cleaning agent is found, it will be recommended to the user.

[0185] Specific examples

[0186] The server analyzes the image and determines that it is a coffee stain.

[0187] The server determines from the user's registration information that "clothing detergent" is suitable for removing coffee stains.

[0188] 4. Obtain product information by scanning barcodes

[0189] Users can also obtain product information by scanning the product barcode with the device's camera. The barcode scanning function allows the system to recommend the most suitable detergent based on the product information.

[0190] Specific examples

[0191] A user scans the barcode of the clothing item they want to purchase in a brick-and-mortar store.

[0192] The terminal sends the barcode information to the server, which then identifies the most suitable cleaning agent.

[0193] 5. Suggestions and Recommendations

[0194] Based on the analysis results, the server will send the suggested cleaning product information to the terminal. If the user does not have the suggested cleaning product in stock, the system will also provide a link to an online sales platform for direct purchase.

[0195] Specific examples

[0196] The server sends a suggestion to the terminal saying "Use laundry detergent."

[0197] The terminal will display this offer to the user and, if the user does not have "clothes detergent", will also display a link to an online sales platform.

[0198] Prompt Sentence Examples

[0199] "Identify the type of stain in this image. Types of stains include coffee stains, oil stains, mud stains, etc."

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] Users register information about the detergent they own on their device. To do this, they can either manually enter the information or take a photo of the detergent label with the device's camera. The captured image of the label is analyzed on the device using image recognition technology, and the necessary information (name, manufacturer, purpose, etc.) is automatically extracted. The extracted information is sent to a server and saved, linked to the user's account.

[0203] Input: Manually enter cleaning agent information or label image

[0204] Data processing: Information extraction using image recognition

[0205] Output: Save cleaning agent information to database

[0206] Step 2:

[0207] When a user finds a product or stain, they take a photo of the stain with their device's camera. The captured image undergoes pre-processing such as color adjustment and noise removal on the device, and then the processed image is sent to the server.

[0208] Input: Image of dirt

[0209] Data processing: Image preprocessing (color adjustment, noise removal)

[0210] Output: Processed images sent to server

[0211] Step 3:

[0212] The server analyzes the received image using a generative AI model to identify the type of stain. The generative AI model uses prompts to analyze the characteristics of the stain and identify the specific stain type (e.g., coffee stain, oil stain, mud stain, etc.). Once the stain type is identified, the information is stored in the server's database.

[0213] Input: Processed image

[0214] Data Computation: Identifying Contamination with Generative AI Models

[0215] Output: Server save of dirt type information

[0216] Step 4:

[0217] The server compares the information on the type of dirt with the information on the detergent registered by the user to identify the most suitable detergent. If an appropriate detergent is identified, the recommendation is sent to the device.

[0218] Input: Dirt type information, user-registered cleaning agent information

[0219] Data calculations: collating information to identify optimal cleaning products

[0220] Output: Sends the proposal to the device.

[0221] Step 5:

[0222] The user can review the suggestions on their device and select the appropriate cleaning product. If the user does not have the suggested cleaning product in stock, the server will also provide a link to an online sales platform so that the user can purchase it directly.

[0223] Enter: View Suggestions

[0224] Data calculation: inventory check and link generation

[0225] Output: Display of online sales platform link

[0226] Step 6:

[0227] When a user selects a product in a physical store, they scan the product's barcode with the device's camera, and the device sends the barcode information to the server, which then identifies the appropriate cleaning product and sends a recommendation to the device.

[0228] Input: Product barcode

[0229] Data calculation: Analysis of barcode data and identification of cleaning agents

[0230] Output: Sends the proposal to the device.

[0231] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0232] This invention is a system that suggests detergents to properly remove dirt, and adds a function to recognize the user's emotions and customize the suggestions based on those emotions. This system communicates between the user, the terminal, and the server, and by combining an emotion engine, makes suggestions that are more appropriate and effective for the user.

[0233] 1. Detergent Registration

[0234] First, users register information about their detergent in the application. This involves manually entering the information or taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the label. This information is then sent to the server and stored in association with the user's account.

[0235] Examples:

[0236] The user manually enters "Laundry Detergent."

[0237] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[0238] 2. Photograph the stain and send the image

[0239] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0240] Examples:

[0241] The user takes a picture of a stain on the carpet.

[0242] The device preprocesses the image and sends it to the server.

[0243] 3. AI-based stain analysis

[0244] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[0245] Examples:

[0246] The server analyzes the image and determines that it is a coffee stain.

[0247] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[0248] 4. Suggestions and Recommendations

[0249] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0250] Examples:

[0251] The server sends a suggestion to the device: "Use laundry detergent."

[0252] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[0253] 5. Use of Emotion Engine

[0254] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Based on this emotion data, the server further customizes the suggestions.

[0255] Examples:

[0256] If the user shows a confused look on their face about which detergent to choose, the emotion engine will recognize that emotion and send it to the server.

[0257] The server receives the emotion data and sends suggestions to the terminal, including detailed usage instructions that are easy for the user to understand.

[0258] In this way, the present invention, which combines an emotion engine, not only suggests the most suitable detergent based on the detergent information the user has and the results of stain analysis, but also recognizes the user's emotions and adjusts the suggestions based on them, providing a more user-friendly system.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[0262] Step 2:

[0263] The device captures the label image and uses image recognition to extract the necessary information from the label.

[0264] Step 3:

[0265] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[0266] Step 4:

[0267] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[0268] Step 5:

[0269] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[0270] Step 6:

[0271] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[0272] Step 7:

[0273] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[0274] Step 8:

[0275] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[0276] Step 9:

[0277] The user can enable the emotion engine in the settings of the application.

[0278] Step 10:

[0279] The device acquires the user's emotional data using a camera and microphone. The device captures the user's facial expressions with the camera and records their voice with the microphone.

[0280] Step 11:

[0281] The device analyzes the emotion data to determine the user's current emotional state, a process that identifies emotions such as stress, frustration, and satisfaction.

[0282] Step 12:

[0283] The device sends the analyzed emotional data to the server, which then customizes the detergent recommendations based on the emotional data received.

[0284] Step 13:

[0285] The server tailors its suggestions depending on the user's emotional state, for example, providing suggestions with detailed instructions if the user is confused.

[0286] Step 14:

[0287] The terminal displays the customized suggestions received from the server to the user, allowing the user to receive appropriate suggestions that match their emotions.

[0288] For example, if a user registers "clothing detergent" and takes a photo of a coffee stain on a carpet, the server will suggest "clothing detergent." If the user shows a confused expression, the device will recognize the emotion, and the server will provide detailed instructions for use, which the device will then display.

[0289] Example 2

[0290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Conventional detergent recommendation systems simply make recommendations based on the type of dirt and the detergent information registered by the user, without taking the user's feelings into consideration. This has led to issues such as users being confused by the recommendations or not knowing how to properly operate the system. Furthermore, when multiple detergents are applicable to a particular dirt, the system is unable to provide appropriate advice to help the user make the optimal selection.

[0292] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using the camera of the terminal, means for identifying the type of stain by analyzing the photographed image of the stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in the user's inventory, and means for recognizing the user's emotional state and customizing the suggestions based on the emotion. This makes it easier to deal with cases where the user is confused about which detergent to select or does not know how to operate the system, and makes it possible to provide a user-friendly system.

[0293] "User" refers to an individual or organization that accesses the system and registers detergent information, photographs stains, receives suggestions, etc.

[0294] A "terminal" refers to a computer device that is equipped with devices such as a camera and microphone and that is operated by a user. Examples include smartphones, tablets, and PCs.

[0295] "Server" refers to a computer system that receives information sent from a terminal and analyzes, stores, and provides the data.

[0296] "Detergent information" refers to detailed information such as the detergent's name, manufacturer, and purpose. This data indicates the characteristics of the detergent the user owns.

[0297] "Camera" refers to a device for capturing images, including those built into a device and those connected externally.

[0298] "Soil" refers to impurities or stains on the surface of an object that needs to be cleaned, such as food stains, mud stains, or oil stains.

[0299] "Image recognition technology" refers to technology that automatically extracts and analyzes specific information from captured images, including character recognition and object identification.

[0300] "Preprocessing" refers to the adjustment and processing techniques used to process images and data before analyzing them, including color adjustment and noise removal.

[0301] An "AI model" refers to a software model that uses artificial intelligence technology to analyze data and make predictions. Many of these models use machine learning algorithms.

[0302] "Emotion engine" refers to technology that uses the device's camera and microphone to analyze and recognize emotions from the user's facial expressions and voice.

[0303] An "e-commerce site" is a website that sells and purchases goods and services over the internet. Amazon is an example of this.

[0304] "Suggestion" refers to information that indicates the optimal detergent and its usage method to the user based on the analysis results.

[0305] "Customization" refers to adjusting and changing the content of suggestions according to the user's characteristics and situation.

[0306] The present invention provides a detergent recommendation system that can recognize a user's emotions and customize the content of recommendations based on those emotions. A specific embodiment of this system will be described below.

[0307] First, the user registers information about the detergent they own in the application. This process involves manual input and taking a photo of the detergent label using the device's camera. The device typically used here is a camera-equipped device such as a smartphone or tablet. Software such as TensorFlow and OpenCV is used for image recognition technology.

[0308] For example, there is a case where the user manually enters "laundry detergent" or a case where the user takes a picture of the label of "dishwashing detergent" with the device's camera and the information is automatically extracted. The information extracted from the label (name, manufacturer, purpose) is sent from the device over the Internet to a server. The server associates this information with the user's account and stores it in a database.

[0309] Next, if the user finds a stain that needs cleaning, they take a photo of it with the device's camera. The captured image undergoes preprocessing, such as color adjustment and noise removal, before being sent to the server. Image processing libraries such as OpenCV are used for preprocessing.

[0310] The server uses an AI model to analyze the received image. Common AI models used here include TensorFlow and PyTorch. The AI ​​identifies stains in the image and determines the type of stain (food stains, oil stains, mud stains, etc.). The analysis results are compared with the user's detergent information stored in the server's database, and if a suitable detergent is found, that information is generated as a suggestion to the user.

[0311] For example, if the server analyzes the image and determines that it is a coffee stain, it can use the user's registration information to determine that "laundry detergent" is suitable for coffee stains. The server then sends this information to the user's device. The suggestion includes a message such as "Use laundry detergent." If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so the user can purchase it directly.

[0312] The system also features an emotion engine, which uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize their emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Emotion recognition uses Microsoft Azure's Emotion API and IBM Watson. Emotion data is sent to a server, which then uses it to further customize the suggestions.

[0313] For example, if a user shows a confused expression about which detergent to choose, the emotion engine will recognize that emotion and the server will send a suggestion to the device, including specific and detailed instructions for use. In this way, the system helps users avoid confusion about which detergent to choose and how to use it.

[0314] Example prompts for generative AI models

[0315] Detergent Information

[0316] Name: Laundry detergent

[0317] Manufacturer: ABC Company

[0318] Uses: stains on clothing

[0319] Dirt information

[0320] Type: Coffee stains

[0321] Part: Carpet

[0322] User Emotion: Confused

[0323] Generate a proposal.

[0324] This invention not only proposes the optimum detergent based on the detergent information and stain analysis results of the user, but also recognizes the user's emotions and adjusts the proposal content based on them, thereby providing a more user-friendly system.

[0325] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0326] Step 1: Enter and register detergent information

[0327] Users use the application to register information about the detergent they own. They can either manually enter the information or use their device's camera to take a photo of the detergent label. When taking a photo of the label, the device uses image recognition technology to extract information such as the detergent's name, manufacturer, and purpose from the label. This information is sent to the server in JSON format.

[0328] Input: Detergent information manually entered by the user or detergent label image taken with a camera

[0329] Data processing: Extracting text information from images using image recognition technology

[0330] Output: Detergent information in JSON format

[0331] Specific behavior:

[0332] The user enters detergent information into the application.

[0333] The user takes a picture of the detergent label with the camera.

[0334] The device uses OpenCV to analyze the label image and extract text information.

[0335] The extracted information is converted into JSON format and sent to the server.

[0336] Step 2: Save your detergent information

[0337] The server parses the received detergent information and stores it in a database, where it is associated with the user's account and used to provide future recommendations.

[0338] Input: Detergent information in JSON format

[0339] Data processing: Parse and save to database

[0340] Output: Detergent information stored in the database

[0341] Specific behavior:

[0342] The server receives the data in JSON format.

[0343] The parser parses the JSON data and extracts the detergent information.

[0344] Generate and execute SQL queries to insert the extracted information into a database.

[0345] Step 3: Capture and pre-process stain images

[0346] When a user finds a stain that needs cleaning, they take a photo of it with their device's camera. Once the image is captured, the device performs pre-processing such as color adjustment and noise removal, which allows the server to analyze the image more accurately.

[0347] Input: Image of dirt

[0348] Data processing: color adjustment and noise removal

[0349] Output: Pre-processed stain image

[0350] Specific behavior:

[0351] The user takes a picture of the stain using the device's camera.

[0352] The device uses OpenCV to apply color adjustment and noise reduction filters.

[0353] Encode the preprocessed image in binary format.

[0354] Step 4: Submit and analyze the stain image

[0355] The pre-processed image is sent to a server, which uses an AI model to analyze the image and identify the type of stain (e.g., food stain, oil stain, mud stain, etc.).

[0356] Input: Preprocessed stain image

[0357] Data processing: Image analysis using AI models

[0358] Output: Analysis results including the type and characteristics of the dirt

[0359] Specific behavior:

[0360] The preprocessed image is sent to the server.

[0361] The server analyzes the image using a TensorFlow model.

[0362] The type and characteristics of the dirt are identified and saved as analysis results.

[0363] Step 5: Recommending the right detergent

[0364] The server compares the analysis results with the detergent information in the user's database to identify the optimal detergent, and then sends the identified detergent information to the user's device.

[0365] Input: Dirt analysis results, user detergent information

[0366] Data processing: matching and identifying the best detergent

[0367] Output: A message containing the suggestion

[0368] Specific behavior:

[0369] The server compares the stain analysis results with the user's detergent information.

[0370] Identify the best detergent and generate recommendations.

[0371] The proposal is encoded in JSON format and sent to the user's device.

[0372] Step 6: View Offer and Purchase Options

[0373] The terminal displays the received suggestions to the user, and if the suggested detergent is not in the user's inventory, the system provides a link to an e-commerce site so the user can purchase it directly.

[0374] Input: Proposal

[0375] Data processing: Converting data into a display format

[0376] Output: The offer and purchase link shown to the user

[0377] Specific behavior:

[0378] The device receives the proposal.

[0379] The suggestions are presented in a user-friendly format.

[0380] If appropriate, links to e-commerce sites are also provided.

[0381] Step 7: Emotion recognition and personalized suggestions

[0382] The device analyzes the user's facial expressions and voice to recognize their emotions, and the emotional data is sent to the server, which then uses this information to further customize the suggestions.

[0383] Input: User's facial expressions and voice

[0384] Data processing: sentiment analysis and data transmission

[0385] Output: Customized suggestions

[0386] Specific behavior:

[0387] The device uses a camera and microphone to capture facial expressions and voice.

[0388] Analyze emotions using an emotion engine (such as Microsoft Azure's Emotion API).

[0389] Emotion data is sent to the server.

[0390] The server regenerates the proposal content based on the received emotion data and transmits it to the terminal.

[0391] (Application example 2)

[0392] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0393] Conventional detergent recommendation systems require users to register their detergent information and analyze images of stains to suggest the optimal detergent. However, because they do not take the user's emotional state into account, the system is unable to make the recommendations easy for users to understand. Furthermore, customers may feel stressed or confused when receiving appropriate recommendations in physical stores, which prevents improved satisfaction. To solve these problems, a system is needed that recognizes the user's emotional state and customizes the recommendations based on that state.

[0394] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0395] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera of the terminal, means for analyzing the photographed image of the stains to identify the type of stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user, means for recognizing the emotional state of the user and customizing the suggestion content based on the emotion, means for recognizing the user's emotion and an image of the stain using smart glasses, and means for making detailed suggestions including detergent usage procedures based on the recognized emotional state. This enables appropriate and easy-to-understand suggestions according to the emotional state of the user.

[0396] "User" means an individual or entity that uses the system to receive detergent suggestions.

[0397] The "means for registering information about detergents owned" is a function that allows a user to input or scan information about detergents that the user owns into the system and store the information in a database.

[0398] A "terminal" is a device equipped with input devices such as a camera and a microphone, and is a computer or smartphone that executes the functions of the system.

[0399] The "means for photographing the dirt using a camera" is a method for capturing an image of the dirt using a camera mounted on the terminal.

[0400] The "means for analyzing the image of the dirt and identifying the type of dirt" is a function for analyzing the captured image of the dirt and identifying the type of dirt from the results.

[0401] The "means for suggesting the optimum detergent" is a function that recommends to the user the detergent that will most effectively remove the stain, based on the type of stain and information about the detergent the user owns.

[0402] "Means for providing the option to purchase suitable detergent from an e-commerce site" refers to a function that provides a link or information that allows the user to purchase the detergent online if the suggested detergent is not in the user's inventory.

[0403] The "means for recognizing emotional state" is a function that uses the device's camera and microphone to analyze emotions from the user's facial expressions and voice and obtain the results.

[0404] The "means for customizing the suggestions" is a function that adjusts detergent suggestions and usage procedures according to the recognized emotional state of the user, making them easier for the user to understand.

[0405] "Smart glasses" are wearable devices equipped with cameras and monitors, and are used to analyze the user's visual information and emotions.

[0406] The "means for providing detailed suggestions" is a function that provides the user with specific instructions and additional information on how to remove stains.

[0407] This invention is a system that registers detergent information owned by a user, analyzes images of stains, and recommends the most suitable detergent, and adds a function to recognize the emotional state of the user and customize the recommendations based on that. Specific embodiments for implementing this system are described below.

[0408] 1. Registering detergent information

[0409] First, the user registers information about the detergent they own in the system. This can be done by scanning the product's barcode or by manually entering the information. The user can also use the device's camera to take a photo of the detergent label and extract the necessary information using image recognition technology. The extracted information is sent to the server and stored in association with the user's account.

[0410] 2. Photograph the stain and send the image

[0411] Next, if the user finds a stain, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0412] 3. AI-based stain analysis

[0413] The server is equipped with a generative AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain and compares the results with the user's registered information stored in the server's database. If a suitable detergent is found, it is generated as a recommendation for the user.

[0414] 4. Suggestions and Recommendations

[0415] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0416] 5. Use of Emotion Engine

[0417] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state. Based on this emotion data, the server further customizes the suggestions.

[0418] Hardware and Software Description

[0419] Devices used include smartphones, tablets, and smart glasses, with Google Glass and Microsoft HoloLens being particularly suitable.

[0420] OpenCV is used for image processing, and Azure Emotion API and EmotionRecognizer library are used for emotion recognition.

[0421] A generative AI model is used to analyze the dirt, and when a prompt is sent to analyze the image, the data is uploaded to a server in the cloud.

[0422] Specific examples

[0423] For example, consider a scenario in which a store staff member uses smart glasses to interact with a customer in a brick-and-mortar store. If the customer asks with a confused expression, "I don't know which detergent to use," the camera and microphone in the smart glasses will recognize the customer's emotional state. If the store staff member takes a photo of the stain brought in by the customer and the system analyzes that the stain is a coffee stain, the system will make a suggestion including detailed instructions, such as, "Use this laundry detergent. Here are the instructions for use."

[0424] Prompt Sentence Examples

[0425] "Analyze this image to identify the type of dirt."

[0426] "If your customer is confused, show them the suggestion with detailed instructions."

[0427] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0428] Step 1:

[0429] Register detergent information owned by the user.

[0430] Users can scan the detergent's barcode using their device's camera or manually enter the information. The device then uses image recognition technology to analyze the barcode and extract the necessary information (name, manufacturer, and purpose). The extracted information is then sent to a server and stored in association with the user's account.

[0431] Input: Detergent barcode image or manually entered information.

[0432] Output: Detergent information stored in the user account.

[0433] Step 2:

[0434] Take a picture of the stain with your device's camera.

[0435] When a user finds a stain, they take a photo of it using their device's camera. The device then pre-processes the image, adjusting color and removing noise, and sends the processed image to the server.

[0436] Input: A photographed image of the dirt.

[0437] Output: Preprocessed image data.

[0438] Step 3:

[0439] The server analyzes the received image of the dirt.

[0440] The server uses the generative AI model to analyze the pre-processed image, which identifies the type of stain in the image (e.g., coffee stain, oil stain, etc.) and stores the identified stain type in a database.

[0441] Input: Preprocessed stain image.

[0442] Output: Identified soil type information.

[0443] Step 4:

[0444] The server recommends the most suitable detergent based on the user's detergent information.

[0445] The server compares the identified stain type with the detergent information of the user and suggests the most suitable detergent. The suggestion is sent to the terminal and displayed to the user.

[0446] Input: Identified stain type information, user detergent information.

[0447] Output: Recommendations for the best detergent.

[0448] Step 5:

[0449] Recognize the user's emotional state.

[0450] The device's camera and microphone are used to analyze the user's facial expressions and voice, and the emotion engine recognizes the user's emotional state (e.g., confusion, satisfaction, stress, etc.). The recognized emotion data is sent to the server.

[0451] Input: User's facial and voice data.

[0452] Output: Recognized emotional state data.

[0453] Step 6:

[0454] The server customizes the suggestions based on the emotion data.

[0455] The server receives the emotional state data and tailors the suggestions. For example, if the user is confused, the system generates suggestions that include detailed instructions and links to e-commerce sites. These customized suggestions are sent to the device and displayed to the user.

[0456] Input: Emotional state data, optimal detergent recommendation.

[0457] Output: A customized proposal.

[0458] Step 7:

[0459] Recognizing dirt and emotions using smart glasses.

[0460] Store staff use smart glasses to recognize customer emotions and images of dirt. The smart glasses' cameras and microphones capture the customer's facial expressions and voice and send the data to a server, which analyzes it and generates suggestions.

[0461] Input: Images of dirt captured by smart glasses and customer emotion data.

[0462] Output: Recognized stain types and emotion data.

[0463] Step 8:

[0464] The server will provide detailed suggestions.

[0465] Based on the recognized emotional state, the server generates detailed recommendations, including instructions for using the detergent, which are displayed to staff through smart glasses and explained to the customer.

[0466] Input: Recognized dirt type, emotional state data.

[0467] Output: Detailed proposal.

[0468] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0469] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0470] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0471] [Second embodiment]

[0472] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0473] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0474] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0475] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0476] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0477] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0478] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0479] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0480] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0481] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0482] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0483] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0484] This invention is a system that registers information about the detergents a user owns and suggests the most suitable detergent for stains photographed with a smartphone or other device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[0485] 1. Detergent Registration

[0486] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to the server and stored in association with the user's account.

[0487] Examples:

[0488] The user manually enters "Laundry Detergent."

[0489] When a user takes a photo of the "dishwashing detergent" label with the device's camera, the device automatically extracts the information and sends it to the server.

[0490] 2. Photograph the stain and send the image

[0491] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0492] Examples:

[0493] The user takes a picture of a stain on the carpet.

[0494] The device preprocesses the image and sends it to the server.

[0495] 3. AI-based stain analysis

[0496] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[0497] Examples:

[0498] The server analyzes the image and determines that it is a coffee stain.

[0499] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[0500] 4. Suggestions and Recommendations

[0501] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0502] Examples:

[0503] The server sends a suggestion to the device: "Use laundry detergent."

[0504] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[0505] In this way, the system utilizes the detergent information the user has, allowing them to efficiently select the optimal detergent to remove dirt. At each step, image recognition technology and AI analysis are used to minimize the user's effort.

[0506] The processing flow will be explained below.

[0507] Step 1:

[0508] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[0509] Step 2:

[0510] The device captures the label image and uses image recognition to extract the necessary information from the label.

[0511] Step 3:

[0512] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[0513] Step 4:

[0514] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[0515] Step 5:

[0516] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[0517] Step 6:

[0518] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[0519] Step 7:

[0520] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[0521] Step 8:

[0522] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[0523] As a specific example, if a user registers "dishwashing detergent" and takes a photo of a coffee stain on a carpet, the server will analyze the image and determine that "laundry detergent" is the best option. If the laundry detergent is not in stock, a purchase link will be provided.

[0524] Example 1

[0525] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] Conventional detergent selection systems lack the functionality to efficiently manage the detergent information owned by the user and suggest the detergent best suited to a specific stain. Furthermore, when users manually input the information about the detergents they own, it is highly likely to result in time-consuming work and input errors. Furthermore, if the suggested detergent is not in stock, the process of purchasing it can be cumbersome. Therefore, an efficient and accurate system that can solve these issues is needed.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0528] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera on the terminal, means for preprocessing the photographed image of the stain and transmitting it to the server, means for analyzing the photographed image of the stain to identify the type of stain, means for suggesting an optimal detergent based on information about the detergents owned by the user, and means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user. This makes it possible to efficiently manage information about detergents owned by the user, quickly suggest an optimal detergent for a specific stain, and easily perform the procedure for purchasing the necessary detergent.

[0529] A "user" is a person who registers detergent information, takes an image of the stain using a terminal, and receives recommendations for the most suitable detergent from the system.

[0530] A "terminal" is a device that has a camera function, takes an image of the stain, performs preprocessing, and transmits the image to the server.

[0531] The "server" is the central computer in the system that stores information about detergents registered by users, analyzes received images of stains, suggests the most suitable detergent, and also provides purchasing options.

[0532] "Means for registering detergent information" refers to a function that allows information such as the name, manufacturer, and purpose of the detergent the user owns to be entered into the application or obtained through image recognition, and then saved on the server.

[0533] The "means for photographing dirt" refers to a function that uses the camera of the terminal to take an image of the dirt that the user needs to clean.

[0534] "Preprocessing" refers to the provision of image processing technology to improve analysis accuracy by performing processes such as color correction and noise removal on captured images.

[0535] "Analysis means" refers to a function that uses an AI model installed on the server to identify the type of dirt in the captured image.

[0536] The "suggestion means" is a function that suggests the most suitable detergent from among the detergents the user owns based on the analyzed type of dirt.

[0537] "Means for providing purchase options" refers to a function that provides a link to an e-commerce site so that the appropriate detergent can be purchased directly if the suggested detergent is not in the user's inventory.

[0538] The "image recognition function" is a technology that extracts information such as the name, manufacturer, and purpose from the detergent label photographed with the device's camera.

[0539] An "AI model" is a model trained using machine learning technology to analyze photographed images of dirt and identify the type of dirt.

[0540] An "e-commerce site" is a website for purchasing goods over the Internet.

[0541] This invention is a system that registers information about the detergents a user owns and suggests the best detergent for stains photographed with a device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[0542] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device then uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to a server and stored in association with the user's account.

[0543] For example, if a user takes a photo of the label of a "laundry detergent" product with their device's camera, the device automatically extracts the information and sends it to the server. This information is then stored in the user's account, allowing them to add other detergents, such as "dish detergent," in the same way.

[0544] Next, if the user finds a stain that needs cleaning, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device, with color adjustment and noise removal performed. The processed image is then sent to the server. For example, if a user takes a photo of a stain on a carpet, the device pre-processes the image and sends it to the server.

[0545] The server has an AI model (e.g., a TensorFlow or PyTorch-based model) that analyzes the received image and identifies the stains in the image. The AI ​​identifies the type of stain (food stain, oil stain, mud stain, etc.) and compares the result with the user's registered information stored in the server's database. If an appropriate detergent is found, it generates a recommendation for the user. For example, if the server analyzes an image and determines that it is a coffee stain, the server will identify from the user's registered information that "laundry detergent" is suitable for coffee stains.

[0546] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so that the user can purchase it directly. For example, if the server sends the suggestion "Use laundry detergent" to the terminal and the user does not have "laundry detergent," a link to Amazon or other sites will also be displayed.

[0547] In this system, examples of prompts to be input to the generative AI model include:

[0548] "A user finds a coffee stain on their carpet and takes a photo of it with their device. The AI ​​should analyze the stain and suggest the best detergent to use. If the suggested detergent is not in stock, the AI ​​should also provide a link to purchase it."

[0549] This prompt allows the system to consistently analyze and make recommendations based on the input data for a specific stain.

[0550] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0551] Step 1:

[0552] Users register information about the detergents they own in the application.

[0553] Input: The user manually enters the name, manufacturer, and purpose of the detergent. Alternatively, the user can take a photo of the detergent label using the device's camera.

[0554] Data processing: In the case of manual input, the information entered in the form is verified and saved. In the case of label photography, the device uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, purpose) from the photographed label.

[0555] Output: Extracted detergent information (name, manufacturer, purpose).

[0556] Specific operation: The user manually enters "Laundry detergent." The user takes a photo of the "Dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[0557] Step 2:

[0558] The device sends the extracted detergent information to a server and stores it in the user's account.

[0559] Input: Extracted detergent information (name, manufacturer, purpose).

[0560] Data calculation: The device communicates with the server using a secure protocol such as HTTPS and transmits information.

[0561] Output: User's detergent information stored in the server's database.

[0562] Specific operation: The device sends the information of "dishwashing detergent," "ABC manufacturer," and "dishware" to the server, and the server stores it in the user's account.

[0563] Step 3:

[0564] When a user finds dirt that needs to be cleaned, they take a picture of the dirt with the device's camera.

[0565] Input: A dirty area (e.g., a carpet), and the device's camera to take a photo.

[0566] Data processing: Take an image of the dirt using the device's camera.

[0567] Output: The captured image of the stain.

[0568] Specific operation: The user takes a photo of a stain on the carpet using the device's camera.

[0569] Step 4:

[0570] The captured images are first pre-processed on the device and then sent to the server.

[0571] Input: A photographed image of the dirt.

[0572] Data processing: The terminal performs pre-processing such as color adjustment and noise removal.

[0573] Output: Preprocessed stain image.

[0574] Specific operation: The device performs color correction and noise reduction on the captured image, and then sends the preprocessed image to the server via HTTPS protocol.

[0575] Step 5:

[0576] The server uses an AI model to analyze the received image, which then identifies the stains in the image.

[0577] Input: Preprocessed stain images.

[0578] Data processing: An AI model (e.g., TensorFlow or PyTorch) installed on the server analyzes the image and identifies the type of dirt.

[0579] Output: The type of stain identified (e.g., coffee stain).

[0580] How it works: The server inputs the preprocessed image into the AI ​​model and identifies it as a coffee stain.

[0581] Step 6:

[0582] The server compares the analysis results with the user's registered information and selects the most suitable detergent.

[0583] Input: Identified stain type, user detergent information.

[0584] Data calculation: Matches the type of dirt with the intended use of the detergent the user owns to identify the most suitable detergent.

[0585] Output: Information on the best detergent.

[0586] Specific behavior: The server determines that "laundry detergent" is best for removing coffee stains.

[0587] Step 7:

[0588] The server transmits the suggested detergent information to the terminal and notifies the user.

[0589] Input: Information on the best detergent.

[0590] Data calculation: Sending data for proposal generation and notification.

[0591] Output: Proposal.

[0592] Specific operation: The server sends a suggestion to the device saying "Use laundry detergent."

[0593] Step 8:

[0594] If the user does not have the suggested detergent, the system will provide a link to an e-commerce site where they can purchase it directly.

[0595] Input: detergent inventory information, e-commerce site link information.

[0596] Data calculation: Checking stock information and generating links.

[0597] Output: Purchase link.

[0598] Specific operation: If the server does not have "laundry detergent," it generates a link to Amazon or another site and sends it to the device.

[0599] (Application example 1)

[0600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] Today's consumers are often confused by the wide variety of cleaning methods available, making it difficult to choose the right cleaning agent. Furthermore, when selecting products in physical stores, there is a lack of a system that instantly suggests the best cleaning agent for each type of dirt and purpose. This creates the risk of using the wrong cleaning agent and damaging the product.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0603] In this invention, the server includes: means for registering information about detergents owned by the user; means for taking pictures of stains using a camera on the terminal; means for analyzing the captured images of stains and using a generative AI model to identify the type of stain; means for suggesting an optimal detergent based on the information about the detergents owned by the user; means for providing an option to purchase an appropriate detergent from an online sales platform if the suggested detergent is not in the user's inventory; and means for acquiring product information by scanning product barcodes and suggesting the optimal detergent based on the product information. This allows users to instantly find the optimal detergent and apply an appropriate cleaning method when selecting products in a physical store.

[0604] "User" refers to an individual or corporation that uses this system.

[0605] "Cleaning agents" refers to a general range of chemical products used to remove dirt from users' hands.

[0606] "Means for registering information" refers to the function of inputting and saving data on the cleaning agents owned by the user into the system.

[0607] "Device" refers to a smartphone, tablet, or other portable electronic device.

[0608] "Camera" refers to an image capturing device built into the terminal.

[0609] A "generative AI model" refers to an algorithm that uses artificial intelligence to analyze images of dirt and identify its features.

[0610] "Means for suggesting the most suitable cleaning agent" refers to the function of selecting and presenting the appropriate cleaning agent based on the user's registered information and analysis results.

[0611] "Online sales platform" refers to an e-commerce site that allows products to be purchased via the Internet.

[0612] "Barcode scanning function" refers to the function of using a camera to read the barcode attached to the product.

[0613] "Means for analyzing images" refers to technology for processing captured images and identifying their contents.

[0614] "Pre-processing" refers to the initial processing steps performed to improve the quality of a captured image.

[0615] "Server" refers to a computer system for processing, storing, and communicating data.

[0616] This invention is a system that suggests optimal detergents when a user selects products in a physical store. The system communicates between the user, terminal, and server to support the effective use of detergents.

[0617] 1. Cleaning Agent Registration

[0618] First, the user registers information about the detergent they own in the device's application. This information can be obtained by manually entering it or by taking a photo of the detergent label with the device's camera and using image recognition. The device extracts the necessary information (name, manufacturer, and purpose) from the photo of the label and sends it to the server. The server stores this information, linking it to each user's account.

[0619] Specific examples

[0620] The user manually enters "clothing detergent."

[0621] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information and sends it to the server.

[0622] 2. Photograph the stain and send the image

[0623] When a user finds a stain on a product, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device (color adjustment, noise removal, etc.), and then the processed image is sent to the server.

[0624] Specific examples

[0625] The user takes a picture of a stain on the carpet.

[0626] The device preprocesses the image and sends it to the server.

[0627] 3. AI-based stain analysis

[0628] The server analyzes the received image using a generative AI model to identify the type of dirt. The AI ​​identifies the type of dirt (food stains, oil stains, mud stains, etc.) and compares the results with the user's registered information stored in the server's database. If an appropriate cleaning agent is found, it will be recommended to the user.

[0629] Specific examples

[0630] The server analyzes the image and determines that it is a coffee stain.

[0631] The server determines from the user's registration information that "clothing detergent" is suitable for removing coffee stains.

[0632] 4. Obtain product information by scanning barcodes

[0633] Users can also obtain product information by scanning the product barcode with the device's camera. The barcode scanning function allows the system to recommend the most suitable detergent based on the product information.

[0634] Specific examples

[0635] A user scans the barcode of the clothing item they want to purchase in a brick-and-mortar store.

[0636] The terminal sends the barcode information to the server, which then identifies the most suitable cleaning agent.

[0637] 5. Suggestions and Recommendations

[0638] Based on the analysis results, the server will send the suggested cleaning product information to the terminal. If the user does not have the suggested cleaning product in stock, the system will also provide a link to an online sales platform for direct purchase.

[0639] Specific examples

[0640] The server sends a suggestion to the terminal saying "Use laundry detergent."

[0641] The terminal will display this offer to the user and, if the user does not have "clothes detergent", will also display a link to an online sales platform.

[0642] Prompt Sentence Examples

[0643] "Identify the type of stain in this image. Types of stains include coffee stains, oil stains, mud stains, etc."

[0644] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0645] Step 1:

[0646] Users register information about the detergent they own on their device. To do this, they can either manually enter the information or take a photo of the detergent label with the device's camera. The captured image of the label is analyzed on the device using image recognition technology, and the necessary information (name, manufacturer, purpose, etc.) is automatically extracted. The extracted information is sent to a server and saved, linked to the user's account.

[0647] Input: Manually enter cleaning agent information or label image

[0648] Data processing: Information extraction using image recognition

[0649] Output: Save cleaning agent information to database

[0650] Step 2:

[0651] When a user finds a product or stain, they take a photo of the stain with their device's camera. The captured image undergoes pre-processing such as color adjustment and noise removal on the device, and then the processed image is sent to the server.

[0652] Input: Image of dirt

[0653] Data processing: Image preprocessing (color adjustment, noise removal)

[0654] Output: Processed images sent to server

[0655] Step 3:

[0656] The server analyzes the received image using a generative AI model to identify the type of stain. The generative AI model uses prompts to analyze the characteristics of the stain and identify the specific stain type (e.g., coffee stain, oil stain, mud stain, etc.). Once the stain type is identified, the information is stored in the server's database.

[0657] Input: Processed image

[0658] Data Computation: Identifying Contamination with Generative AI Models

[0659] Output: Server save of dirt type information

[0660] Step 4:

[0661] The server compares the information on the type of dirt with the information on the detergent registered by the user to identify the most suitable detergent. If an appropriate detergent is identified, the recommendation is sent to the device.

[0662] Input: Dirt type information, user-registered cleaning agent information

[0663] Data calculations: collating information to identify optimal cleaning products

[0664] Output: Sends the proposal to the device.

[0665] Step 5:

[0666] The user can review the suggestions on their device and select the appropriate cleaning product. If the user does not have the suggested cleaning product in stock, the server will also provide a link to an online sales platform so that the user can purchase it directly.

[0667] Enter: View Suggestions

[0668] Data calculation: inventory check and link generation

[0669] Output: Display of online sales platform link

[0670] Step 6:

[0671] When a user selects a product in a physical store, they scan the product's barcode with the device's camera, and the device sends the barcode information to the server, which then identifies the appropriate cleaning product and sends a recommendation to the device.

[0672] Input: Product barcode

[0673] Data calculation: Analysis of barcode data and identification of cleaning agents

[0674] Output: Sends the proposal to the device.

[0675] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0676] This invention is a system that suggests detergents to properly remove dirt, and adds a function to recognize the user's emotions and customize the suggestions based on those emotions. This system communicates between the user, the terminal, and the server, and by combining an emotion engine, makes suggestions that are more appropriate and effective for the user.

[0677] 1. Detergent Registration

[0678] First, users register information about their detergent in the application. This involves manually entering the information or taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the label. This information is then sent to the server and stored in association with the user's account.

[0679] Examples:

[0680] The user manually enters "Laundry Detergent."

[0681] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[0682] 2. Photograph the stain and send the image

[0683] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0684] Examples:

[0685] The user takes a picture of a stain on the carpet.

[0686] The device preprocesses the image and sends it to the server.

[0687] 3. AI-based stain analysis

[0688] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[0689] Examples:

[0690] The server analyzes the image and determines that it is a coffee stain.

[0691] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[0692] 4. Suggestions and Recommendations

[0693] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0694] Examples:

[0695] The server sends a suggestion to the device: "Use laundry detergent."

[0696] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[0697] 5. Use of Emotion Engine

[0698] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Based on this emotion data, the server further customizes the suggestions.

[0699] Examples:

[0700] If the user shows a confused look on their face about which detergent to choose, the emotion engine will recognize that emotion and send it to the server.

[0701] The server receives the emotion data and sends suggestions to the terminal, including detailed usage instructions that are easy for the user to understand.

[0702] In this way, the present invention, which combines an emotion engine, not only suggests the most suitable detergent based on the detergent information the user has and the results of stain analysis, but also recognizes the user's emotions and adjusts the suggestions based on them, providing a more user-friendly system.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[0706] Step 2:

[0707] The device captures the label image and uses image recognition to extract the necessary information from the label.

[0708] Step 3:

[0709] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[0710] Step 4:

[0711] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[0712] Step 5:

[0713] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[0714] Step 6:

[0715] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[0716] Step 7:

[0717] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[0718] Step 8:

[0719] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[0720] Step 9:

[0721] The user can enable the emotion engine in the settings of the application.

[0722] Step 10:

[0723] The device acquires the user's emotional data using a camera and microphone. The device captures the user's facial expressions with the camera and records their voice with the microphone.

[0724] Step 11:

[0725] The device analyzes the emotion data to determine the user's current emotional state, a process that identifies emotions such as stress, frustration, and satisfaction.

[0726] Step 12:

[0727] The device sends the analyzed emotional data to the server, which then customizes the detergent recommendations based on the emotional data received.

[0728] Step 13:

[0729] The server tailors its suggestions depending on the user's emotional state, for example, providing suggestions with detailed instructions if the user is confused.

[0730] Step 14:

[0731] The terminal displays the customized suggestions received from the server to the user, allowing the user to receive appropriate suggestions that match their emotions.

[0732] For example, if a user registers "clothing detergent" and takes a photo of a coffee stain on a carpet, the server will suggest "clothing detergent." If the user shows a confused expression, the device will recognize the emotion, and the server will provide detailed instructions for use, which the device will then display.

[0733] Example 2

[0734] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0735] Conventional detergent recommendation systems simply make recommendations based on the type of dirt and the detergent information registered by the user, without taking the user's feelings into consideration. This has led to issues such as users being confused by the recommendations or not knowing how to properly operate the system. Furthermore, when multiple detergents are applicable to a particular dirt, the system is unable to provide appropriate advice to help the user make the optimal selection.

[0736] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using the camera of the terminal, means for identifying the type of stain by analyzing the photographed image of the stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in the user's inventory, and means for recognizing the user's emotional state and customizing the suggestions based on the emotion. This makes it easier to deal with cases where the user is confused about which detergent to select or does not know how to operate the system, and makes it possible to provide a user-friendly system.

[0737] "User" refers to an individual or organization that accesses the system and registers detergent information, photographs stains, receives suggestions, etc.

[0738] A "terminal" refers to a computer device that is equipped with devices such as a camera and microphone and that is operated by a user. Examples include smartphones, tablets, and PCs.

[0739] "Server" refers to a computer system that receives information sent from a terminal and analyzes, stores, and provides the data.

[0740] "Detergent information" refers to detailed information such as the detergent's name, manufacturer, and purpose. This data indicates the characteristics of the detergent the user owns.

[0741] "Camera" refers to a device for capturing images, including those built into a device and those connected externally.

[0742] "Soil" refers to impurities or stains on the surface of an object that needs to be cleaned, such as food stains, mud stains, or oil stains.

[0743] "Image recognition technology" refers to technology that automatically extracts and analyzes specific information from captured images, including character recognition and object identification.

[0744] "Preprocessing" refers to the adjustment and processing techniques used to process images and data before analyzing them, including color adjustment and noise removal.

[0745] An "AI model" refers to a software model that uses artificial intelligence technology to analyze data and make predictions. Many of these models use machine learning algorithms.

[0746] "Emotion engine" refers to technology that uses the device's camera and microphone to analyze and recognize emotions from the user's facial expressions and voice.

[0747] An "e-commerce site" is a website that sells and purchases goods and services over the internet. Amazon is an example of this.

[0748] "Suggestion" refers to information that indicates the optimal detergent and its usage method to the user based on the analysis results.

[0749] "Customization" refers to adjusting and changing the content of suggestions according to the user's characteristics and situation.

[0750] The present invention provides a detergent recommendation system that can recognize a user's emotions and customize the content of recommendations based on those emotions. A specific embodiment of this system will be described below.

[0751] First, the user registers information about the detergent they own in the application. This process involves manual input and taking a photo of the detergent label using the device's camera. The device typically used here is a camera-equipped device such as a smartphone or tablet. Software such as TensorFlow and OpenCV is used for image recognition technology.

[0752] For example, there is a case where the user manually enters "laundry detergent" or a case where the user takes a picture of the label of "dishwashing detergent" with the device's camera and the information is automatically extracted. The information extracted from the label (name, manufacturer, purpose) is sent from the device over the Internet to a server. The server associates this information with the user's account and stores it in a database.

[0753] Next, if the user finds a stain that needs cleaning, they take a photo of it with the device's camera. The captured image undergoes preprocessing, such as color adjustment and noise removal, before being sent to the server. Image processing libraries such as OpenCV are used for preprocessing.

[0754] The server uses an AI model to analyze the received image. Common AI models used here include TensorFlow and PyTorch. The AI ​​identifies stains in the image and determines the type of stain (food stains, oil stains, mud stains, etc.). The analysis results are compared with the user's detergent information stored in the server's database, and if a suitable detergent is found, that information is generated as a suggestion to the user.

[0755] For example, if the server analyzes the image and determines that it is a coffee stain, it can use the user's registration information to determine that "laundry detergent" is suitable for coffee stains. The server then sends this information to the user's device. The suggestion includes a message such as "Use laundry detergent." If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so the user can purchase it directly.

[0756] The system also features an emotion engine, which uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize their emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Emotion recognition uses Microsoft Azure's Emotion API and IBM Watson. Emotion data is sent to a server, which then uses it to further customize the suggestions.

[0757] For example, if a user shows a confused expression about which detergent to choose, the emotion engine will recognize that emotion and the server will send a suggestion to the device, including specific and detailed instructions for use. In this way, the system helps users avoid confusion about which detergent to choose and how to use it.

[0758] Example prompts for generative AI models

[0759] Detergent Information

[0760] Name: Laundry detergent

[0761] Manufacturer: ABC Company

[0762] Uses: stains on clothing

[0763] Dirt information

[0764] Type: Coffee stains

[0765] Part: Carpet

[0766] User Emotion: Confused

[0767] Generate a proposal.

[0768] This invention not only proposes the optimum detergent based on the detergent information and stain analysis results of the user, but also recognizes the user's emotions and adjusts the proposal content based on them, thereby providing a more user-friendly system.

[0769] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0770] Step 1: Enter and register detergent information

[0771] Users use the application to register information about the detergent they own. They can either manually enter the information or use their device's camera to take a photo of the detergent label. When taking a photo of the label, the device uses image recognition technology to extract information such as the detergent's name, manufacturer, and purpose from the label. This information is sent to the server in JSON format.

[0772] Input: Detergent information manually entered by the user or detergent label image taken with a camera

[0773] Data processing: Extracting text information from images using image recognition technology

[0774] Output: Detergent information in JSON format

[0775] Specific behavior:

[0776] The user enters detergent information into the application.

[0777] The user takes a picture of the detergent label with the camera.

[0778] The device uses OpenCV to analyze the label image and extract text information.

[0779] The extracted information is converted into JSON format and sent to the server.

[0780] Step 2: Save your detergent information

[0781] The server parses the received detergent information and stores it in a database, where it is associated with the user's account and used to provide future recommendations.

[0782] Input: Detergent information in JSON format

[0783] Data processing: Parse and save to database

[0784] Output: Detergent information stored in the database

[0785] Specific behavior:

[0786] The server receives the data in JSON format.

[0787] The parser parses the JSON data and extracts the detergent information.

[0788] Generate and execute SQL queries to insert the extracted information into a database.

[0789] Step 3: Capture and pre-process stain images

[0790] When a user finds a stain that needs cleaning, they take a photo of it with their device's camera. Once the image is captured, the device performs pre-processing such as color adjustment and noise removal, which allows the server to analyze the image more accurately.

[0791] Input: Image of dirt

[0792] Data processing: color adjustment and noise removal

[0793] Output: Pre-processed stain image

[0794] Specific behavior:

[0795] The user takes a picture of the stain using the device's camera.

[0796] The device uses OpenCV to apply color adjustment and noise reduction filters.

[0797] Encode the preprocessed image in binary format.

[0798] Step 4: Submit and analyze the stain image

[0799] The pre-processed image is sent to a server, which uses an AI model to analyze the image and identify the type of stain (e.g., food stain, oil stain, mud stain, etc.).

[0800] Input: Preprocessed stain image

[0801] Data processing: Image analysis using AI models

[0802] Output: Analysis results including the type and characteristics of the dirt

[0803] Specific behavior:

[0804] The preprocessed image is sent to the server.

[0805] The server analyzes the image using a TensorFlow model.

[0806] The type and characteristics of the dirt are identified and saved as analysis results.

[0807] Step 5: Recommending the right detergent

[0808] The server compares the analysis results with the detergent information in the user's database to identify the optimal detergent, and then sends the identified detergent information to the user's device.

[0809] Input: Dirt analysis results, user detergent information

[0810] Data processing: matching and identifying the best detergent

[0811] Output: A message containing the suggestion

[0812] Specific behavior:

[0813] The server compares the stain analysis results with the user's detergent information.

[0814] Identify the best detergent and generate recommendations.

[0815] The proposal is encoded in JSON format and sent to the user's device.

[0816] Step 6: View Offer and Purchase Options

[0817] The terminal displays the received suggestions to the user, and if the suggested detergent is not in the user's inventory, the system provides a link to an e-commerce site so the user can purchase it directly.

[0818] Input: Proposal

[0819] Data processing: Converting data into a display format

[0820] Output: The offer and purchase link shown to the user

[0821] Specific behavior:

[0822] The device receives the proposal.

[0823] The suggestions are presented in a user-friendly format.

[0824] If appropriate, links to e-commerce sites are also provided.

[0825] Step 7: Emotion recognition and personalized suggestions

[0826] The device analyzes the user's facial expressions and voice to recognize their emotions, and the emotional data is sent to the server, which then uses this information to further customize the suggestions.

[0827] Input: User's facial expressions and voice

[0828] Data processing: sentiment analysis and data transmission

[0829] Output: Customized suggestions

[0830] Specific behavior:

[0831] The device uses a camera and microphone to capture facial expressions and voice.

[0832] Analyze emotions using an emotion engine (such as Microsoft Azure's Emotion API).

[0833] Emotion data is sent to the server.

[0834] The server regenerates the proposal content based on the received emotion data and transmits it to the terminal.

[0835] (Application example 2)

[0836] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0837] Conventional detergent recommendation systems require users to register their detergent information and analyze images of stains to suggest the optimal detergent. However, because they do not take the user's emotional state into account, the system is unable to make the recommendations easy for users to understand. Furthermore, customers may feel stressed or confused when receiving appropriate recommendations in physical stores, which prevents improved satisfaction. To solve these problems, a system is needed that recognizes the user's emotional state and customizes the recommendations based on that state.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0839] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera of the terminal, means for analyzing the photographed image of the stains to identify the type of stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user, means for recognizing the emotional state of the user and customizing the suggestion content based on the emotion, means for recognizing the user's emotion and an image of the stain using smart glasses, and means for making detailed suggestions including detergent usage procedures based on the recognized emotional state. This enables appropriate and easy-to-understand suggestions according to the emotional state of the user.

[0840] "User" means an individual or entity that uses the system to receive detergent suggestions.

[0841] The "means for registering information about detergents owned" is a function that allows a user to input or scan information about detergents that the user owns into the system and store the information in a database.

[0842] A "terminal" is a device equipped with input devices such as a camera and a microphone, and is a computer or smartphone that executes the functions of the system.

[0843] The "means for photographing the dirt using a camera" is a method for capturing an image of the dirt using a camera mounted on the terminal.

[0844] The "means for analyzing the image of the dirt and identifying the type of dirt" is a function for analyzing the captured image of the dirt and identifying the type of dirt from the results.

[0845] The "means for suggesting the optimum detergent" is a function that recommends to the user the detergent that will most effectively remove the stain, based on the type of stain and information about the detergent the user owns.

[0846] "Means for providing the option to purchase suitable detergent from an e-commerce site" refers to a function that provides a link or information that allows the user to purchase the detergent online if the suggested detergent is not in the user's inventory.

[0847] The "means for recognizing emotional state" is a function that uses the device's camera and microphone to analyze emotions from the user's facial expressions and voice and obtain the results.

[0848] The "means for customizing the suggestions" is a function that adjusts detergent suggestions and usage procedures according to the recognized emotional state of the user, making them easier for the user to understand.

[0849] "Smart glasses" are wearable devices equipped with cameras and monitors, and are used to analyze the user's visual information and emotions.

[0850] The "means for providing detailed suggestions" is a function that provides the user with specific instructions and additional information on how to remove stains.

[0851] This invention is a system that registers detergent information owned by a user, analyzes images of stains, and recommends the most suitable detergent, and adds a function to recognize the emotional state of the user and customize the recommendations based on that. Specific embodiments for implementing this system are described below.

[0852] 1. Registering detergent information

[0853] First, the user registers information about the detergent they own in the system. This can be done by scanning the product's barcode or by manually entering the information. The user can also use the device's camera to take a photo of the detergent label and extract the necessary information using image recognition technology. The extracted information is sent to the server and stored in association with the user's account.

[0854] 2. Photograph the stain and send the image

[0855] Next, if the user finds a stain, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0856] 3. AI-based stain analysis

[0857] The server is equipped with a generative AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain and compares the results with the user's registered information stored in the server's database. If a suitable detergent is found, it is generated as a recommendation for the user.

[0858] 4. Suggestions and Recommendations

[0859] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0860] 5. Use of Emotion Engine

[0861] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state. Based on this emotion data, the server further customizes the suggestions.

[0862] Hardware and Software Description

[0863] Devices used include smartphones, tablets, and smart glasses, with Google Glass and Microsoft HoloLens being particularly suitable.

[0864] OpenCV is used for image processing, and Azure Emotion API and EmotionRecognizer library are used for emotion recognition.

[0865] A generative AI model is used to analyze the dirt, and when a prompt is sent to analyze the image, the data is uploaded to a server in the cloud.

[0866] Specific examples

[0867] For example, consider a scenario in which a store staff member uses smart glasses to interact with a customer in a brick-and-mortar store. If the customer asks with a confused expression, "I don't know which detergent to use," the camera and microphone in the smart glasses will recognize the customer's emotional state. If the store staff member takes a photo of the stain brought in by the customer and the system analyzes that the stain is a coffee stain, the system will make a suggestion including detailed instructions, such as, "Use this laundry detergent. Here are the instructions for use."

[0868] Prompt Sentence Examples

[0869] "Analyze this image to identify the type of dirt."

[0870] "If your customer is confused, show them the suggestion with detailed instructions."

[0871] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0872] Step 1:

[0873] Register detergent information owned by the user.

[0874] Users can scan the detergent's barcode using their device's camera or manually enter the information. The device then uses image recognition technology to analyze the barcode and extract the necessary information (name, manufacturer, and purpose). The extracted information is then sent to a server and stored in association with the user's account.

[0875] Input: Detergent barcode image or manually entered information.

[0876] Output: Detergent information stored in the user account.

[0877] Step 2:

[0878] Take a picture of the stain with your device's camera.

[0879] When a user finds a stain, they take a photo of it using their device's camera. The device then pre-processes the image, adjusting color and removing noise, and sends the processed image to the server.

[0880] Input: A photographed image of the dirt.

[0881] Output: Preprocessed image data.

[0882] Step 3:

[0883] The server analyzes the received image of the dirt.

[0884] The server uses the generative AI model to analyze the pre-processed image, which identifies the type of stain in the image (e.g., coffee stain, oil stain, etc.) and stores the identified stain type in a database.

[0885] Input: Preprocessed stain image.

[0886] Output: Identified soil type information.

[0887] Step 4:

[0888] The server recommends the most suitable detergent based on the user's detergent information.

[0889] The server compares the identified stain type with the detergent information of the user and suggests the most suitable detergent. The suggestion is sent to the terminal and displayed to the user.

[0890] Input: Identified stain type information, user detergent information.

[0891] Output: Recommendations for the best detergent.

[0892] Step 5:

[0893] Recognize the user's emotional state.

[0894] The device's camera and microphone are used to analyze the user's facial expressions and voice, and the emotion engine recognizes the user's emotional state (e.g., confusion, satisfaction, stress, etc.). The recognized emotion data is sent to the server.

[0895] Input: User's facial and voice data.

[0896] Output: Recognized emotional state data.

[0897] Step 6:

[0898] The server customizes the suggestions based on the emotion data.

[0899] The server receives the emotional state data and tailors the suggestions. For example, if the user is confused, the system generates suggestions that include detailed instructions and links to e-commerce sites. These customized suggestions are sent to the device and displayed to the user.

[0900] Input: Emotional state data, optimal detergent recommendation.

[0901] Output: A customized proposal.

[0902] Step 7:

[0903] Recognizing dirt and emotions using smart glasses.

[0904] Store staff use smart glasses to recognize customer emotions and images of dirt. The smart glasses' cameras and microphones capture the customer's facial expressions and voice and send the data to a server, which analyzes it and generates suggestions.

[0905] Input: Images of dirt captured by smart glasses and customer emotion data.

[0906] Output: Recognized stain types and emotion data.

[0907] Step 8:

[0908] The server will provide detailed suggestions.

[0909] Based on the recognized emotional state, the server generates detailed recommendations, including instructions for using the detergent, which are displayed to staff through smart glasses and explained to the customer.

[0910] Input: Recognized dirt type, emotional state data.

[0911] Output: Detailed proposal.

[0912] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0913] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0914] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0915] [Third embodiment]

[0916] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0917] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0918] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0919] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0920] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0921] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0922] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0923] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0924] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0925] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0926] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0927] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0928] This invention is a system that registers information about the detergents a user owns and suggests the most suitable detergent for stains photographed with a smartphone or other device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[0929] 1. Detergent Registration

[0930] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to the server and stored in association with the user's account.

[0931] Examples:

[0932] The user manually enters "Laundry Detergent."

[0933] When a user takes a photo of the "dishwashing detergent" label with the device's camera, the device automatically extracts the information and sends it to the server.

[0934] 2. Photograph the stain and send the image

[0935] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[0936] Examples:

[0937] The user takes a picture of a stain on the carpet.

[0938] The device preprocesses the image and sends it to the server.

[0939] 3. AI-based stain analysis

[0940] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[0941] Examples:

[0942] The server analyzes the image and determines that it is a coffee stain.

[0943] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[0944] 4. Suggestions and Recommendations

[0945] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[0946] Examples:

[0947] The server sends a suggestion to the device: "Use laundry detergent."

[0948] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[0949] In this way, the system utilizes the detergent information the user has, allowing them to efficiently select the optimal detergent to remove dirt. At each step, image recognition technology and AI analysis are used to minimize the user's effort.

[0950] The processing flow will be explained below.

[0951] Step 1:

[0952] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[0953] Step 2:

[0954] The device captures the label image and uses image recognition to extract the necessary information from the label.

[0955] Step 3:

[0956] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[0957] Step 4:

[0958] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[0959] Step 5:

[0960] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[0961] Step 6:

[0962] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[0963] Step 7:

[0964] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[0965] Step 8:

[0966] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[0967] As a specific example, if a user registers "dishwashing detergent" and takes a photo of a coffee stain on a carpet, the server will analyze the image and determine that "laundry detergent" is the best option. If the laundry detergent is not in stock, a purchase link will be provided.

[0968] Example 1

[0969] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0970] Conventional detergent selection systems lack the functionality to efficiently manage the detergent information owned by the user and suggest the detergent best suited to a specific stain. Furthermore, when users manually input the information about the detergents they own, it is highly likely to result in time-consuming work and input errors. Furthermore, if the suggested detergent is not in stock, the process of purchasing it can be cumbersome. Therefore, an efficient and accurate system that can solve these issues is needed.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0972] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera on the terminal, means for preprocessing the photographed image of the stain and transmitting it to the server, means for analyzing the photographed image of the stain to identify the type of stain, means for suggesting an optimal detergent based on information about the detergents owned by the user, and means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user. This makes it possible to efficiently manage information about detergents owned by the user, quickly suggest an optimal detergent for a specific stain, and easily perform the procedure for purchasing the necessary detergent.

[0973] A "user" is a person who registers detergent information, takes an image of the stain using a terminal, and receives recommendations for the most suitable detergent from the system.

[0974] A "terminal" is a device that has a camera function, takes an image of the stain, performs preprocessing, and transmits the image to the server.

[0975] The "server" is the central computer in the system that stores information about detergents registered by users, analyzes received images of stains, suggests the most suitable detergent, and also provides purchasing options.

[0976] "Means for registering detergent information" refers to a function that allows information such as the name, manufacturer, and purpose of the detergent the user owns to be entered into the application or obtained through image recognition, and then saved on the server.

[0977] The "means for photographing dirt" refers to a function that uses the camera of the terminal to take an image of the dirt that the user needs to clean.

[0978] "Preprocessing" refers to the provision of image processing technology to improve analysis accuracy by performing processes such as color correction and noise removal on captured images.

[0979] "Analysis means" refers to a function that uses an AI model installed on the server to identify the type of dirt in the captured image.

[0980] The "suggestion means" is a function that suggests the most suitable detergent from among the detergents the user owns based on the analyzed type of dirt.

[0981] "Means for providing purchase options" refers to a function that provides a link to an e-commerce site so that the appropriate detergent can be purchased directly if the suggested detergent is not in the user's inventory.

[0982] The "image recognition function" is a technology that extracts information such as the name, manufacturer, and purpose from the detergent label photographed with the device's camera.

[0983] An "AI model" is a model trained using machine learning technology to analyze photographed images of dirt and identify the type of dirt.

[0984] An "e-commerce site" is a website for purchasing goods over the Internet.

[0985] This invention is a system that registers information about the detergents a user owns and suggests the best detergent for stains photographed with a device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[0986] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device then uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to a server and stored in association with the user's account.

[0987] For example, if a user takes a photo of the label of a "laundry detergent" product with their device's camera, the device automatically extracts the information and sends it to the server. This information is then stored in the user's account, allowing them to add other detergents, such as "dish detergent," in the same way.

[0988] Next, if the user finds a stain that needs cleaning, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device, with color adjustment and noise removal performed. The processed image is then sent to the server. For example, if a user takes a photo of a stain on a carpet, the device pre-processes the image and sends it to the server.

[0989] The server has an AI model (e.g., a TensorFlow or PyTorch-based model) that analyzes the received image and identifies the stains in the image. The AI ​​identifies the type of stain (food stain, oil stain, mud stain, etc.) and compares the result with the user's registered information stored in the server's database. If an appropriate detergent is found, it generates a recommendation for the user. For example, if the server analyzes an image and determines that it is a coffee stain, the server will identify from the user's registered information that "laundry detergent" is suitable for coffee stains.

[0990] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so that the user can purchase it directly. For example, if the server sends the suggestion "Use laundry detergent" to the terminal and the user does not have "laundry detergent," a link to Amazon or other sites will also be displayed.

[0991] In this system, examples of prompts to be input to the generative AI model include:

[0992] "A user finds a coffee stain on their carpet and takes a photo of it with their device. The AI ​​should analyze the stain and suggest the best detergent to use. If the suggested detergent is not in stock, the AI ​​should also provide a link to purchase it."

[0993] This prompt allows the system to consistently analyze and make recommendations based on the input data for a specific stain.

[0994] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0995] Step 1:

[0996] Users register information about the detergents they own in the application.

[0997] Input: The user manually enters the name, manufacturer, and purpose of the detergent. Alternatively, the user can take a photo of the detergent label using the device's camera.

[0998] Data processing: In the case of manual input, the information entered in the form is verified and saved. In the case of label photography, the device uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, purpose) from the photographed label.

[0999] Output: Extracted detergent information (name, manufacturer, purpose).

[1000] Specific operation: The user manually enters "Laundry detergent." The user takes a photo of the "Dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[1001] Step 2:

[1002] The device sends the extracted detergent information to a server and stores it in the user's account.

[1003] Input: Extracted detergent information (name, manufacturer, purpose).

[1004] Data calculation: The device communicates with the server using a secure protocol such as HTTPS and transmits information.

[1005] Output: User's detergent information stored in the server's database.

[1006] Specific operation: The device sends the information of "dishwashing detergent," "ABC manufacturer," and "dishware" to the server, and the server stores it in the user's account.

[1007] Step 3:

[1008] When a user finds dirt that needs to be cleaned, they take a picture of the dirt with the device's camera.

[1009] Input: A dirty area (e.g., a carpet), and the device's camera to take a photo.

[1010] Data processing: Take an image of the dirt using the device's camera.

[1011] Output: The captured image of the stain.

[1012] Specific operation: The user takes a photo of a stain on the carpet using the device's camera.

[1013] Step 4:

[1014] The captured images are first pre-processed on the device and then sent to the server.

[1015] Input: A photographed image of the dirt.

[1016] Data processing: The terminal performs pre-processing such as color adjustment and noise removal.

[1017] Output: Preprocessed stain image.

[1018] Specific operation: The device performs color correction and noise reduction on the captured image, and then sends the preprocessed image to the server via HTTPS protocol.

[1019] Step 5:

[1020] The server uses an AI model to analyze the received image, which then identifies the stains in the image.

[1021] Input: Preprocessed stain images.

[1022] Data processing: An AI model (e.g., TensorFlow or PyTorch) installed on the server analyzes the image and identifies the type of dirt.

[1023] Output: The type of stain identified (e.g., coffee stain).

[1024] How it works: The server inputs the preprocessed image into the AI ​​model and identifies it as a coffee stain.

[1025] Step 6:

[1026] The server compares the analysis results with the user's registered information and selects the most suitable detergent.

[1027] Input: Identified stain type, user detergent information.

[1028] Data calculation: Matches the type of dirt with the intended use of the detergent the user owns to identify the most suitable detergent.

[1029] Output: Information on the best detergent.

[1030] Specific behavior: The server determines that "laundry detergent" is best for removing coffee stains.

[1031] Step 7:

[1032] The server transmits the suggested detergent information to the terminal and notifies the user.

[1033] Input: Information on the best detergent.

[1034] Data calculation: Sending data for proposal generation and notification.

[1035] Output: Proposal.

[1036] Specific operation: The server sends a suggestion to the device saying "Use laundry detergent."

[1037] Step 8:

[1038] If the user does not have the suggested detergent, the system will provide a link to an e-commerce site where they can purchase it directly.

[1039] Input: detergent inventory information, e-commerce site link information.

[1040] Data calculation: Checking stock information and generating links.

[1041] Output: Purchase link.

[1042] Specific operation: If the server does not have "laundry detergent," it generates a link to Amazon or another site and sends it to the device.

[1043] (Application example 1)

[1044] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1045] Today's consumers are often confused by the wide variety of cleaning methods available, making it difficult to choose the right cleaning agent. Furthermore, when selecting products in physical stores, there is a lack of a system that instantly suggests the best cleaning agent for each type of dirt and purpose. This creates the risk of using the wrong cleaning agent and damaging the product.

[1046] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1047] In this invention, the server includes: means for registering information about detergents owned by the user; means for taking pictures of stains using a camera on the terminal; means for analyzing the captured images of stains and using a generative AI model to identify the type of stain; means for suggesting an optimal detergent based on the information about the detergents owned by the user; means for providing an option to purchase an appropriate detergent from an online sales platform if the suggested detergent is not in the user's inventory; and means for acquiring product information by scanning product barcodes and suggesting the optimal detergent based on the product information. This allows users to instantly find the optimal detergent and apply an appropriate cleaning method when selecting products in a physical store.

[1048] "User" refers to an individual or corporation that uses this system.

[1049] "Cleaning agents" refers to a general range of chemical products used to remove dirt from users' hands.

[1050] "Means for registering information" refers to the function of inputting and saving data on the cleaning agents owned by the user into the system.

[1051] "Device" refers to a smartphone, tablet, or other portable electronic device.

[1052] "Camera" refers to an image capturing device built into the terminal.

[1053] A "generative AI model" refers to an algorithm that uses artificial intelligence to analyze images of dirt and identify its features.

[1054] "Means for suggesting the most suitable cleaning agent" refers to the function of selecting and presenting the appropriate cleaning agent based on the user's registered information and analysis results.

[1055] "Online sales platform" refers to an e-commerce site that allows products to be purchased via the Internet.

[1056] "Barcode scanning function" refers to the function of using a camera to read the barcode attached to the product.

[1057] "Means for analyzing images" refers to technology for processing captured images and identifying their contents.

[1058] "Pre-processing" refers to the initial processing steps performed to improve the quality of a captured image.

[1059] "Server" refers to a computer system for processing, storing, and communicating data.

[1060] This invention is a system that suggests optimal detergents when a user selects products in a physical store. The system communicates between the user, terminal, and server to support the effective use of detergents.

[1061] 1. Cleaning Agent Registration

[1062] First, the user registers information about the detergent they own in the device's application. This information can be obtained by manually entering it or by taking a photo of the detergent label with the device's camera and using image recognition. The device extracts the necessary information (name, manufacturer, and purpose) from the photo of the label and sends it to the server. The server stores this information, linking it to each user's account.

[1063] Specific examples

[1064] The user manually enters "clothing detergent."

[1065] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information and sends it to the server.

[1066] 2. Photograph the stain and send the image

[1067] When a user finds a stain on a product, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device (color adjustment, noise removal, etc.), and then the processed image is sent to the server.

[1068] Specific examples

[1069] The user takes a picture of a stain on the carpet.

[1070] The device preprocesses the image and sends it to the server.

[1071] 3. AI-based stain analysis

[1072] The server analyzes the received image using a generative AI model to identify the type of dirt. The AI ​​identifies the type of dirt (food stains, oil stains, mud stains, etc.) and compares the results with the user's registered information stored in the server's database. If an appropriate cleaning agent is found, it will be recommended to the user.

[1073] Specific examples

[1074] The server analyzes the image and determines that it is a coffee stain.

[1075] The server determines from the user's registration information that "clothing detergent" is suitable for removing coffee stains.

[1076] 4. Obtain product information by scanning barcodes

[1077] Users can also obtain product information by scanning the product barcode with the device's camera. The barcode scanning function allows the system to recommend the most suitable detergent based on the product information.

[1078] Specific examples

[1079] A user scans the barcode of the clothing item they want to purchase in a brick-and-mortar store.

[1080] The terminal sends the barcode information to the server, which then identifies the most suitable cleaning agent.

[1081] 5. Suggestions and Recommendations

[1082] Based on the analysis results, the server will send the suggested cleaning product information to the terminal. If the user does not have the suggested cleaning product in stock, the system will also provide a link to an online sales platform for direct purchase.

[1083] Specific examples

[1084] The server sends a suggestion to the terminal saying "Use laundry detergent."

[1085] The terminal will display this offer to the user and, if the user does not have "clothes detergent", will also display a link to an online sales platform.

[1086] Prompt Sentence Examples

[1087] "Identify the type of stain in this image. Types of stains include coffee stains, oil stains, mud stains, etc."

[1088] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1089] Step 1:

[1090] Users register information about the detergent they own on their device. To do this, they can either manually enter the information or take a photo of the detergent label with the device's camera. The captured image of the label is analyzed on the device using image recognition technology, and the necessary information (name, manufacturer, purpose, etc.) is automatically extracted. The extracted information is sent to a server and saved, linked to the user's account.

[1091] Input: Manually enter cleaning agent information or label image

[1092] Data processing: Information extraction using image recognition

[1093] Output: Save cleaning agent information to database

[1094] Step 2:

[1095] When a user finds a product or stain, they take a photo of the stain with their device's camera. The captured image undergoes pre-processing such as color adjustment and noise removal on the device, and then the processed image is sent to the server.

[1096] Input: Image of dirt

[1097] Data processing: Image preprocessing (color adjustment, noise removal)

[1098] Output: Processed images sent to server

[1099] Step 3:

[1100] The server analyzes the received image using a generative AI model to identify the type of stain. The generative AI model uses prompts to analyze the characteristics of the stain and identify the specific stain type (e.g., coffee stain, oil stain, mud stain, etc.). Once the stain type is identified, the information is stored in the server's database.

[1101] Input: Processed image

[1102] Data Computation: Identifying Contamination with Generative AI Models

[1103] Output: Server save of dirt type information

[1104] Step 4:

[1105] The server compares the information on the type of dirt with the information on the detergent registered by the user to identify the most suitable detergent. If an appropriate detergent is identified, the recommendation is sent to the device.

[1106] Input: Dirt type information, user-registered cleaning agent information

[1107] Data calculations: collating information to identify optimal cleaning products

[1108] Output: Sends the proposal to the device.

[1109] Step 5:

[1110] The user can review the suggestions on their device and select the appropriate cleaning product. If the user does not have the suggested cleaning product in stock, the server will also provide a link to an online sales platform so that the user can purchase it directly.

[1111] Enter: View Suggestions

[1112] Data calculation: inventory check and link generation

[1113] Output: Display of online sales platform link

[1114] Step 6:

[1115] When a user selects a product in a physical store, they scan the product's barcode with the device's camera, and the device sends the barcode information to the server, which then identifies the appropriate cleaning product and sends a recommendation to the device.

[1116] Input: Product barcode

[1117] Data calculation: Analysis of barcode data and identification of cleaning agents

[1118] Output: Sends the proposal to the device.

[1119] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1120] This invention is a system that suggests detergents to properly remove dirt, and adds a function to recognize the user's emotions and customize the suggestions based on those emotions. This system communicates between the user, the terminal, and the server, and by combining an emotion engine, makes suggestions that are more appropriate and effective for the user.

[1121] 1. Detergent Registration

[1122] First, users register information about their detergent in the application. This involves manually entering the information or taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the label. This information is then sent to the server and stored in association with the user's account.

[1123] Examples:

[1124] The user manually enters "Laundry Detergent."

[1125] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[1126] 2. Photograph the stain and send the image

[1127] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[1128] Examples:

[1129] The user takes a picture of a stain on the carpet.

[1130] The device preprocesses the image and sends it to the server.

[1131] 3. AI-based stain analysis

[1132] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[1133] Examples:

[1134] The server analyzes the image and determines that it is a coffee stain.

[1135] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[1136] 4. Suggestions and Recommendations

[1137] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[1138] Examples:

[1139] The server sends a suggestion to the device: "Use laundry detergent."

[1140] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[1141] 5. Use of Emotion Engine

[1142] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Based on this emotion data, the server further customizes the suggestions.

[1143] Examples:

[1144] If the user shows a confused look on their face about which detergent to choose, the emotion engine will recognize that emotion and send it to the server.

[1145] The server receives the emotion data and sends suggestions to the terminal, including detailed usage instructions that are easy for the user to understand.

[1146] In this way, the present invention, which combines an emotion engine, not only suggests the most suitable detergent based on the detergent information the user has and the results of stain analysis, but also recognizes the user's emotions and adjusts the suggestions based on them, providing a more user-friendly system.

[1147] The processing flow will be explained below.

[1148] Step 1:

[1149] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[1150] Step 2:

[1151] The device captures the label image and uses image recognition to extract the necessary information from the label.

[1152] Step 3:

[1153] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[1154] Step 4:

[1155] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[1156] Step 5:

[1157] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[1158] Step 6:

[1159] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[1160] Step 7:

[1161] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[1162] Step 8:

[1163] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[1164] Step 9:

[1165] The user can enable the emotion engine in the settings of the application.

[1166] Step 10:

[1167] The device acquires the user's emotional data using a camera and microphone. The device captures the user's facial expressions with the camera and records their voice with the microphone.

[1168] Step 11:

[1169] The device analyzes the emotion data to determine the user's current emotional state, a process that identifies emotions such as stress, frustration, and satisfaction.

[1170] Step 12:

[1171] The device sends the analyzed emotional data to the server, which then customizes the detergent recommendations based on the emotional data received.

[1172] Step 13:

[1173] The server tailors its suggestions depending on the user's emotional state, for example, providing suggestions with detailed instructions if the user is confused.

[1174] Step 14:

[1175] The terminal displays the customized suggestions received from the server to the user, allowing the user to receive appropriate suggestions that match their emotions.

[1176] For example, if a user registers "clothing detergent" and takes a photo of a coffee stain on a carpet, the server will suggest "clothing detergent." If the user shows a confused expression, the device will recognize the emotion, and the server will provide detailed instructions for use, which the device will then display.

[1177] Example 2

[1178] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1179] Conventional detergent recommendation systems simply make recommendations based on the type of dirt and the detergent information registered by the user, without taking the user's feelings into consideration. This has led to issues such as users being confused by the recommendations or not knowing how to properly operate the system. Furthermore, when multiple detergents are applicable to a particular dirt, the system is unable to provide appropriate advice to help the user make the optimal selection.

[1180] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using the camera of the terminal, means for identifying the type of stain by analyzing the photographed image of the stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in the user's inventory, and means for recognizing the user's emotional state and customizing the suggestions based on the emotion. This makes it easier to deal with cases where the user is confused about which detergent to select or does not know how to operate the system, and makes it possible to provide a user-friendly system.

[1181] "User" refers to an individual or organization that accesses the system and registers detergent information, photographs stains, receives suggestions, etc.

[1182] A "terminal" refers to a computer device that is equipped with devices such as a camera and microphone and that is operated by a user. Examples include smartphones, tablets, and PCs.

[1183] "Server" refers to a computer system that receives information sent from a terminal and analyzes, stores, and provides the data.

[1184] "Detergent information" refers to detailed information such as the detergent's name, manufacturer, and purpose. This data indicates the characteristics of the detergent the user owns.

[1185] "Camera" refers to a device for capturing images, including those built into a device and those connected externally.

[1186] "Soil" refers to impurities or stains on the surface of an object that needs to be cleaned, such as food stains, mud stains, or oil stains.

[1187] "Image recognition technology" refers to technology that automatically extracts and analyzes specific information from captured images, including character recognition and object identification.

[1188] "Preprocessing" refers to the adjustment and processing techniques used to process images and data before analyzing them, including color adjustment and noise removal.

[1189] An "AI model" refers to a software model that uses artificial intelligence technology to analyze data and make predictions. Many of these models use machine learning algorithms.

[1190] "Emotion engine" refers to technology that uses the device's camera and microphone to analyze and recognize emotions from the user's facial expressions and voice.

[1191] An "e-commerce site" is a website that sells and purchases goods and services over the internet. Amazon is an example of this.

[1192] "Suggestion" refers to information that indicates the optimal detergent and its usage method to the user based on the analysis results.

[1193] "Customization" refers to adjusting and changing the content of suggestions according to the user's characteristics and situation.

[1194] The present invention provides a detergent recommendation system that can recognize a user's emotions and customize the content of recommendations based on those emotions. A specific embodiment of this system will be described below.

[1195] First, the user registers information about the detergent they own in the application. This process involves manual input and taking a photo of the detergent label using the device's camera. The device typically used here is a camera-equipped device such as a smartphone or tablet. Software such as TensorFlow and OpenCV is used for image recognition technology.

[1196] For example, there is a case where the user manually enters "laundry detergent" or a case where the user takes a picture of the label of "dishwashing detergent" with the device's camera and the information is automatically extracted. The information extracted from the label (name, manufacturer, purpose) is sent from the device over the Internet to a server. The server associates this information with the user's account and stores it in a database.

[1197] Next, if the user finds a stain that needs cleaning, they take a photo of it with the device's camera. The captured image undergoes preprocessing, such as color adjustment and noise removal, before being sent to the server. Image processing libraries such as OpenCV are used for preprocessing.

[1198] The server uses an AI model to analyze the received image. Common AI models used here include TensorFlow and PyTorch. The AI ​​identifies stains in the image and determines the type of stain (food stains, oil stains, mud stains, etc.). The analysis results are compared with the user's detergent information stored in the server's database, and if a suitable detergent is found, that information is generated as a suggestion to the user.

[1199] For example, if the server analyzes the image and determines that it is a coffee stain, it can use the user's registration information to determine that "laundry detergent" is suitable for coffee stains. The server then sends this information to the user's device. The suggestion includes a message such as "Use laundry detergent." If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so the user can purchase it directly.

[1200] The system also features an emotion engine, which uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize their emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Emotion recognition uses Microsoft Azure's Emotion API and IBM Watson. Emotion data is sent to a server, which then uses it to further customize the suggestions.

[1201] For example, if a user shows a confused expression about which detergent to choose, the emotion engine will recognize that emotion and the server will send a suggestion to the device, including specific and detailed instructions for use. In this way, the system helps users avoid confusion about which detergent to choose and how to use it.

[1202] Example prompts for generative AI models

[1203] Detergent Information

[1204] Name: Laundry detergent

[1205] Manufacturer: ABC Company

[1206] Uses: stains on clothing

[1207] Dirt information

[1208] Type: Coffee stains

[1209] Part: Carpet

[1210] User Emotion: Confused

[1211] Generate a proposal.

[1212] This invention not only proposes the optimum detergent based on the detergent information and stain analysis results of the user, but also recognizes the user's emotions and adjusts the proposal content based on them, thereby providing a more user-friendly system.

[1213] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1214] Step 1: Enter and register detergent information

[1215] Users use the application to register information about the detergent they own. They can either manually enter the information or use their device's camera to take a photo of the detergent label. When taking a photo of the label, the device uses image recognition technology to extract information such as the detergent's name, manufacturer, and purpose from the label. This information is sent to the server in JSON format.

[1216] Input: Detergent information manually entered by the user or detergent label image taken with a camera

[1217] Data processing: Extracting text information from images using image recognition technology

[1218] Output: Detergent information in JSON format

[1219] Specific behavior:

[1220] The user enters detergent information into the application.

[1221] The user takes a picture of the detergent label with the camera.

[1222] The device uses OpenCV to analyze the label image and extract text information.

[1223] The extracted information is converted into JSON format and sent to the server.

[1224] Step 2: Save your detergent information

[1225] The server parses the received detergent information and stores it in a database, where it is associated with the user's account and used to provide future recommendations.

[1226] Input: Detergent information in JSON format

[1227] Data processing: Parse and save to database

[1228] Output: Detergent information stored in the database

[1229] Specific behavior:

[1230] The server receives the data in JSON format.

[1231] The parser parses the JSON data and extracts the detergent information.

[1232] Generate and execute SQL queries to insert the extracted information into a database.

[1233] Step 3: Capture and pre-process stain images

[1234] When a user finds a stain that needs cleaning, they take a photo of it with their device's camera. Once the image is captured, the device performs pre-processing such as color adjustment and noise removal, which allows the server to analyze the image more accurately.

[1235] Input: Image of dirt

[1236] Data processing: color adjustment and noise removal

[1237] Output: Pre-processed stain image

[1238] Specific behavior:

[1239] The user takes a picture of the stain using the device's camera.

[1240] The device uses OpenCV to apply color adjustment and noise reduction filters.

[1241] Encode the preprocessed image in binary format.

[1242] Step 4: Submit and analyze the stain image

[1243] The pre-processed image is sent to a server, which uses an AI model to analyze the image and identify the type of stain (e.g., food stain, oil stain, mud stain, etc.).

[1244] Input: Preprocessed stain image

[1245] Data processing: Image analysis using AI models

[1246] Output: Analysis results including the type and characteristics of the dirt

[1247] Specific behavior:

[1248] The preprocessed image is sent to the server.

[1249] The server analyzes the image using a TensorFlow model.

[1250] The type and characteristics of the dirt are identified and saved as analysis results.

[1251] Step 5: Recommending the right detergent

[1252] The server compares the analysis results with the detergent information in the user's database to identify the optimal detergent, and then sends the identified detergent information to the user's device.

[1253] Input: Dirt analysis results, user detergent information

[1254] Data processing: matching and identifying the best detergent

[1255] Output: A message containing the suggestion

[1256] Specific behavior:

[1257] The server compares the stain analysis results with the user's detergent information.

[1258] Identify the best detergent and generate recommendations.

[1259] The proposal is encoded in JSON format and sent to the user's device.

[1260] Step 6: View Offer and Purchase Options

[1261] The terminal displays the received suggestions to the user, and if the suggested detergent is not in the user's inventory, the system provides a link to an e-commerce site so the user can purchase it directly.

[1262] Input: Proposal

[1263] Data processing: Converting data into a display format

[1264] Output: The offer and purchase link shown to the user

[1265] Specific behavior:

[1266] The device receives the proposal.

[1267] The suggestions are presented in a user-friendly format.

[1268] If appropriate, links to e-commerce sites are also provided.

[1269] Step 7: Emotion recognition and personalized suggestions

[1270] The device analyzes the user's facial expressions and voice to recognize their emotions, and the emotional data is sent to the server, which then uses this information to further customize the suggestions.

[1271] Input: User's facial expressions and voice

[1272] Data processing: sentiment analysis and data transmission

[1273] Output: Customized suggestions

[1274] Specific behavior:

[1275] The device uses a camera and microphone to capture facial expressions and voice.

[1276] Analyze emotions using an emotion engine (such as Microsoft Azure's Emotion API).

[1277] Emotion data is sent to the server.

[1278] The server regenerates the proposal content based on the received emotion data and transmits it to the terminal.

[1279] (Application example 2)

[1280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1281] Conventional detergent recommendation systems require users to register their detergent information and analyze images of stains to suggest the optimal detergent. However, because they do not take the user's emotional state into account, the system is unable to make the recommendations easy for users to understand. Furthermore, customers may feel stressed or confused when receiving appropriate recommendations in physical stores, which prevents improved satisfaction. To solve these problems, a system is needed that recognizes the user's emotional state and customizes the recommendations based on that state.

[1282] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1283] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera of the terminal, means for analyzing the photographed image of the stains to identify the type of stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user, means for recognizing the emotional state of the user and customizing the suggestion content based on the emotion, means for recognizing the user's emotion and an image of the stain using smart glasses, and means for making detailed suggestions including detergent usage procedures based on the recognized emotional state. This enables appropriate and easy-to-understand suggestions according to the emotional state of the user.

[1284] "User" means an individual or entity that uses the system to receive detergent suggestions.

[1285] The "means for registering information about detergents owned" is a function that allows a user to input or scan information about detergents that the user owns into the system and store the information in a database.

[1286] A "terminal" is a device equipped with input devices such as a camera and a microphone, and is a computer or smartphone that executes the functions of the system.

[1287] The "means for photographing the dirt using a camera" is a method for capturing an image of the dirt using a camera mounted on the terminal.

[1288] The "means for analyzing the image of the dirt and identifying the type of dirt" is a function for analyzing the captured image of the dirt and identifying the type of dirt from the results.

[1289] The "means for suggesting the optimum detergent" is a function that recommends to the user the detergent that will most effectively remove the stain, based on the type of stain and information about the detergent the user owns.

[1290] "Means for providing the option to purchase suitable detergent from an e-commerce site" refers to a function that provides a link or information that allows the user to purchase the detergent online if the suggested detergent is not in the user's inventory.

[1291] The "means for recognizing emotional state" is a function that uses the device's camera and microphone to analyze emotions from the user's facial expressions and voice and obtain the results.

[1292] The "means for customizing the suggestions" is a function that adjusts detergent suggestions and usage procedures according to the recognized emotional state of the user, making them easier for the user to understand.

[1293] "Smart glasses" are wearable devices equipped with cameras and monitors, and are used to analyze the user's visual information and emotions.

[1294] The "means for providing detailed suggestions" is a function that provides the user with specific instructions and additional information on how to remove stains.

[1295] This invention is a system that registers detergent information owned by a user, analyzes images of stains, and recommends the most suitable detergent, and adds a function to recognize the emotional state of the user and customize the recommendations based on that. Specific embodiments for implementing this system are described below.

[1296] 1. Registering detergent information

[1297] First, the user registers information about the detergent they own in the system. This can be done by scanning the product's barcode or by manually entering the information. The user can also use the device's camera to take a photo of the detergent label and extract the necessary information using image recognition technology. The extracted information is sent to the server and stored in association with the user's account.

[1298] 2. Photograph the stain and send the image

[1299] Next, if the user finds a stain, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[1300] 3. AI-based stain analysis

[1301] The server is equipped with a generative AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain and compares the results with the user's registered information stored in the server's database. If a suitable detergent is found, it is generated as a recommendation for the user.

[1302] 4. Suggestions and Recommendations

[1303] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[1304] 5. Use of Emotion Engine

[1305] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state. Based on this emotion data, the server further customizes the suggestions.

[1306] Hardware and Software Description

[1307] Devices used include smartphones, tablets, and smart glasses, with Google Glass and Microsoft HoloLens being particularly suitable.

[1308] OpenCV is used for image processing, and Azure Emotion API and EmotionRecognizer library are used for emotion recognition.

[1309] A generative AI model is used to analyze the dirt, and when a prompt is sent to analyze the image, the data is uploaded to a server in the cloud.

[1310] Specific examples

[1311] For example, consider a scenario in which a store staff member uses smart glasses to interact with a customer in a brick-and-mortar store. If the customer asks with a confused expression, "I don't know which detergent to use," the camera and microphone in the smart glasses will recognize the customer's emotional state. If the store staff member takes a photo of the stain brought in by the customer and the system analyzes that the stain is a coffee stain, the system will make a suggestion including detailed instructions, such as, "Use this laundry detergent. Here are the instructions for use."

[1312] Prompt Sentence Examples

[1313] "Analyze this image to identify the type of dirt."

[1314] "If your customer is confused, show them the suggestion with detailed instructions."

[1315] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1316] Step 1:

[1317] Register detergent information owned by the user.

[1318] Users can scan the detergent's barcode using their device's camera or manually enter the information. The device then uses image recognition technology to analyze the barcode and extract the necessary information (name, manufacturer, and purpose). The extracted information is then sent to a server and stored in association with the user's account.

[1319] Input: Detergent barcode image or manually entered information.

[1320] Output: Detergent information stored in the user account.

[1321] Step 2:

[1322] Take a picture of the stain with your device's camera.

[1323] When a user finds a stain, they take a photo of it using their device's camera. The device then pre-processes the image, adjusting color and removing noise, and sends the processed image to the server.

[1324] Input: A photographed image of the dirt.

[1325] Output: Preprocessed image data.

[1326] Step 3:

[1327] The server analyzes the received image of the dirt.

[1328] The server uses the generative AI model to analyze the pre-processed image, which identifies the type of stain in the image (e.g., coffee stain, oil stain, etc.) and stores the identified stain type in a database.

[1329] Input: Preprocessed stain image.

[1330] Output: Identified soil type information.

[1331] Step 4:

[1332] The server recommends the most suitable detergent based on the user's detergent information.

[1333] The server compares the identified stain type with the detergent information of the user and suggests the most suitable detergent. The suggestion is sent to the terminal and displayed to the user.

[1334] Input: Identified stain type information, user detergent information.

[1335] Output: Recommendations for the best detergent.

[1336] Step 5:

[1337] Recognize the user's emotional state.

[1338] The device's camera and microphone are used to analyze the user's facial expressions and voice, and the emotion engine recognizes the user's emotional state (e.g., confusion, satisfaction, stress, etc.). The recognized emotion data is sent to the server.

[1339] Input: User's facial and voice data.

[1340] Output: Recognized emotional state data.

[1341] Step 6:

[1342] The server customizes the suggestions based on the emotion data.

[1343] The server receives the emotional state data and tailors the suggestions. For example, if the user is confused, the system generates suggestions that include detailed instructions and links to e-commerce sites. These customized suggestions are sent to the device and displayed to the user.

[1344] Input: Emotional state data, optimal detergent recommendation.

[1345] Output: A customized proposal.

[1346] Step 7:

[1347] Recognizing dirt and emotions using smart glasses.

[1348] Store staff use smart glasses to recognize customer emotions and images of dirt. The smart glasses' cameras and microphones capture the customer's facial expressions and voice and send the data to a server, which analyzes it and generates suggestions.

[1349] Input: Images of dirt captured by smart glasses and customer emotion data.

[1350] Output: Recognized stain types and emotion data.

[1351] Step 8:

[1352] The server will provide detailed suggestions.

[1353] Based on the recognized emotional state, the server generates detailed recommendations, including instructions for using the detergent, which are displayed to staff through smart glasses and explained to the customer.

[1354] Input: Recognized dirt type, emotional state data.

[1355] Output: Detailed proposal.

[1356] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1357] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1358] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1359] [Fourth embodiment]

[1360] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1361] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1362] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1363] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1364] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1365] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1366] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1367] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1368] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1369] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1370] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1371] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1372] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1373] This invention is a system that registers information about the detergents a user owns and suggests the most suitable detergent for stains photographed with a smartphone or other device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[1374] 1. Detergent Registration

[1375] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to the server and stored in association with the user's account.

[1376] Examples:

[1377] The user manually enters "Laundry Detergent."

[1378] When a user takes a photo of the "dishwashing detergent" label with the device's camera, the device automatically extracts the information and sends it to the server.

[1379] 2. Photograph the stain and send the image

[1380] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[1381] Examples:

[1382] The user takes a picture of a stain on the carpet.

[1383] The device preprocesses the image and sends it to the server.

[1384] 3. AI-based stain analysis

[1385] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[1386] Examples:

[1387] The server analyzes the image and determines that it is a coffee stain.

[1388] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[1389] 4. Suggestions and Recommendations

[1390] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[1391] Examples:

[1392] The server sends a suggestion to the device: "Use laundry detergent."

[1393] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[1394] In this way, the system utilizes the detergent information the user has, allowing them to efficiently select the optimal detergent to remove dirt. At each step, image recognition technology and AI analysis are used to minimize the user's effort.

[1395] The processing flow will be explained below.

[1396] Step 1:

[1397] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[1398] Step 2:

[1399] The device captures the label image and uses image recognition to extract the necessary information from the label.

[1400] Step 3:

[1401] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[1402] Step 4:

[1403] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[1404] Step 5:

[1405] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[1406] Step 6:

[1407] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[1408] Step 7:

[1409] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[1410] Step 8:

[1411] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[1412] As a specific example, if a user registers "dishwashing detergent" and takes a photo of a coffee stain on a carpet, the server will analyze the image and determine that "laundry detergent" is the best option. If the laundry detergent is not in stock, a purchase link will be provided.

[1413] Example 1

[1414] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1415] Conventional detergent selection systems lack the functionality to efficiently manage the detergent information owned by the user and suggest the detergent best suited to a specific stain. Furthermore, when users manually input the information about the detergents they own, it is highly likely to result in time-consuming work and input errors. Furthermore, if the suggested detergent is not in stock, the process of purchasing it can be cumbersome. Therefore, an efficient and accurate system that can solve these issues is needed.

[1416] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1417] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera on the terminal, means for preprocessing the photographed image of the stain and transmitting it to the server, means for analyzing the photographed image of the stain to identify the type of stain, means for suggesting an optimal detergent based on information about the detergents owned by the user, and means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user. This makes it possible to efficiently manage information about detergents owned by the user, quickly suggest an optimal detergent for a specific stain, and easily perform the procedure for purchasing the necessary detergent.

[1418] A "user" is a person who registers detergent information, takes an image of the stain using a terminal, and receives recommendations for the most suitable detergent from the system.

[1419] A "terminal" is a device that has a camera function, takes an image of the stain, performs preprocessing, and transmits the image to the server.

[1420] The "server" is the central computer in the system that stores information about detergents registered by users, analyzes received images of stains, suggests the most suitable detergent, and also provides purchasing options.

[1421] "Means for registering detergent information" refers to a function that allows information such as the name, manufacturer, and purpose of the detergent the user owns to be entered into the application or obtained through image recognition, and then saved on the server.

[1422] The "means for photographing dirt" refers to a function that uses the camera of the terminal to take an image of the dirt that the user needs to clean.

[1423] "Preprocessing" refers to the provision of image processing technology to improve analysis accuracy by performing processes such as color correction and noise removal on captured images.

[1424] "Analysis means" refers to a function that uses an AI model installed on the server to identify the type of dirt in the captured image.

[1425] The "suggestion means" is a function that suggests the most suitable detergent from among the detergents the user owns based on the analyzed type of dirt.

[1426] "Means for providing purchase options" refers to a function that provides a link to an e-commerce site so that the appropriate detergent can be purchased directly if the suggested detergent is not in the user's inventory.

[1427] The "image recognition function" is a technology that extracts information such as the name, manufacturer, and purpose from the detergent label photographed with the device's camera.

[1428] An "AI model" is a model trained using machine learning technology to analyze photographed images of dirt and identify the type of dirt.

[1429] An "e-commerce site" is a website for purchasing goods over the Internet.

[1430] This invention is a system that registers information about the detergents a user owns and suggests the best detergent for stains photographed with a device camera. The system communicates between the user, device, and server to support the effective use of detergents.

[1431] First, the user registers information about the detergent they own in the application. This information can be obtained by manually entering it or by taking a photo of the detergent label using the device's camera. The device then uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, and purpose) from the photographed label. This information is then sent to a server and stored in association with the user's account.

[1432] For example, if a user takes a photo of the label of a "laundry detergent" product with their device's camera, the device automatically extracts the information and sends it to the server. This information is then stored in the user's account, allowing them to add other detergents, such as "dish detergent," in the same way.

[1433] Next, if the user finds a stain that needs cleaning, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device, with color adjustment and noise removal performed. The processed image is then sent to the server. For example, if a user takes a photo of a stain on a carpet, the device pre-processes the image and sends it to the server.

[1434] The server has an AI model (e.g., a TensorFlow or PyTorch-based model) that analyzes the received image and identifies the stains in the image. The AI ​​identifies the type of stain (food stain, oil stain, mud stain, etc.) and compares the result with the user's registered information stored in the server's database. If an appropriate detergent is found, it generates a recommendation for the user. For example, if the server analyzes an image and determines that it is a coffee stain, the server will identify from the user's registered information that "laundry detergent" is suitable for coffee stains.

[1435] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so that the user can purchase it directly. For example, if the server sends the suggestion "Use laundry detergent" to the terminal and the user does not have "laundry detergent," a link to Amazon or other sites will also be displayed.

[1436] In this system, examples of prompts to be input to the generative AI model include:

[1437] "A user finds a coffee stain on their carpet and takes a photo of it with their device. The AI ​​should analyze the stain and suggest the best detergent to use. If the suggested detergent is not in stock, the AI ​​should also provide a link to purchase it."

[1438] This prompt allows the system to consistently analyze and make recommendations based on the input data for a specific stain.

[1439] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1440] Step 1:

[1441] Users register information about the detergents they own in the application.

[1442] Input: The user manually enters the name, manufacturer, and purpose of the detergent. Alternatively, the user can take a photo of the detergent label using the device's camera.

[1443] Data processing: In the case of manual input, the information entered in the form is verified and saved. In the case of label photography, the device uses image recognition technology such as Google Cloud Vision API to extract the necessary information (name, manufacturer, purpose) from the photographed label.

[1444] Output: Extracted detergent information (name, manufacturer, purpose).

[1445] Specific operation: The user manually enters "Laundry detergent." The user takes a photo of the "Dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[1446] Step 2:

[1447] The device sends the extracted detergent information to a server and stores it in the user's account.

[1448] Input: Extracted detergent information (name, manufacturer, purpose).

[1449] Data calculation: The device communicates with the server using a secure protocol such as HTTPS and transmits information.

[1450] Output: User's detergent information stored in the server's database.

[1451] Specific operation: The device sends the information of "dishwashing detergent," "ABC manufacturer," and "dishware" to the server, and the server stores it in the user's account.

[1452] Step 3:

[1453] When a user finds dirt that needs to be cleaned, they take a picture of the dirt with the device's camera.

[1454] Input: A dirty area (e.g., a carpet), and the device's camera to take a photo.

[1455] Data processing: Take an image of the dirt using the device's camera.

[1456] Output: The captured image of the stain.

[1457] Specific operation: The user takes a photo of a stain on the carpet using the device's camera.

[1458] Step 4:

[1459] The captured images are first pre-processed on the device and then sent to the server.

[1460] Input: A photographed image of the dirt.

[1461] Data processing: The terminal performs pre-processing such as color adjustment and noise removal.

[1462] Output: Preprocessed stain image.

[1463] Specific operation: The device performs color correction and noise reduction on the captured image, and then sends the preprocessed image to the server via HTTPS protocol.

[1464] Step 5:

[1465] The server uses an AI model to analyze the received image, which then identifies the stains in the image.

[1466] Input: Preprocessed stain images.

[1467] Data processing: An AI model (e.g., TensorFlow or PyTorch) installed on the server analyzes the image and identifies the type of dirt.

[1468] Output: The type of stain identified (e.g., coffee stain).

[1469] How it works: The server inputs the preprocessed image into the AI ​​model and identifies it as a coffee stain.

[1470] Step 6:

[1471] The server compares the analysis results with the user's registered information and selects the most suitable detergent.

[1472] Input: Identified stain type, user detergent information.

[1473] Data calculation: Matches the type of dirt with the intended use of the detergent the user owns to identify the most suitable detergent.

[1474] Output: Information on the best detergent.

[1475] Specific behavior: The server determines that "laundry detergent" is best for removing coffee stains.

[1476] Step 7:

[1477] The server transmits the suggested detergent information to the terminal and notifies the user.

[1478] Input: Information on the best detergent.

[1479] Data calculation: Sending data for proposal generation and notification.

[1480] Output: Proposal.

[1481] Specific operation: The server sends a suggestion to the device saying "Use laundry detergent."

[1482] Step 8:

[1483] If the user does not have the suggested detergent, the system will provide a link to an e-commerce site where they can purchase it directly.

[1484] Input: detergent inventory information, e-commerce site link information.

[1485] Data calculation: Checking stock information and generating links.

[1486] Output: Purchase link.

[1487] Specific operation: If the server does not have "laundry detergent," it generates a link to Amazon or another site and sends it to the device.

[1488] (Application example 1)

[1489] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1490] Today's consumers are often confused by the wide variety of cleaning methods available, making it difficult to choose the right cleaning agent. Furthermore, when selecting products in physical stores, there is a lack of a system that instantly suggests the best cleaning agent for each type of dirt and purpose. This creates the risk of using the wrong cleaning agent and damaging the product.

[1491] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1492] In this invention, the server includes: means for registering information about detergents owned by the user; means for taking pictures of stains using a camera on the terminal; means for analyzing the captured images of stains and using a generative AI model to identify the type of stain; means for suggesting an optimal detergent based on the information about the detergents owned by the user; means for providing an option to purchase an appropriate detergent from an online sales platform if the suggested detergent is not in the user's inventory; and means for acquiring product information by scanning product barcodes and suggesting the optimal detergent based on the product information. This allows users to instantly find the optimal detergent and apply an appropriate cleaning method when selecting products in a physical store.

[1493] "User" refers to an individual or corporation that uses this system.

[1494] "Cleaning agents" refers to a general range of chemical products used to remove dirt from users' hands.

[1495] "Means for registering information" refers to the function of inputting and saving data on the cleaning agents owned by the user into the system.

[1496] "Device" refers to a smartphone, tablet, or other portable electronic device.

[1497] "Camera" refers to an image capturing device built into the terminal.

[1498] A "generative AI model" refers to an algorithm that uses artificial intelligence to analyze images of dirt and identify its features.

[1499] "Means for suggesting the most suitable cleaning agent" refers to the function of selecting and presenting the appropriate cleaning agent based on the user's registered information and analysis results.

[1500] "Online sales platform" refers to an e-commerce site that allows products to be purchased via the Internet.

[1501] "Barcode scanning function" refers to the function of using a camera to read the barcode attached to the product.

[1502] "Means for analyzing images" refers to technology for processing captured images and identifying their contents.

[1503] "Pre-processing" refers to the initial processing steps performed to improve the quality of a captured image.

[1504] "Server" refers to a computer system for processing, storing, and communicating data.

[1505] This invention is a system that suggests optimal detergents when a user selects products in a physical store. The system communicates between the user, terminal, and server to support the effective use of detergents.

[1506] 1. Cleaning Agent Registration

[1507] First, the user registers information about the detergent they own in the device's application. This information can be obtained by manually entering it or by taking a photo of the detergent label with the device's camera and using image recognition. The device extracts the necessary information (name, manufacturer, and purpose) from the photo of the label and sends it to the server. The server stores this information, linking it to each user's account.

[1508] Specific examples

[1509] The user manually enters "clothing detergent."

[1510] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information and sends it to the server.

[1511] 2. Photograph the stain and send the image

[1512] When a user finds a stain on a product, they take a photo of the stain with their device's camera. The captured image is pre-processed on the device (color adjustment, noise removal, etc.), and then the processed image is sent to the server.

[1513] Specific examples

[1514] The user takes a picture of a stain on the carpet.

[1515] The device preprocesses the image and sends it to the server.

[1516] 3. AI-based stain analysis

[1517] The server analyzes the received image using a generative AI model to identify the type of dirt. The AI ​​identifies the type of dirt (food stains, oil stains, mud stains, etc.) and compares the results with the user's registered information stored in the server's database. If an appropriate cleaning agent is found, it will be recommended to the user.

[1518] Specific examples

[1519] The server analyzes the image and determines that it is a coffee stain.

[1520] The server determines from the user's registration information that "clothing detergent" is suitable for removing coffee stains.

[1521] 4. Obtain product information by scanning barcodes

[1522] Users can also obtain product information by scanning the product barcode with the device's camera. The barcode scanning function allows the system to recommend the most suitable detergent based on the product information.

[1523] Specific examples

[1524] A user scans the barcode of the clothing item they want to purchase in a brick-and-mortar store.

[1525] The terminal sends the barcode information to the server, which then identifies the most suitable cleaning agent.

[1526] 5. Suggestions and Recommendations

[1527] Based on the analysis results, the server will send the suggested cleaning product information to the terminal. If the user does not have the suggested cleaning product in stock, the system will also provide a link to an online sales platform for direct purchase.

[1528] Specific examples

[1529] The server sends a suggestion to the terminal saying "Use laundry detergent."

[1530] The terminal will display this offer to the user and, if the user does not have "clothes detergent", will also display a link to an online sales platform.

[1531] Prompt Sentence Examples

[1532] "Identify the type of stain in this image. Types of stains include coffee stains, oil stains, mud stains, etc."

[1533] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1534] Step 1:

[1535] Users register information about the detergent they own on their device. To do this, they can either manually enter the information or take a photo of the detergent label with the device's camera. The captured image of the label is analyzed on the device using image recognition technology, and the necessary information (name, manufacturer, purpose, etc.) is automatically extracted. The extracted information is sent to a server and saved, linked to the user's account.

[1536] Input: Manually enter cleaning agent information or label image

[1537] Data processing: Information extraction using image recognition

[1538] Output: Save cleaning agent information to database

[1539] Step 2:

[1540] When a user finds a product or stain, they take a photo of the stain with their device's camera. The captured image undergoes pre-processing such as color adjustment and noise removal on the device, and then the processed image is sent to the server.

[1541] Input: Image of dirt

[1542] Data processing: Image preprocessing (color adjustment, noise removal)

[1543] Output: Processed images sent to server

[1544] Step 3:

[1545] The server analyzes the received image using a generative AI model to identify the type of stain. The generative AI model uses prompts to analyze the characteristics of the stain and identify the specific stain type (e.g., coffee stain, oil stain, mud stain, etc.). Once the stain type is identified, the information is stored in the server's database.

[1546] Input: Processed image

[1547] Data Computation: Identifying Contamination with Generative AI Models

[1548] Output: Server save of dirt type information

[1549] Step 4:

[1550] The server compares the information on the type of dirt with the information on the detergent registered by the user to identify the most suitable detergent. If an appropriate detergent is identified, the recommendation is sent to the device.

[1551] Input: Dirt type information, user-registered cleaning agent information

[1552] Data calculations: collating information to identify optimal cleaning products

[1553] Output: Sends the proposal to the device.

[1554] Step 5:

[1555] The user can review the suggestions on their device and select the appropriate cleaning product. If the user does not have the suggested cleaning product in stock, the server will also provide a link to an online sales platform so that the user can purchase it directly.

[1556] Enter: View Suggestions

[1557] Data calculation: inventory check and link generation

[1558] Output: Display of online sales platform link

[1559] Step 6:

[1560] When a user selects a product in a physical store, they scan the product's barcode with the device's camera, and the device sends the barcode information to the server, which then identifies the appropriate cleaning product and sends a recommendation to the device.

[1561] Input: Product barcode

[1562] Data calculation: Analysis of barcode data and identification of cleaning agents

[1563] Output: Sends the proposal to the device.

[1564] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1565] This invention is a system that suggests detergents to properly remove dirt, and adds a function to recognize the user's emotions and customize the suggestions based on those emotions. This system communicates between the user, the terminal, and the server, and by combining an emotion engine, makes suggestions that are more appropriate and effective for the user.

[1566] 1. Detergent Registration

[1567] First, users register information about their detergent in the application. This involves manually entering the information or taking a photo of the detergent label using the device's camera. The device uses image recognition technology to extract the necessary information (name, manufacturer, and purpose) from the label. This information is then sent to the server and stored in association with the user's account.

[1568] Examples:

[1569] The user manually enters "Laundry Detergent."

[1570] The user takes a photo of the "dishwashing detergent" label with the device's camera, and the device automatically extracts the information.

[1571] 2. Photograph the stain and send the image

[1572] Next, if the user finds a stain that needs cleaning, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[1573] Examples:

[1574] The user takes a picture of a stain on the carpet.

[1575] The device preprocesses the image and sends it to the server.

[1576] 3. AI-based stain analysis

[1577] The server is equipped with an AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain (food stains, oil stains, mud stains, etc.) and compares the results with the user registration information stored in the server's database. If a suitable detergent is found, it generates a recommendation for the user.

[1578] Examples:

[1579] The server analyzes the image and determines that it is a coffee stain.

[1580] The server determines from the user's registration information that "laundry detergent" is suitable for removing coffee stains.

[1581] 4. Suggestions and Recommendations

[1582] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[1583] Examples:

[1584] The server sends a suggestion to the device: "Use laundry detergent."

[1585] The device will display suggestions to the user, including links to Amazon if the user doesn't have "laundry detergent."

[1586] 5. Use of Emotion Engine

[1587] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Based on this emotion data, the server further customizes the suggestions.

[1588] Examples:

[1589] If the user shows a confused look on their face about which detergent to choose, the emotion engine will recognize that emotion and send it to the server.

[1590] The server receives the emotion data and sends suggestions to the terminal, including detailed usage instructions that are easy for the user to understand.

[1591] In this way, the present invention, which combines an emotion engine, not only suggests the most suitable detergent based on the detergent information the user has and the results of stain analysis, but also recognizes the user's emotions and adjusts the suggestions based on them, providing a more user-friendly system.

[1592] The processing flow will be explained below.

[1593] Step 1:

[1594] The user launches the application and opens the detergent registration screen. The user manually enters detergent information (such as name, manufacturer, and purpose) or takes a photo of the detergent label using the device's camera.

[1595] Step 2:

[1596] The device captures the label image and uses image recognition to extract the necessary information from the label.

[1597] Step 3:

[1598] The device sends the extracted detergent information to the server, which then associates the information with the user's account and stores it in a database.

[1599] Step 4:

[1600] The user finds a dirty spot and takes a photo of the dirt using the device's camera.

[1601] Step 5:

[1602] The device preprocesses the image of the dirt it has taken, specifically by adjusting the color and removing noise, and then sends the preprocessed image to the server.

[1603] Step 6:

[1604] The server uses an AI model to analyze the received image, which identifies the type of stain in the image (e.g., coffee stain, grease stain, etc.).

[1605] Step 7:

[1606] The server compares the analysis results with the detergent information registered by the user, selects the optimal detergent, and sends that information to the device.

[1607] Step 8:

[1608] The terminal displays the suggested detergent information received from the server to the user, and if the user does not have the suggested detergent in stock, also displays a link to an e-commerce site.

[1609] Step 9:

[1610] The user can enable the emotion engine in the settings of the application.

[1611] Step 10:

[1612] The device acquires the user's emotional data using a camera and microphone. The device captures the user's facial expressions with the camera and records their voice with the microphone.

[1613] Step 11:

[1614] The device analyzes the emotion data to determine the user's current emotional state, a process that identifies emotions such as stress, frustration, and satisfaction.

[1615] Step 12:

[1616] The device sends the analyzed emotional data to the server, which then customizes the detergent recommendations based on the emotional data received.

[1617] Step 13:

[1618] The server tailors its suggestions depending on the user's emotional state, for example, providing suggestions with detailed instructions if the user is confused.

[1619] Step 14:

[1620] The terminal displays the customized suggestions received from the server to the user, allowing the user to receive appropriate suggestions that match their emotions.

[1621] For example, if a user registers "clothing detergent" and takes a photo of a coffee stain on a carpet, the server will suggest "clothing detergent." If the user shows a confused expression, the device will recognize the emotion, and the server will provide detailed instructions for use, which the device will then display.

[1622] Example 2

[1623] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1624] Conventional detergent recommendation systems simply make recommendations based on the type of dirt and the detergent information registered by the user, without taking the user's feelings into consideration. This has led to issues such as users being confused by the recommendations or not knowing how to properly operate the system. Furthermore, when multiple detergents are applicable to a particular dirt, the system is unable to provide appropriate advice to help the user make the optimal selection.

[1625] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using the camera of the terminal, means for identifying the type of stain by analyzing the photographed image of the stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in the user's inventory, and means for recognizing the user's emotional state and customizing the suggestions based on the emotion. This makes it easier to deal with cases where the user is confused about which detergent to select or does not know how to operate the system, and makes it possible to provide a user-friendly system.

[1626] "User" refers to an individual or organization that accesses the system and registers detergent information, photographs stains, receives suggestions, etc.

[1627] A "terminal" refers to a computer device that is equipped with devices such as a camera and microphone and that is operated by a user. Examples include smartphones, tablets, and PCs.

[1628] "Server" refers to a computer system that receives information sent from a terminal and analyzes, stores, and provides the data.

[1629] "Detergent information" refers to detailed information such as the detergent's name, manufacturer, and purpose. This data indicates the characteristics of the detergent the user owns.

[1630] "Camera" refers to a device for capturing images, including those built into a device and those connected externally.

[1631] "Soil" refers to impurities or stains on the surface of an object that needs to be cleaned, such as food stains, mud stains, or oil stains.

[1632] "Image recognition technology" refers to technology that automatically extracts and analyzes specific information from captured images, including character recognition and object identification.

[1633] "Preprocessing" refers to the adjustment and processing techniques used to process images and data before analyzing them, including color adjustment and noise removal.

[1634] An "AI model" refers to a software model that uses artificial intelligence technology to analyze data and make predictions. Many of these models use machine learning algorithms.

[1635] "Emotion engine" refers to technology that uses the device's camera and microphone to analyze and recognize emotions from the user's facial expressions and voice.

[1636] An "e-commerce site" is a website that sells and purchases goods and services over the internet. Amazon is an example of this.

[1637] "Suggestion" refers to information that indicates the optimal detergent and its usage method to the user based on the analysis results.

[1638] "Customization" refers to adjusting and changing the content of suggestions according to the user's characteristics and situation.

[1639] The present invention provides a detergent recommendation system that can recognize a user's emotions and customize the content of recommendations based on those emotions. A specific embodiment of this system will be described below.

[1640] First, the user registers information about the detergent they own in the application. This process involves manual input and taking a photo of the detergent label using the device's camera. The device typically used here is a camera-equipped device such as a smartphone or tablet. Software such as TensorFlow and OpenCV is used for image recognition technology.

[1641] For example, there is a case where the user manually enters "laundry detergent" or a case where the user takes a picture of the label of "dishwashing detergent" with the device's camera and the information is automatically extracted. The information extracted from the label (name, manufacturer, purpose) is sent from the device over the Internet to a server. The server associates this information with the user's account and stores it in a database.

[1642] Next, if the user finds a stain that needs cleaning, they take a photo of it with the device's camera. The captured image undergoes preprocessing, such as color adjustment and noise removal, before being sent to the server. Image processing libraries such as OpenCV are used for preprocessing.

[1643] The server uses an AI model to analyze the received image. Common AI models used here include TensorFlow and PyTorch. The AI ​​identifies stains in the image and determines the type of stain (food stains, oil stains, mud stains, etc.). The analysis results are compared with the user's detergent information stored in the server's database, and if a suitable detergent is found, that information is generated as a suggestion to the user.

[1644] For example, if the server analyzes the image and determines that it is a coffee stain, it can use the user's registration information to determine that "laundry detergent" is suitable for coffee stains. The server then sends this information to the user's device. The suggestion includes a message such as "Use laundry detergent." If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site (e.g., Amazon) so the user can purchase it directly.

[1645] The system also features an emotion engine, which uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize their emotional state (e.g., stress, satisfaction, dissatisfaction, etc.). Emotion recognition uses Microsoft Azure's Emotion API and IBM Watson. Emotion data is sent to a server, which then uses it to further customize the suggestions.

[1646] For example, if a user shows a confused expression about which detergent to choose, the emotion engine will recognize that emotion and the server will send a suggestion to the device, including specific and detailed instructions for use. In this way, the system helps users avoid confusion about which detergent to choose and how to use it.

[1647] Example prompts for generative AI models

[1648] Detergent Information

[1649] Name: Laundry detergent

[1650] Manufacturer: ABC Company

[1651] Uses: stains on clothing

[1652] Dirt information

[1653] Type: Coffee stains

[1654] Part: Carpet

[1655] User Emotion: Confused

[1656] Generate a proposal.

[1657] This invention not only proposes the optimum detergent based on the detergent information and stain analysis results of the user, but also recognizes the user's emotions and adjusts the proposal content based on them, thereby providing a more user-friendly system.

[1658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1659] Step 1: Enter and register detergent information

[1660] Users use the application to register information about the detergent they own. They can either manually enter the information or use their device's camera to take a photo of the detergent label. When taking a photo of the label, the device uses image recognition technology to extract information such as the detergent's name, manufacturer, and purpose from the label. This information is sent to the server in JSON format.

[1661] Input: Detergent information manually entered by the user or detergent label image taken with a camera

[1662] Data processing: Extracting text information from images using image recognition technology

[1663] Output: Detergent information in JSON format

[1664] Specific behavior:

[1665] The user enters detergent information into the application.

[1666] The user takes a picture of the detergent label with the camera.

[1667] The device uses OpenCV to analyze the label image and extract text information.

[1668] The extracted information is converted into JSON format and sent to the server.

[1669] Step 2: Save your detergent information

[1670] The server parses the received detergent information and stores it in a database, where it is associated with the user's account and used to provide future recommendations.

[1671] Input: Detergent information in JSON format

[1672] Data processing: Parse and save to database

[1673] Output: Detergent information stored in the database

[1674] Specific behavior:

[1675] The server receives the data in JSON format.

[1676] The parser parses the JSON data and extracts the detergent information.

[1677] Generate and execute SQL queries to insert the extracted information into a database.

[1678] Step 3: Capture and pre-process stain images

[1679] When a user finds a stain that needs cleaning, they take a photo of it with their device's camera. Once the image is captured, the device performs pre-processing such as color adjustment and noise removal, which allows the server to analyze the image more accurately.

[1680] Input: Image of dirt

[1681] Data processing: color adjustment and noise removal

[1682] Output: Pre-processed stain image

[1683] Specific behavior:

[1684] The user takes a picture of the stain using the device's camera.

[1685] The device uses OpenCV to apply color adjustment and noise reduction filters.

[1686] Encode the preprocessed image in binary format.

[1687] Step 4: Submit and analyze the stain image

[1688] The pre-processed image is sent to a server, which uses an AI model to analyze the image and identify the type of stain (e.g., food stain, oil stain, mud stain, etc.).

[1689] Input: Preprocessed stain image

[1690] Data processing: Image analysis using AI models

[1691] Output: Analysis results including the type and characteristics of the dirt

[1692] Specific behavior:

[1693] The preprocessed image is sent to the server.

[1694] The server analyzes the image using a TensorFlow model.

[1695] The type and characteristics of the dirt are identified and saved as analysis results.

[1696] Step 5: Recommending the right detergent

[1697] The server compares the analysis results with the detergent information in the user's database to identify the optimal detergent, and then sends the identified detergent information to the user's device.

[1698] Input: Dirt analysis results, user detergent information

[1699] Data processing: matching and identifying the best detergent

[1700] Output: A message containing the suggestion

[1701] Specific behavior:

[1702] The server compares the stain analysis results with the user's detergent information.

[1703] Identify the best detergent and generate recommendations.

[1704] The proposal is encoded in JSON format and sent to the user's device.

[1705] Step 6: View Offer and Purchase Options

[1706] The terminal displays the received suggestions to the user, and if the suggested detergent is not in the user's inventory, the system provides a link to an e-commerce site so the user can purchase it directly.

[1707] Input: Proposal

[1708] Data processing: Converting data into a display format

[1709] Output: The offer and purchase link shown to the user

[1710] Specific behavior:

[1711] The device receives the proposal.

[1712] The suggestions are presented in a user-friendly format.

[1713] If appropriate, links to e-commerce sites are also provided.

[1714] Step 7: Emotion recognition and personalized suggestions

[1715] The device analyzes the user's facial expressions and voice to recognize their emotions, and the emotional data is sent to the server, which then uses this information to further customize the suggestions.

[1716] Input: User's facial expressions and voice

[1717] Data processing: sentiment analysis and data transmission

[1718] Output: Customized suggestions

[1719] Specific behavior:

[1720] The device uses a camera and microphone to capture facial expressions and voice.

[1721] Analyze emotions using an emotion engine (such as Microsoft Azure's Emotion API).

[1722] Emotion data is sent to the server.

[1723] The server regenerates the proposal content based on the received emotion data and transmits it to the terminal.

[1724] (Application example 2)

[1725] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1726] Conventional detergent recommendation systems require users to register their detergent information and analyze images of stains to suggest the optimal detergent. However, because they do not take the user's emotional state into account, the system is unable to make the recommendations easy for users to understand. Furthermore, customers may feel stressed or confused when receiving appropriate recommendations in physical stores, which prevents improved satisfaction. To solve these problems, a system is needed that recognizes the user's emotional state and customizes the recommendations based on that state.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1728] In this invention, the server includes means for registering information about detergents owned by the user, means for photographing stains using a camera of the terminal, means for analyzing the photographed image of the stains to identify the type of stain, means for suggesting an optimal detergent based on the information about the detergent owned by the user, means for providing an option to purchase an appropriate detergent from an e-commerce site if the suggested detergent is not in stock with the user, means for recognizing the emotional state of the user and customizing the suggestion content based on the emotion, means for recognizing the user's emotion and an image of the stain using smart glasses, and means for making detailed suggestions including detergent usage procedures based on the recognized emotional state. This enables appropriate and easy-to-understand suggestions according to the emotional state of the user.

[1729] "User" means an individual or entity that uses the system to receive detergent suggestions.

[1730] The "means for registering information about detergents owned" is a function that allows a user to input or scan information about detergents that the user owns into the system and store the information in a database.

[1731] A "terminal" is a device equipped with input devices such as a camera and a microphone, and is a computer or smartphone that executes the functions of the system.

[1732] The "means for photographing the dirt using a camera" is a method for capturing an image of the dirt using a camera mounted on the terminal.

[1733] The "means for analyzing the image of the dirt and identifying the type of dirt" is a function for analyzing the captured image of the dirt and identifying the type of dirt from the results.

[1734] The "means for suggesting the optimum detergent" is a function that recommends to the user the detergent that will most effectively remove the stain, based on the type of stain and information about the detergent the user owns.

[1735] "Means for providing the option to purchase suitable detergent from an e-commerce site" refers to a function that provides a link or information that allows the user to purchase the detergent online if the suggested detergent is not in the user's inventory.

[1736] The "means for recognizing emotional state" is a function that uses the device's camera and microphone to analyze emotions from the user's facial expressions and voice and obtain the results.

[1737] The "means for customizing the suggestions" is a function that adjusts detergent suggestions and usage procedures according to the recognized emotional state of the user, making them easier for the user to understand.

[1738] "Smart glasses" are wearable devices equipped with cameras and monitors, and are used to analyze the user's visual information and emotions.

[1739] The "means for providing detailed suggestions" is a function that provides the user with specific instructions and additional information on how to remove stains.

[1740] This invention is a system that registers detergent information owned by a user, analyzes images of stains, and recommends the most suitable detergent, and adds a function to recognize the emotional state of the user and customize the recommendations based on that. Specific embodiments for implementing this system are described below.

[1741] 1. Registering detergent information

[1742] First, the user registers information about the detergent they own in the system. This can be done by scanning the product's barcode or by manually entering the information. The user can also use the device's camera to take a photo of the detergent label and extract the necessary information using image recognition technology. The extracted information is sent to the server and stored in association with the user's account.

[1743] 2. Photograph the stain and send the image

[1744] Next, if the user finds a stain, they take a photo of it with their device's camera. The image is pre-processed on the device, with color adjustment and noise removal performed, and then the processed image is sent to the server.

[1745] 3. AI-based stain analysis

[1746] The server is equipped with a generative AI model that analyzes the received images and identifies the stains in the image. The AI ​​identifies the type of stain and compares the results with the user's registered information stored in the server's database. If a suitable detergent is found, it is generated as a recommendation for the user.

[1747] 4. Suggestions and Recommendations

[1748] Based on the analysis results, the server sends the suggested detergent information to the terminal. If the user does not have the suggested detergent in stock, the system also provides a link to an e-commerce site so that the user can purchase it directly.

[1749] 5. Use of Emotion Engine

[1750] Furthermore, the system is equipped with an emotion engine to recognize the user's emotions. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state. Based on this emotion data, the server further customizes the suggestions.

[1751] Hardware and Software Description

[1752] Devices used include smartphones, tablets, and smart glasses, with Google Glass and Microsoft HoloLens being particularly suitable.

[1753] OpenCV is used for image processing, and Azure Emotion API and EmotionRecognizer library are used for emotion recognition.

[1754] A generative AI model is used to analyze the dirt, and when a prompt is sent to analyze the image, the data is uploaded to a server in the cloud.

[1755] Specific examples

[1756] For example, consider a scenario in which a store staff member uses smart glasses to interact with a customer in a brick-and-mortar store. If the customer asks with a confused expression, "I don't know which detergent to use," the camera and microphone in the smart glasses will recognize the customer's emotional state. If the store staff member takes a photo of the stain brought in by the customer and the system analyzes that the stain is a coffee stain, the system will make a suggestion including detailed instructions, such as, "Use this laundry detergent. Here are the instructions for use."

[1757] Prompt Sentence Examples

[1758] "Analyze this image to identify the type of dirt."

[1759] "If your customer is confused, show them the suggestion with detailed instructions."

[1760] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1761] Step 1:

[1762] Register detergent information owned by the user.

[1763] Users can scan the detergent's barcode using their device's camera or manually enter the information. The device then uses image recognition technology to analyze the barcode and extract the necessary information (name, manufacturer, and purpose). The extracted information is then sent to a server and stored in association with the user's account.

[1764] Input: Detergent barcode image or manually entered information.

[1765] Output: Detergent information stored in the user account.

[1766] Step 2:

[1767] Take a picture of the stain with your device's camera.

[1768] When a user finds a stain, they take a photo of it using their device's camera. The device then pre-processes the image, adjusting color and removing noise, and sends the processed image to the server.

[1769] Input: A photographed image of the dirt.

[1770] Output: Preprocessed image data.

[1771] Step 3:

[1772] The server analyzes the received image of the dirt.

[1773] The server uses the generative AI model to analyze the pre-processed image, which identifies the type of stain in the image (e.g., coffee stain, oil stain, etc.) and stores the identified stain type in a database.

[1774] Input: Preprocessed stain image.

[1775] Output: Identified soil type information.

[1776] Step 4:

[1777] The server recommends the most suitable detergent based on the user's detergent information.

[1778] The server compares the identified stain type with the detergent information of the user and suggests the most suitable detergent. The suggestion is sent to the terminal and displayed to the user.

[1779] Input: Identified stain type information, user detergent information.

[1780] Output: Recommendations for the best detergent.

[1781] Step 5:

[1782] Recognize the user's emotional state.

[1783] The device's camera and microphone are used to analyze the user's facial expressions and voice, and the emotion engine recognizes the user's emotional state (e.g., confusion, satisfaction, stress, etc.). The recognized emotion data is sent to the server.

[1784] Input: User's facial and voice data.

[1785] Output: Recognized emotional state data.

[1786] Step 6:

[1787] The server customizes the suggestions based on the emotion data.

[1788] The server receives the emotional state data and tailors the suggestions. For example, if the user is confused, the system generates suggestions that include detailed instructions and links to e-commerce sites. These customized suggestions are sent to the device and displayed to the user.

[1789] Input: Emotional state data, optimal detergent recommendation.

[1790] Output: A customized proposal.

[1791] Step 7:

[1792] Recognizing dirt and emotions using smart glasses.

[1793] Store staff use smart glasses to recognize customer emotions and images of dirt. The smart glasses' cameras and microphones capture the customer's facial expressions and voice and send the data to a server, which analyzes it and generates suggestions.

[1794] Input: Images of dirt captured by smart glasses and customer emotion data.

[1795] Output: Recognized stain types and emotion data.

[1796] Step 8:

[1797] The server will provide detailed suggestions.

[1798] Based on the recognized emotional state, the server generates detailed recommendations, including instructions for using the detergent, which are displayed to staff through smart glasses and explained to the customer.

[1799] Input: Recognized dirt type, emotional state data.

[1800] Output: Detailed proposal.

[1801] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1802] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1803] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1804] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1805] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1806] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1807] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1808] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1809] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1810] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1811] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1812] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1813] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1814] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1815] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1816] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1817] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1818] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1819] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1820] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1821] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1822] The following is further disclosed regarding the above embodiment.

[1823] (Claim 1)

[1824] A means for registering information about detergents owned by a user;

[1825] A means for photographing the dirt using a camera of the terminal;

[1826] A means for analyzing the captured image of the dirt to identify the type of dirt;

[1827] A means for suggesting an optimal detergent based on information about detergents possessed by a user;

[1828] means for providing the user with the option to purchase a suitable detergent from the e-commerce site if the suggested detergent is not in stock;

[1829] A system including:

[1830] (Claim 2)

[1831] 2. The system according to claim 1, wherein detergent information is registered using an image recognition function.

[1832] (Claim 3)

[1833] The system according to claim 1, wherein the captured image of the dirt is preprocessed and transmitted to the server.

[1834] "Example 1"

[1835] (Claim 1)

[1836] A means for registering information about detergents owned by a user;

[1837] A means for photographing the dirt using a camera of the terminal;

[1838] means for pre-processing the captured image of the dirt and transmitting it to a server;

[1839] A means for analyzing the captured image of the dirt to identify the type of dirt;

[1840] A means for suggesting an optimal detergent based on information about detergents possessed by a user;

[1841] means for providing the user with the option to purchase a suitable detergent from the e-commerce site if the suggested detergent is not in stock;

[1842] A system including:

[1843] (Claim 2)

[1844] 2. The system according to claim 1, wherein detergent information is registered using an image recognition function.

[1845] (Claim 3)

[1846] The system of claim 1, wherein the system analyzes captured images of dirt using an AI model.

[1847] "Application Example 1"

[1848] (Claim 1)

[1849] A means for registering information about cleaning agents owned by a user;

[1850] A means for photographing the dirt using a camera of the terminal;

[1851] using a generative AI model to analyze captured images of the dirt and identify the type of dirt;

[1852] A means for suggesting an optimal cleaning agent based on information on cleaning agents possessed by the user;

[1853] means for providing the user with the option to purchase a suitable cleaning agent from an online sales platform if the suggested cleaning agent is not in the user's inventory;

[1854] A means for acquiring product information by scanning a product barcode and proposing an optimal cleaning agent based on the product information;

[1855] A system including:

[1856] (Claim 2)

[1857] 2. The system according to claim 1, wherein information about the cleaning agent is registered using an image recognition function.

[1858] (Claim 3)

[1859] The system according to claim 1, wherein the captured image of the dirt is preprocessed and transmitted to the server.

[1860] (Claim 4)

[1861] 10. The system of claim 1, further comprising the capability to scan product barcodes.

[1862] "Example 2: Combining Emotion Engines"

[1863] (Claim 1)

[1864] A means for registering information about detergents owned by a user;

[1865] A means for photographing the dirt using a camera of the terminal;

[1866] A means for analyzing the captured image of the dirt to identify the type of dirt;

[1867] A means for suggesting an optimal detergent based on information about detergents possessed by a user;

[1868] means for providing the user with the option to purchase a suitable detergent from the e-commerce site if the suggested detergent is not in stock;

[1869] means for recognizing the emotional state of the user and customizing suggestions based on the emotional state;

[1870] A system including:

[1871] (Claim 2)

[1872] 2. The system according to claim 1, wherein detergent information is registered using an image recognition function.

[1873] (Claim 3)

[1874] The system according to claim 1, wherein the captured image of the dirt is preprocessed and transmitted to the server.

[1875] "Application example 2 when combining emotion engines"

[1876] (Claim 1)

[1877] A means for registering information about detergents owned by a user;

[1878] A means for photographing the dirt using a camera of the terminal;

[1879] A means for analyzing the captured image of the dirt to identify the type of dirt;

[1880] A means for suggesting an optimal detergent based on information about detergents possessed by a user;

[1881] means for providing the user with the option to purchase a suitable detergent from the e-commerce site if the suggested detergent is not in stock;

[1882] means for recognizing the emotional state of the user and customizing suggestions based on the emotional state;

[1883] A system including:

[1884] (Claim 2)

[1885] 2. The system according to claim 1, wherein detergent information is registered using an image recognition function.

[1886] (Claim 3)

[1887] The system according to claim 1, wherein the captured image of the dirt is preprocessed and transmitted to the server.

[1888] (Claim 4)

[1889] The system of claim 1, wherein smart glasses are used to recognize user emotions and images of dirt.

[1890] (Claim 5)

[1891] 10. The system of claim 1, wherein the system provides detailed recommendations, including instructions for using the detergent, based on the recognized emotional state. [Explanation of symbols]

[1892] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for registering information about detergents owned by a user; A means for photographing the dirt using a camera of the terminal; A means for analyzing the captured image of the dirt to identify the type of dirt; A means for suggesting an optimal detergent based on information about detergents possessed by a user; means for providing the user with the option to purchase a suitable detergent from the e-commerce site if the suggested detergent is not in stock; A system including:

2. 2. The system according to claim 1, wherein information about the detergent is registered using an image recognition function.

3. The system according to claim 1, wherein the captured image of the dirt is pre-processed and transmitted to the server.

Images