System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an image of clothing owned by a user and analyzing the image to extract feature information of the clothing; means for receiving information about food purchased by the user and analyzing the information to store the information in a predetermined database; means for obtaining temperature data and suggesting suitable clothing coordination based on the temperature data and the feature information of the clothing of the user; and means for suggesting a predetermined menu based on predetermined food life data and generating a list of insufficient food based on the menu.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, people have limited time to think about coordinating their daily outfits and planning meals amid their busy daily lives. This is a major challenge, especially for those who love fashion but lack confidence in their ability to combine outfits, or who struggle with planning daily meals. Furthermore, as the number of clothes they own and the ingredients they buy increase, it's not uncommon for them to become unable to properly manage them, leading to inefficient behavior. Specifically, they face problems such as not being able to coordinate outfits appropriately because they don't know what clothes are in their closet, or buying the same food twice because they don't know what ingredients are in their refrigerator. A system is needed to solve these problems, efficiently manage clothing and ingredients, and enrich daily life. [Means for solving the problem]

[0005] The present invention provides a system including: means for receiving images of clothing owned by a user and analyzing the images to extract characteristic information about the clothing; means for receiving information about food purchased by the user and analyzing the information to store it in a predetermined database; means for acquiring temperature data and suggesting appropriate clothing coordination based on the temperature data and the characteristic information about the user's clothing; means for suggesting predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus. Specifically, the system further includes means for analyzing predetermined receipt images to extract information about foods purchased by the user, and means for suggesting predetermined menus using a learning algorithm based on the user's past consumption data and current inventory data. In this way, the system can improve the efficiency of the user's daily life and save time and effort.

[0006] "User" refers to anyone who uses this system, including individuals who want to coordinate their daily clothing, manage ingredients, and request menu suggestions.

[0007] "Receiving an image" refers to the system acquiring image data of the clothes or receipt provided by the user.

[0008] "Analyzing and extracting characteristic information about clothing" means using image recognition technology to identify attributes such as clothing type, color, and design from the received image data and extract that information.

[0009] "Food information" refers to data such as the name, quantity, and price of the food purchased by the user.

[0010] "Saving in a database" means recording the analyzed information in a database within the system and retaining it for later use.

[0011] "Temperature Data" refers to information about current or predicted temperatures obtained by the system from an external source.

[0012] "Suggesting outfits" means that the system presents appropriate clothing combinations to the user based on temperature data and clothing characteristic information.

[0013] "Food life data" refers to detailed information related to food, such as expiration dates, frequency of use, and purchase history.

[0014] "Menu suggestion" means that the system presents the user with a menu of appropriate dishes based on the ingredients they have on hand.

[0015] "Generating a list of missing foods" refers to creating a list of foods that are required to execute the proposed menu but that are not currently in possession.

[0016] "Analyzing the receipt image" means using character recognition technology to extract detailed information about the purchased items from the image of the receipt provided by the user.

[0017] "Consumption Data" refers to information about foods a user has consumed in the past and the amount of such food consumed.

[0018] "Inventory data" refers to information about the types and quantities of food items currently owned by a user.

[0019] A "learning algorithm" refers to a model of a computer program that uses past data to predict future actions and outcomes. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] MODE FOR CARRYING OUT THE INVENTION

[0042] This invention is a system for efficiently managing a user's daily life, saving time and effort. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[0043] Closet clothing management

[0044] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0045] Management of purchased ingredients

[0046] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the received receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0047] Coordination suggestions

[0048] The server receives temperature data from an external device in real time. It compares the temperature data with the user's clothing characteristics information and suggests appropriate outfits to the user. The device then displays the optimal outfits to the user. The user can then review the suggested outfits and make fine adjustments if necessary.

[0049] Menu suggestions and shopping list generation

[0050] The server uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to plan their shopping efficiently.

[0051] Specific examples

[0052] Example 1: Morning Outfit

[0053] 1. The user wakes up in the morning and opens the app on their smartphone.

[0054] 2. The device requests temperature data from the server.

[0055] 3. The server obtains real-time temperature data from an external API and compares it with the user's closet database to select the optimal outfit.

[0056] 4. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[0057] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0058] Example 2: Dinner menu suggestions

[0059] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0060] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0061] 3. The server adds the food information to the database, updates the refrigerator inventory, and uses a learning algorithm to generate data to suggest menus for the next day.

[0062] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[0063] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0064] This system allows users to efficiently manage the items in their closets and refrigerators, freeing them from small everyday worries and allowing them to live a richer life.

[0065] The processing flow will be explained below.

[0066] Closet clothing management

[0067] Step 1:

[0068] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[0069] Step 2:

[0070] The terminal receives the uploaded image.

[0071] Step 3:

[0072] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[0073] Step 4:

[0074] The terminal transmits the analysis results to the server.

[0075] Step 5:

[0076] The server stores the received clothing data in a database for each user.

[0077] Management of purchased ingredients

[0078] Step 1:

[0079] Users take a photo of the receipt for the food they purchased and upload it to the app.

[0080] Step 2:

[0081] The terminal receives the uploaded receipt image.

[0082] Step 3:

[0083] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[0084] Step 4:

[0085] The terminal transmits the analyzed food information to the server.

[0086] Step 5:

[0087] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[0088] Coordination suggestions

[0089] Step 1:

[0090] The server obtains temperature data for the user's location in real time (using an external API).

[0091] Step 2:

[0092] The server compares the acquired temperature data with the clothing information recorded in the user's closet database.

[0093] Step 3:

[0094] The server selects the best outfit for the temperature.

[0095] Step 4:

[0096] The terminal displays the coordination suggestions received from the server on the user's app.

[0097] Step 5:

[0098] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[0099] Menu suggestions and shopping list generation

[0100] Step 1:

[0101] The server refers to the user's past consumption data based on the refrigerator inventory data.

[0102] Step 2:

[0103] The server uses a learning algorithm to generate the next day's menu.

[0104] Step 3:

[0105] The server creates a list of missing foods based on the generated menu.

[0106] Step 4:

[0107] The device displays the menu suggestions and shopping list received from the server in the user's app.

[0108] Step 5:

[0109] The user reviews the suggested menu and plans their shopping based on the list of missing foods.

[0110] Through the above steps, the system can efficiently assist users in their daily lives and save them a lot of time and effort.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In modern life, managing clothing and food inventory, as well as suggesting clothing coordination according to the weather and planning daily meal menus, are complicated tasks that require time and effort. Furthermore, managing these pieces of information individually is inefficient, so a system that can manage them all comprehensively is needed.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and suggesting appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for suggesting predetermined menus based on predetermined food life data and generating a list of missing foods based on the menus, means for receiving receipt images photographed by the user and extracting purchased food information using OCR technology, and means for suggesting menus using a learning algorithm based on the user's past consumption data and current inventory data and generating a list of missing ingredients, thereby enabling users to comprehensively and efficiently manage various aspects of their daily lives.

[0116] "User" refers to any individual or organization that uses this system.

[0117] "Clothing image" refers to image data of clothing owned by the user.

[0118] "Feature information" refers to attribute information such as clothing type, color, and design extracted through image analysis.

[0119] "Food information" refers to detailed information about the food purchased by the user, such as the food name, quantity, purchase date and time, etc.

[0120] A "database" refers to an information management system for systematically storing analyzed information.

[0121] "Temperature data" refers to information about the current temperature obtained from an external weather data provider.

[0122] "Coordination suggestions" refers to a function that suggests optimal clothing combinations based on the user's clothing characteristics information and temperature data.

[0123] "Food life data" refers to life cycle data about food, such as expiration dates and inventory information.

[0124] "Menu suggestion" refers to a function that suggests optimal meal menus based on the user's inventory information and consumption data.

[0125] "List of missing foods" refers to a list of additional foods that need to be purchased in order to implement the proposed menu.

[0126] "Receipt image" refers to image data of the receipt for the purchased product photographed by the user.

[0127] "OCR technology" is an abbreviation for optical character recognition technology, and refers to the technology of extracting character data from images.

[0128] "Learning algorithm" refers to a machine learning technique that learns from a user's past consumption data and current inventory data to make predictions and suggestions.

[0129] This system efficiently manages users' daily lives, saving them time and effort. This system has four main functions: managing clothes in the closet, managing purchased ingredients, suggesting outfits based on temperature data, and creating menus and shopping lists.

[0130] Closet clothing management

[0131] Users take photos of their clothing using their smartphones or computers and upload them to a dedicated app. The app uses well-known image processing technologies such as TensorFlow and OpenCV. The device receives the uploaded image and uses an image recognition algorithm to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is then sent to a server and stored in a database for each user.

[0132] Management of purchased ingredients

[0133] Users take a photo of the receipt for the food they purchased and upload it to a dedicated app. The device then analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[0134] Coordination suggestions

[0135] The server obtains temperature data in real time from an external weather data provider (e.g., OpenWeatherMap API). It compares this temperature data with clothing data stored in the user's closet and suggests optimal outfits for the user. The device displays the outfit suggestions received from the server to the user. The user can review the suggested outfits and make fine adjustments as necessary.

[0136] Menu suggestions and shopping list generation

[0137] After returning home from the supermarket, the user takes a photo of their purchase receipt and uploads it to a dedicated app. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server. The server adds the food information to a database and updates the refrigerator inventory information. Using a learning algorithm (e.g., Scikit-learn), the system suggests a menu for the next day based on the user's past consumption data and current inventory data. The suggested menu makes the most of existing ingredients and generates a list of any missing ingredients. The generated list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to efficiently plan their shopping.

[0138] Specific examples

[0139] Example 1: Morning Outfit

[0140] 1. The user wakes up in the morning and opens the app on their smartphone.

[0141] 2. The device requests temperature data from the server.

[0142] 3. The server obtains real-time temperature data from an external weather data provider and compares it with the user's closet database to select the optimal outfit.

[0143] 4. The terminal displays the outfit suggestions received from the server to the user (e.g., "white shirt and blue jeans").

[0144] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0145] Example 2: Dinner menu suggestions

[0146] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0147] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0148] 3. The server adds the food information to the database, updates the refrigerator inventory, and generates the next day's menu (e.g., "chicken curry") using a learning algorithm.

[0149] 4. The terminal displays the menu suggestions received from the server and the list of missing foods to the user.

[0150] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0151] Prompt Sentence Examples

[0152] 1. "What's the temperature today?" → Server: "It's 20 degrees."

[0153] 2. "What is appropriate clothing for this temperature?" → Server: "I recommend a white shirt and blue jeans."

[0154] 3. "Based on what ingredients you have in the fridge, what would be a good menu item for tomorrow?" → Server: "I think it would be good to have the ingredients needed to make chicken curry."

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Closet clothing management

[0157] Step 1:

[0158] Users take photos of their own clothes using a smartphone or computer and upload them to a dedicated app. The input is the photo of the clothes taken by the user, and the output is the image file uploaded to the app.

[0159] Step 2:

[0160] The terminal receives the uploaded image. The input is the image file from the user, and the output is the received image data. At this point, a communication protocol (e.g., HTTP) is used for receiving.

[0161] Step 3:

[0162] The device analyzes images using image recognition algorithms such as TensorFlow and OpenCV to extract feature information such as the type, color, and design of clothing. For example, it can obtain information such as "white shirt" or "blue jeans" from an image. The input is the received image data, and the output is the extracted feature information.

[0163] Step 4:

[0164] The terminal sends the analysis results to the server. The input is the feature information, and the output is the analyzed data sent via the network.

[0165] Step 5:

[0166] The server stores the transmitted feature information in a database for each user. For example, "User A's closet" is registered as "white shirt" and "blue jeans." The input is the transmitted analyzed data, and the output is the updated database.

[0167] Management of purchased ingredients

[0168] Step 1:

[0169] The user takes a photo of the receipt for the food they purchased and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[0170] Step 2:

[0171] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the received receipt image, and the output is the food information extracted through analysis. For example, information such as "milk" and "eggs" is extracted from the receipt.

[0172] Step 3:

[0173] The terminal sends the extracted information to the server. The input is the extracted food information, and the output is the food data sent via the network.

[0174] Step 4:

[0175] The server stores the food information in a database. The input is the food information sent, and the output is the updated database. For example, add "milk" and "eggs" to "User A's food inventory."

[0176] Coordination suggestions

[0177] Step 1:

[0178] The server sends a request to an external weather data provider (e.g., OpenWeatherMap API) to obtain real-time temperature data. The input is the weather data request, and the output is the obtained temperature data.

[0179] Step 2:

[0180] The server compares the acquired temperature data with the user's closet database. For example, it selects an outfit suitable for a temperature of 20 degrees. The input is temperature data and closet data, and the output is a suggestion of the optimal outfit.

[0181] Step 3:

[0182] The server sends the proposed coordinates to the terminal. The input is the proposed coordinates, and the output is the coordination proposal sent over the network.

[0183] Step 4:

[0184] The terminal displays the coordinate suggestion received from the server to the user. The input is the suggested coordinate information, and the output is the coordinate displayed on the terminal.

[0185] Step 5:

[0186] The user can check the suggested outfits and either accept them as they are or adjust them themselves to choose the outfit. The input is the displayed outfit, and the output is the final outfit chosen by the user.

[0187] Menu suggestions and shopping list generation

[0188] Step 1:

[0189] After returning home from the supermarket, the user takes a photo of the receipt and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[0190] Step 2:

[0191] The device analyzes the received receipt image using OCR technology and extracts information about the purchased food. The input is the received receipt image, and the output is the food information extracted by the analysis.

[0192] Step 3:

[0193] The terminal sends the purchased food information to the server. The input is the extracted food information, and the output is the food data sent via the network.

[0194] Step 4:

[0195] The server adds the food information to the database and updates the refrigerator inventory information. The input is the submitted food information, and the output is the updated refrigerator inventory data.

[0196] Step 5:

[0197] The server uses a learning algorithm (e.g., Scikit-learn) to suggest the next day's menu based on the user's past consumption data and current inventory data. The input is consumption data and inventory data, and the output is the suggested menu.

[0198] Step 6:

[0199] The server generates a list of missing ingredients and sends it to the terminal. The input is the proposed menu and the output is the missing ingredients list.

[0200] Step 7:

[0201] The terminal displays the received menu suggestions and the list of missing foods to the user. The input is the suggested menu and the list of missing foods, and the output is the menu and list displayed on the terminal.

[0202] Step 8:

[0203] The user can then review the proposed menu and list of missing ingredients to efficiently plan their shopping. The input is the displayed menu and list, and the output is the user's final shopping plan.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Today's busy consumers are seeking systems that can efficiently manage the items in their closets and refrigerators and suggest optimal outfits and meal plans in their daily lives. They also have numerous needs for virtual store shopping experiences, such as suggestions for coordinating items with items they plan to purchase and smooth food inventory management. Conventional systems have difficulty meeting these needs in an integrated manner. Therefore, a new system that can comprehensively and efficiently support users' lives is needed.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, means for suggesting coordination with products that the user is considering purchasing in a virtual store, and means for managing information about purchased products and keeping track of the user's inventory. This allows the user to efficiently manage their daily closet and refrigerator, and enables optimal coordination suggestions and inventory management through a shopping experience in a virtual store.

[0209] "User" means an individual or corporation that uses this system.

[0210] "Clothing feature information" is data such as clothing type, color, and design extracted using image recognition technology.

[0211] "Food life data" refers to information on the type, quantity, and date of consumption of food purchased and consumed.

[0212] A "menu" is a meal plan suggested to a user, including the types and amounts of ingredients used.

[0213] A "candidate product for purchase" is a product that the user is considering purchasing in the virtual store.

[0214] "Temperature data" is current temperature information obtained from an external weather information service.

[0215] "Coordination" is a suggestion on how to combine and apply clothing.

[0216] A "receipt image" is an image of a receipt that lists details of the products purchased by the user.

[0217] "Inventory" refers to the quantity and type of items, such as food and clothing, that a user has already purchased and possesses.

[0218] A "learning algorithm" is a computational method for analyzing past data and predicting future trends and patterns.

[0219] A "virtual store" is a virtual shopping mall where users can browse and purchase products online.

[0220] The "server" is a computer device that serves as the center of the system and stores, analyzes, and makes recommendations on data.

[0221] This invention is a system designed to enable users to efficiently manage their lifestyles. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[0222] First, a user uses a smartphone application to take a photo of their clothing and upload it to the application. The device analyzes the uploaded image and uses an image recognition algorithm (e.g., Google Cloud Vision API) to extract feature information such as the type, color, and design of the clothing. The extracted feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0223] Next, the user takes a photo of the receipt for the purchased food and uploads it to the application. The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows the server to keep track of the inventory status of the ingredients purchased by the user.

[0224] In addition, the server obtains real-time temperature data from external weather information services (e.g., OpenWeather API), compares this data with the user's clothing characteristics, and suggests appropriate outfits for the user. The suggested outfits are then displayed to the user on their device.

[0225] The server also uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is then sent to the device and displayed to the user.

[0226] In particular, in the virtual store, users can receive suggestions for coordinating products with potential purchases. Purchased product information is managed by the server and added to the user's inventory database. This feature enhances the virtual shopping experience.

[0227] Specific examples include suggestions for morning outfits and dinner menus. For example, when a user wakes up in the morning and opens a smartphone application, the device requests temperature data from the server, and the server compares the real-time temperature data with the user's closet information to select the optimal outfit and display it on the device.

[0228] When a user returns home from the supermarket and uploads their purchase receipt to the application, the device analyzes the received receipt image using OCR technology and sends the purchased food information to the server, which then analyzes the data and presents the user with the next day's menu and a list of any missing foods.

[0229] An example prompt is, "The user has provided an image of a dress they would like to purchase. Please suggest outfits that would go well with this dress. The temperature is 20 degrees and the user prefers a casual style. The user has the following items in their closet: a red jacket, black leather pants, and a white T-shirt. Which would go well together?"

[0230] In this way, this system efficiently manages the user's daily life and supports a more comfortable life.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] Users use a smartphone application to take a photo of their clothing and upload it to the application. This input data is an image file, which is received by the device.

[0234] Step 2:

[0235] The device analyzes the uploaded image file using an image recognition algorithm (e.g., Google Cloud Vision API) and extracts characteristic information such as the type of clothing, color, and design. The extracted characteristic information is output as data and sent to the server.

[0236] Step 3:

[0237] The server stores the received feature information in a database for each user, which is later used to suggest outfits.

[0238] Step 4:

[0239] The user takes a photo of the receipt for the food they purchased and uploads it to the application. This input data is also an image file, and the device receives it.

[0240] Step 5:

[0241] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted food information is output as data and sent to the server.

[0242] Step 6:

[0243] The server stores the received food information in a designated database, allowing the user to keep track of the inventory of ingredients purchased.

[0244] Step 7:

[0245] The server obtains real-time temperature data from an external weather information service (e.g., OpenWeather API). This temperature data is input data and is received by the server.

[0246] Step 8:

[0247] The server compares the temperature data with the user's clothing characteristics information and uses a generative AI model to suggest appropriate outfits. This output data is the outfit suggestion and is sent to the device.

[0248] Step 9:

[0249] The terminal displays the coordinated outfit suggestions received from the server to the user, who can then check the suggested outfits and either accept them or make minor adjustments.

[0250] Step 10:

[0251] The server uses the refrigerator's inventory data, past consumption data, and a learning algorithm to propose a menu for the next day. This output data is the menu proposal, and is sent to the terminal along with a list of missing ingredients.

[0252] Step 11:

[0253] The terminal displays the menu suggestions and the list of missing ingredients received from the server to the user, who then checks the suggested menu and list and makes a plan to purchase the necessary ingredients.

[0254] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0255] MODE FOR CARRYING OUT THE INVENTION

[0256] The present invention is a system for efficiently managing a user's daily life, saving time and effort. In particular, it has the function of recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions.

[0257] Closet clothing management

[0258] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0259] Management of purchased ingredients

[0260] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0261] Coordination suggestions

[0262] The server obtains temperature data from an external source in real time. It compares the temperature data with the user's clothing characteristics information and proposes appropriate outfits to the user. In addition, it uses an emotion engine to obtain the user's emotional information and integrates the temperature data with the emotional information to optimize the outfit proposals. This allows the device to display the optimal outfits to the user. The user can review the proposed outfits and make fine adjustments if necessary.

[0263] Menu suggestions and shopping list generation

[0264] The server references the user's past consumption data based on refrigerator inventory data. Using a learning algorithm, the server takes into account the user's emotional information obtained by the emotion engine when generating the next day's menu. The proposed menu maximizes the use of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the device and displayed to the user. The user can check the proposed menu and the list of missing foods to efficiently plan their shopping.

[0265] Specific examples

[0266] Example 1: Morning Outfit

[0267] 1. The user wakes up in the morning and opens the app on their smartphone.

[0268] 2. The device requests the current temperature data and the user's emotion information from the server.

[0269] 3. The server obtains real-time temperature data from an external API and simultaneously analyzes the user's emotional information using an emotion engine.

[0270] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[0271] 5. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[0272] 6. The user checks the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0273] Example 2: Dinner menu suggestions

[0274] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0275] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0276] 3. The server adds the food information to the database and updates the refrigerator inventory information. It uses an emotion engine to obtain the user's emotional information and uses a learning algorithm to generate the next day's menu.

[0277] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[0278] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0279] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions tailored to their emotions, and live a richer life.

[0280] The processing flow will be explained below.

[0281] Closet clothing management

[0282] Step 1:

[0283] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[0284] Step 2:

[0285] The terminal receives the uploaded image.

[0286] Step 3:

[0287] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[0288] Step 4:

[0289] The terminal transmits the analysis results to the server.

[0290] Step 5:

[0291] The server stores the received clothing data in a database for each user.

[0292] Management of purchased ingredients

[0293] Step 1:

[0294] Users take a photo of the receipt for the food they purchased and upload it to the app.

[0295] Step 2:

[0296] The terminal receives the uploaded receipt image.

[0297] Step 3:

[0298] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[0299] Step 4:

[0300] The terminal transmits the analyzed food information to the server.

[0301] Step 5:

[0302] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[0303] Coordination suggestions

[0304] Step 1:

[0305] The server obtains temperature data for the user's location in real time (using an external API).

[0306] Step 2:

[0307] The server uses an emotion engine to acquire and analyze the user's emotion information.

[0308] Step 3:

[0309] The server compares the acquired temperature data and emotion information with the clothing information recorded in the user's closet database.

[0310] Step 4:

[0311] The server selects the best outfit that matches the temperature and your emotions.

[0312] Step 5:

[0313] The terminal displays the coordination suggestions received from the server on the user's app.

[0314] Step 6:

[0315] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[0316] Menu suggestions and shopping list generation

[0317] Step 1:

[0318] The server refers to the user's past consumption data based on the refrigerator inventory data.

[0319] Step 2:

[0320] The server uses an emotion engine to acquire and analyze the user's emotion information.

[0321] Step 3:

[0322] The server uses a learning algorithm to generate the next day's menu, taking emotional information into account.

[0323] Step 4:

[0324] The server creates a list of missing foods based on the generated menu.

[0325] Step 5:

[0326] The device displays the menu suggestions and shopping list received from the server in the user's app.

[0327] Step 6:

[0328] Users can plan their shopping by reviewing suggested menus and a list of missing foods.

[0329] This system allows users to efficiently manage the items in their closets and refrigerators and receive optimal suggestions tailored to their mood.

[0330] Example 2

[0331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] In modern life, many users need to efficiently manage their daily lives to save time and effort. However, current systems have difficulty consistently managing clothing and food inventory, and suggesting appropriate outfits and menus. Furthermore, suggestions rarely reflect the user's mood or emotions, which prevents users from increasing their satisfaction. Therefore, there is a need for a system that can comprehensively support users' daily lives and make optimal suggestions based on their emotions.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0334] In this invention, the server includes means for acquiring emotional information of a user and optimizing clothing coordination suggestions based on the emotional information, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, and means for optimizing the menus using a learning algorithm based on the emotional information and the user's past consumption data and current inventory data, thereby enabling the user to efficiently manage the items in their closet and refrigerator and receive optimal suggestions based on their emotions.

[0335] "User" means an individual or organization that uses this system.

[0336] "Clothing images" refer to photographs or image data of clothing owned by the user.

[0337] "Means for analyzing images and extracting characteristic information about clothing" refers to algorithms and technologies that analyze images of clothing and extract information such as color, type, and design.

[0338] "Food information" refers to data such as the type, quantity, and price of food purchased by the user.

[0339] The "predetermined database" refers to a database system for storing information about a user's clothing and purchased food.

[0340] "Temperature Data" refers to data regarding local temperatures obtained from external weather information services.

[0341] "User emotion information" refers to data that shows the results of analyzing the user's emotions and moods.

[0342] "Specified food life data" refers to data regarding food storage conditions, expiration dates, etc.

[0343] "Means for suggesting menus and generating a list of missing foods" refers to algorithms and technologies that suggest appropriate cooking menus based on the ingredients in the refrigerator and the user's past consumption patterns, and that create a list of missing foods for that purpose.

[0344] A "learning algorithm" is an algorithm that learns patterns from past data and makes inferences and predictions about new data.

[0345] MODE FOR CARRYING OUT THE INVENTION

[0346] The present invention is a system for efficiently managing a user's daily life and saving time and effort. In particular, it has a function for recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions. Specific embodiments of this system will be described in detail below.

[0347] Closet clothing management

[0348] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms (such as TensorFlow or OpenCV) to analyze feature information such as the type, color, and design of the clothing. The analyzed feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0349] For example, if a user uploads a photo of a blue shirt, the device analyzes the image and sends the characteristic information that the shirt is blue to the server, which stores this information in the user's database.

[0350] Management of purchased ingredients

[0351] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0352] For example, when a user uploads a receipt, the device uses OCR technology to send the information about "milk," "eggs," and "bread" to the server, which adds this information to the database and updates the user's refrigerator inventory.

[0353] Coordination suggestions

[0354] The server acquires real-time temperature data using an external API (such as the OpenWeatherMap API). It then acquires the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The server compares the acquired temperature data and emotional information with the user's closet database and generates an appropriate outfit. The generated outfit suggestions are sent to the device and displayed to the user.

[0355] For example, if the server receives data indicating that the temperature is 28 degrees and the user is in good spirits, the server will suggest a white shirt and blue jeans. This suggestion will be sent to the device and displayed to the user.

[0356] Menu suggestions and shopping list generation

[0357] The server generates a menu based on refrigerator inventory data and the user's past consumption data. It also takes into account the user's emotional information using an emotion engine. A learning algorithm (such as Scikit-learn or TensorFlow) is used in the generation process. The proposed menu makes the most of existing ingredients, and any ingredients that are missing are generated as a list. This list is sent to the device and displayed to the user.

[0358] For example, if the server recognizes that the user has chicken, onions, and potatoes in the refrigerator and considers that the user feels like refreshing, it will suggest chicken curry as a menu item for the next day. It will also add missing spices to the shopping list. This list is sent to the terminal and displayed to the user.

[0359] Prompt Sentence Examples

[0360] "Write a Python program that uses temperature data and user emotion information to generate suggestions for morning outfits."

[0361] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions based on their emotions, and live a richer life.

[0362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0363] Step 1:

[0364] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The input is the image of the clothing taken. The device receives the uploaded image and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze the image for feature information such as the type, color, and design of the clothing. This analysis extracts feature information such as color, type, and design. The extracted feature information is sent to the server and stored in a database for each user. As an output, the feature information of the clothing is saved in a database for each user. For example, if a user uploads an image of a blue shirt, the device analyzes the image and extracts the feature information "blue shirt, long sleeves, size M" and sends it to the server.

[0365] Step 2:

[0366] The user takes a photo of the receipt for the food they purchased and uploads it to the app. The input is an image of the receipt for the purchased food. The device analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. Specifically, it extracts the product name, quantity, price, and other information listed on the receipt as text data. The extracted information is sent to the server and stored in a database. As an output, the user's food purchase information is added to the database. For example, if a user uploads a receipt that lists "Milk 200 yen," "Eggs 150 yen," and "Bread 300 yen," the device analyzes it using OCR technology and sends this information to the server. The server adds this information to the database.

[0367] Step 3:

[0368] The server obtains real-time temperature data using an external API (such as the OpenWeatherMap API). The input is the temperature data from the external API. The obtained temperature data is analyzed and the results are stored on the server. Next, the server obtains the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The input is the user's emotional data obtained by the emotion engine. This emotional data is analyzed and stored on the server. Once the temperature data and emotional information are collected, the server compares this information with the user's closet database and generates an appropriate outfit. As an output, the server generates an optimal outfit suggestion and sends it to the device. For example, if the server receives data that the temperature is 28 degrees Celsius and sunny, and information that the user is feeling cheerful, the server will suggest an outfit consisting of a white shirt and blue jeans.

[0369] Step 4:

[0370] The terminal displays the outfit suggestions received from the server to the user. The displayed suggestions are presented visually on the screen. The input is the outfit suggestions sent from the server. The user can review the displayed outfit and either accept it as is or fine-tune it to suit their own preferences. The output is the user's final outfit selection. For example, the user can review the suggested white shirt and blue jeans and choose to accept them as is or select different pants.

[0371] Step 5:

[0372] The server generates the next day's menu based on refrigerator inventory data and the user's past consumption data. The inputs are refrigerator inventory data and past consumption data. In addition, the user's emotional information is also obtained (input) using an emotion engine and taken into consideration. A learning algorithm (such as Scikit-learn or TensorFlow) is used to make the most of ingredients and identify any missing ingredients. The output is a menu suggestion for the user and a list of missing foods. For example, if the server recognizes that there is chicken, onions, and potatoes in the refrigerator, it will suggest chicken curry as the next day's menu based on the user's emotional data and add any missing spices to the list.

[0373] Step 6:

[0374] The terminal displays the menu suggestions and the list of missing foods received from the server to the user. The input is the menu suggestions and ingredient list sent from the server. The user checks this and makes a shopping plan. For example, the terminal displays chicken curry and a list of missing spices, and the user uses the list to plan the necessary shopping. The output is an efficient shopping list.

[0375] (Application example 2)

[0376] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0377] Conventional systems were unable to analyze the user's emotions and make suggestions based on them, making it difficult to propose coordination and menus that matched the user's mood and emotions. Furthermore, because suggestions were not based on the user's emotions, it was not possible to expect an improvement in satisfaction.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0379] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store it in a predetermined database, means for analyzing the user's emotions using an emotion recognition model and acquiring emotional information, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and the characteristic information and emotional information about the user's clothes, and means for proposing a predetermined menu based on predetermined food inventory data and the user's emotional information and generating a list of foods that are in short supply based on the menu, thereby enabling optimal coordination and menu proposals that take into account the user's emotions and real-time temperature data.

[0380] A "user" is someone who utilizes the system to effectively manage their life.

[0381] "Clothing image" is photographic data of clothing owned by the user.

[0382] "Characteristic information" is information such as the type, color, and design of clothing extracted through image analysis.

[0383] "Information about food" is data including the name, quantity, price, etc. of the food purchased by the user.

[0384] The "database" is a storage device for storing analyzed characteristic information and information about food.

[0385] An "emotion recognition model" is an algorithm for analyzing a user's emotions and acquiring emotional information.

[0386] "Emotion information" is data that represents the user's current emotion.

[0387] "Temperature data" is information about current and forecast temperatures.

[0388] "Coordination suggestions" are optimal clothing combinations suggested based on the user's clothing characteristics information and temperature data.

[0389] "Food inventory data" is data that indicates the inventory status of food items owned by the user.

[0390] "Menu Suggestion" is a menu of dishes suggested based on food inventory data and emotional information.

[0391] The "missing food list" is a list of foods that the user does not currently own that are required to create the proposed menu.

[0392] "Analysis" refers to processing the received images and information to extract the necessary data.

[0393] This invention is a system for efficiently managing a user's daily life and saving time and effort. This system improves the user experience by recognizing the user's emotions and suggesting outfits and menus that suit those emotions. Specific embodiments for implementing the invention are described below.

[0394] Clothing Management

[0395] Users use their smartphones to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze the clothing's characteristics, such as type, color, and design. This analyzed information is sent to a server and stored in a database for each user. This database is then used to suggest outfits.

[0396] Food Management

[0397] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[0398] emotion recognition

[0399] The device captures the user's facial image in real time using a camera built into a smartphone or smart glasses. It then analyzes the user's emotions using an emotion recognition model (e.g., a deep learning model trained using Keras). The analyzed emotion information is sent to the server and used to suggest outfits and menus.

[0400] Coordination suggestions

[0401] The server obtains real-time temperature data using an external API. It integrates the temperature data with the user's clothing characteristics and emotional information to suggest appropriate outfits to the user. The device displays the outfit suggestions received from the server to the user. For example, it may suggest, "If today's weather is sunny and you are feeling happy, we recommend a white shirt and blue jeans."

[0402] Menu suggestions and shopping list generation

[0403] The server proposes a predetermined menu based on refrigerator inventory data and the user's emotional information. The proposed menu makes the most of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the terminal and displayed to the user. The user can check the proposed menu and the list of missing foods to plan their shopping efficiently.

[0404] Specific examples

[0405] Example 1: Morning outfit suggestions

[0406] 1. The user wakes up in the morning and opens the app on their smartphone.

[0407] 2. The device requests the current temperature data and the user's emotion information from the server.

[0408] 3. The server obtains real-time temperature data from an external API and analyzes the user's emotions using an emotion recognition model.

[0409] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[0410] 5. The device displays the outfit suggestions received from the server to the user. For example, "If the weather is sunny today and you're feeling happy, I recommend a white shirt and blue jeans."

[0411] Example prompt sentence:

[0412] "To suggest a new outfit, please tell us what you would wear if you were feeling happy and the weather today was sunny."

[0413] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0414] Step 1:

[0415] The user opens the smartphone app, takes a picture of the clothing they own, and uploads it to the app.

[0416] Input: Photographed image of clothing

[0417] Output: Image upload

[0418] Specific operation: The user takes a photo of their clothing using their smartphone and taps the upload button in the app to send the image to the server.

[0419] Step 2:

[0420] The terminal receives the uploaded image and analyzes the clothing's characteristic information using an image recognition algorithm.

[0421] Input: Uploaded clothing image

[0422] Output: Analyzed feature information (type, color, design, etc.)

[0423] Specific operation: The device preprocesses the received image and uses an image recognition algorithm (e.g., a CNN model) to identify the type, color, and design of the clothing.

[0424] Step 3:

[0425] The terminal transmits the analyzed feature information to the server and stores it in a database.

[0426] Input: Analyzed clothing feature information

[0427] Output: Feature information sent to the server and stored in the database

[0428] Specific operation: The device converts the identified feature information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[0429] Step 4:

[0430] The user takes a photo of the receipt for the food they purchased and uploads it to the app.

[0431] Input: Image of the receipt

[0432] Output: Upload receipt image

[0433] Specific operation: The user takes a photo of the receipt for the purchased food using their smartphone and taps the upload button in the app to send the image to the server.

[0434] Step 5:

[0435] The device analyzes the uploaded receipt image using OCR technology to extract information about the food purchased.

[0436] Input: Uploaded receipt image

[0437] Output: Extracted food information (name, quantity, price, etc.)

[0438] Specific operation: The terminal passes the received receipt image through an OCR (optical character recognition) engine, extracts character data, and analyzes food information.

[0439] Step 6:

[0440] The terminal transmits the extracted food information to the server and stores it in a database.

[0441] Input: Extracted food information

[0442] Output: Food information sent to the server and stored in the database

[0443] Specific operation: The terminal converts the extracted food information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[0444] Step 7:

[0445] The terminal uses a camera built into a smartphone or smart glasses to capture a user's facial image in real time.

[0446] Input: Real-time captured face image

[0447] Output: Get face image

[0448] Specific operation: While the user is using the app, the device activates the camera and periodically captures images of the user's face.

[0449] Step 8:

[0450] The terminal analyzes the user's emotions using the emotion recognition model and acquires emotion information.

[0451] Input: Real-time captured face image

[0452] Output: Analyzed emotion information (e.g., happiness, sadness)

[0453] Specific operation: The device inputs the acquired facial image into an emotion recognition model (e.g., a deep learning model trained with Keras) to identify the emotion.

[0454] Step 9:

[0455] The terminal transmits the acquired emotion information to the server and uses it to propose coordination and menu items.

[0456] Input: Parsed emotion information

[0457] Output: Emotion information sent to the server

[0458] Specific operation: The terminal sends the identified emotion information to the server and adds it to the user profile.

[0459] Step 10:

[0460] The server uses an external API to obtain real-time temperature data.

[0461] Input: Temperature data obtained from an external API

[0462] Output: Real-time temperature data

[0463] Specific operation: The server sends a request to an external API that provides temperature data and obtains the current temperature.

[0464] Step 11:

[0465] The server integrates the temperature data with the user's clothing characteristics information and emotion information to suggest appropriate outfits to the user.

[0466] Input: Temperature data, clothing feature information, emotion information

[0467] Output: The optimal outfit suggested to the user

[0468] Specific operation: The server compares the temperature data with the emotional information, and refers to the clothing feature database to select the optimal outfit.

[0469] Step 12:

[0470] The terminal displays the coordination proposal received from the server to the user.

[0471] Input: Coordinate proposal from the server

[0472] Output: Coordination suggestions displayed on the user's smartphone or smart glasses

[0473] Specific operation: The device displays the coordination suggestions received from the server within the app, allowing the user to easily check them.

[0474] Step 13:

[0475] The server proposes a predetermined menu based on the refrigerator's inventory data and the user's emotional information.

[0476] Input: refrigerator inventory data, emotional information

[0477] Output: Suggested menu

[0478] Specific operation: The server compares the refrigerator inventory data with the user's emotional information to generate the optimal menu.

[0479] Step 14:

[0480] The server generates a list of missing foods based on the proposed menu and sends it to the terminal.

[0481] Input: Suggested menu

[0482] Output: List of missing foods

[0483] Specific operation: The server creates a list of ingredients required based on the proposed menu, compares it with existing stock, and identifies any ingredients that are in short supply. It then generates a list of ingredients that are in short supply and sends it to the user's device.

[0484] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0486] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0487] [Second embodiment]

[0488] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0489] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0491] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0495] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0496] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0497] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0498] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0499] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0500] MODE FOR CARRYING OUT THE INVENTION

[0501] This invention is a system for efficiently managing a user's daily life, saving time and effort. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[0502] Closet clothing management

[0503] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0504] Management of purchased ingredients

[0505] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the received receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0506] Coordination suggestions

[0507] The server receives temperature data from an external device in real time. It compares the temperature data with the user's clothing characteristics information and suggests appropriate outfits to the user. The device then displays the optimal outfits to the user. The user can then review the suggested outfits and make fine adjustments if necessary.

[0508] Menu suggestions and shopping list generation

[0509] The server uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to plan their shopping efficiently.

[0510] Specific examples

[0511] Example 1: Morning Outfit

[0512] 1. The user wakes up in the morning and opens the app on their smartphone.

[0513] 2. The device requests temperature data from the server.

[0514] 3. The server obtains real-time temperature data from an external API and compares it with the user's closet database to select the optimal outfit.

[0515] 4. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[0516] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0517] Example 2: Dinner menu suggestions

[0518] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0519] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0520] 3. The server adds the food information to the database, updates the refrigerator inventory, and uses a learning algorithm to generate data to suggest menus for the next day.

[0521] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[0522] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0523] This system allows users to efficiently manage the items in their closets and refrigerators, freeing them from small everyday worries and allowing them to live a richer life.

[0524] The processing flow will be explained below.

[0525] Closet clothing management

[0526] Step 1:

[0527] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[0528] Step 2:

[0529] The terminal receives the uploaded image.

[0530] Step 3:

[0531] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[0532] Step 4:

[0533] The terminal transmits the analysis results to the server.

[0534] Step 5:

[0535] The server stores the received clothing data in a database for each user.

[0536] Management of purchased ingredients

[0537] Step 1:

[0538] Users take a photo of the receipt for the food they purchased and upload it to the app.

[0539] Step 2:

[0540] The terminal receives the uploaded receipt image.

[0541] Step 3:

[0542] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[0543] Step 4:

[0544] The terminal transmits the analyzed food information to the server.

[0545] Step 5:

[0546] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[0547] Coordination suggestions

[0548] Step 1:

[0549] The server obtains temperature data for the user's location in real time (using an external API).

[0550] Step 2:

[0551] The server compares the acquired temperature data with the clothing information recorded in the user's closet database.

[0552] Step 3:

[0553] The server selects the best outfit for the temperature.

[0554] Step 4:

[0555] The terminal displays the coordination suggestions received from the server on the user's app.

[0556] Step 5:

[0557] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[0558] Menu suggestions and shopping list generation

[0559] Step 1:

[0560] The server refers to the user's past consumption data based on the refrigerator inventory data.

[0561] Step 2:

[0562] The server uses a learning algorithm to generate the next day's menu.

[0563] Step 3:

[0564] The server creates a list of missing foods based on the generated menu.

[0565] Step 4:

[0566] The device displays the menu suggestions and shopping list received from the server in the user's app.

[0567] Step 5:

[0568] The user reviews the suggested menu and plans their shopping based on the list of missing foods.

[0569] Through the above steps, the system can efficiently assist users in their daily lives and save them a lot of time and effort.

[0570] Example 1

[0571] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0572] In modern life, managing clothing and food inventory, as well as suggesting clothing coordination according to the weather and planning daily meal menus, are complicated tasks that require time and effort. Furthermore, managing these pieces of information individually is inefficient, so a system that can manage them all comprehensively is needed.

[0573] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0574] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and suggesting appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for suggesting predetermined menus based on predetermined food life data and generating a list of missing foods based on the menus, means for receiving receipt images photographed by the user and extracting purchased food information using OCR technology, and means for suggesting menus using a learning algorithm based on the user's past consumption data and current inventory data and generating a list of missing ingredients, thereby enabling users to comprehensively and efficiently manage various aspects of their daily lives.

[0575] "User" refers to any individual or organization that uses this system.

[0576] "Clothing image" refers to image data of clothing owned by the user.

[0577] "Feature information" refers to attribute information such as clothing type, color, and design extracted through image analysis.

[0578] "Food information" refers to detailed information about the food purchased by the user, such as the food name, quantity, purchase date and time, etc.

[0579] A "database" refers to an information management system for systematically storing analyzed information.

[0580] "Temperature data" refers to information about the current temperature obtained from an external weather data provider.

[0581] "Coordination suggestions" refers to a function that suggests optimal clothing combinations based on the user's clothing characteristics information and temperature data.

[0582] "Food life data" refers to life cycle data about food, such as expiration dates and inventory information.

[0583] "Menu suggestion" refers to a function that suggests optimal meal menus based on the user's inventory information and consumption data.

[0584] "List of missing foods" refers to a list of additional foods that need to be purchased in order to implement the proposed menu.

[0585] "Receipt image" refers to image data of the receipt for the purchased product photographed by the user.

[0586] "OCR technology" is an abbreviation for optical character recognition technology, and refers to the technology of extracting character data from images.

[0587] "Learning algorithm" refers to a machine learning technique that learns from a user's past consumption data and current inventory data to make predictions and suggestions.

[0588] This system efficiently manages users' daily lives, saving them time and effort. This system has four main functions: managing clothes in the closet, managing purchased ingredients, suggesting outfits based on temperature data, and creating menus and shopping lists.

[0589] Closet clothing management

[0590] Users take photos of their clothing using their smartphones or computers and upload them to a dedicated app. The app uses well-known image processing technologies such as TensorFlow and OpenCV. The device receives the uploaded image and uses an image recognition algorithm to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is then sent to a server and stored in a database for each user.

[0591] Management of purchased ingredients

[0592] Users take a photo of the receipt for the food they purchased and upload it to a dedicated app. The device then analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[0593] Coordination suggestions

[0594] The server obtains temperature data in real time from an external weather data provider (e.g., OpenWeatherMap API). It compares this temperature data with clothing data stored in the user's closet and suggests optimal outfits for the user. The device displays the outfit suggestions received from the server to the user. The user can review the suggested outfits and make fine adjustments as necessary.

[0595] Menu suggestions and shopping list generation

[0596] After returning home from the supermarket, the user takes a photo of their purchase receipt and uploads it to a dedicated app. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server. The server adds the food information to a database and updates the refrigerator inventory information. Using a learning algorithm (e.g., Scikit-learn), the system suggests a menu for the next day based on the user's past consumption data and current inventory data. The suggested menu makes the most of existing ingredients and generates a list of any missing ingredients. The generated list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to efficiently plan their shopping.

[0597] Specific examples

[0598] Example 1: Morning Outfit

[0599] 1. The user wakes up in the morning and opens the app on their smartphone.

[0600] 2. The device requests temperature data from the server.

[0601] 3. The server obtains real-time temperature data from an external weather data provider and compares it with the user's closet database to select the optimal outfit.

[0602] 4. The terminal displays the outfit suggestions received from the server to the user (e.g., "white shirt and blue jeans").

[0603] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0604] Example 2: Dinner menu suggestions

[0605] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0606] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0607] 3. The server adds the food information to the database, updates the refrigerator inventory, and generates the next day's menu (e.g., "chicken curry") using a learning algorithm.

[0608] 4. The terminal displays the menu suggestions received from the server and the list of missing foods to the user.

[0609] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0610] Prompt Sentence Examples

[0611] 1. "What's the temperature today?" → Server: "It's 20 degrees."

[0612] 2. "What is appropriate clothing for this temperature?" → Server: "I recommend a white shirt and blue jeans."

[0613] 3. "Based on what ingredients you have in the fridge, what would be a good menu item for tomorrow?" → Server: "I think it would be good to have the ingredients needed to make chicken curry."

[0614] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0615] Closet clothing management

[0616] Step 1:

[0617] Users take photos of their own clothes using a smartphone or computer and upload them to a dedicated app. The input is the photo of the clothes taken by the user, and the output is the image file uploaded to the app.

[0618] Step 2:

[0619] The terminal receives the uploaded image. The input is the image file from the user, and the output is the received image data. At this point, a communication protocol (e.g., HTTP) is used for receiving.

[0620] Step 3:

[0621] The device analyzes images using image recognition algorithms such as TensorFlow and OpenCV to extract feature information such as the type, color, and design of clothing. For example, it can obtain information such as "white shirt" or "blue jeans" from an image. The input is the received image data, and the output is the extracted feature information.

[0622] Step 4:

[0623] The terminal sends the analysis results to the server. The input is the feature information, and the output is the analyzed data sent via the network.

[0624] Step 5:

[0625] The server stores the transmitted feature information in a database for each user. For example, "User A's closet" is registered as "white shirt" and "blue jeans." The input is the transmitted analyzed data, and the output is the updated database.

[0626] Management of purchased ingredients

[0627] Step 1:

[0628] The user takes a photo of the receipt for the food they purchased and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[0629] Step 2:

[0630] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the received receipt image, and the output is the food information extracted through analysis. For example, information such as "milk" and "eggs" is extracted from the receipt.

[0631] Step 3:

[0632] The terminal sends the extracted information to the server. The input is the extracted food information, and the output is the food data transmitted via the network.

[0633] Step 4:

[0634] The server stores the food information in a database. The input is the food information sent, and the output is the updated database. For example, add "milk" and "eggs" to "User A's food inventory."

[0635] Coordination suggestions

[0636] Step 1:

[0637] The server sends a request to an external weather data provider (e.g., OpenWeatherMap API) to obtain real-time temperature data. The input is the weather data request, and the output is the obtained temperature data.

[0638] Step 2:

[0639] The server compares the acquired temperature data with the user's closet database. For example, it selects an outfit suitable for a temperature of 20 degrees. The input is temperature data and closet data, and the output is a suggestion of the optimal outfit.

[0640] Step 3:

[0641] The server sends the proposed coordinates to the terminal. The input is the proposed coordinates, and the output is the coordination proposal sent over the network.

[0642] Step 4:

[0643] The terminal displays the coordinate suggestion received from the server to the user. The input is the suggested coordinate information, and the output is the coordinate displayed on the terminal.

[0644] Step 5:

[0645] The user can check the suggested outfits and either accept them as they are or adjust them themselves to choose the outfit. The input is the displayed outfit, and the output is the final outfit decided by the user.

[0646] Menu suggestions and shopping list generation

[0647] Step 1:

[0648] After returning home from the supermarket, the user takes a photo of the receipt and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[0649] Step 2:

[0650] The device analyzes the received receipt image using OCR technology and extracts information about the purchased food. The input is the received receipt image, and the output is the food information extracted by the analysis.

[0651] Step 3:

[0652] The terminal sends the purchased food information to the server. The input is the extracted food information, and the output is the food data sent via the network.

[0653] Step 4:

[0654] The server adds the food information to the database and updates the refrigerator inventory information. The input is the submitted food information, and the output is the updated refrigerator inventory data.

[0655] Step 5:

[0656] The server uses a learning algorithm (e.g., Scikit-learn) to suggest the next day's menu based on the user's past consumption data and current inventory data. The input is consumption data and inventory data, and the output is the suggested menu.

[0657] Step 6:

[0658] The server generates a list of missing ingredients and sends it to the terminal. The input is the proposed menu and the output is the missing ingredients list.

[0659] Step 7:

[0660] The terminal displays the received menu suggestions and the list of missing foods to the user. The input is the suggested menu and the list of missing foods, and the output is the menu and list displayed on the terminal.

[0661] Step 8:

[0662] The user can then review the suggested menu and list of ingredients needed to efficiently plan their shopping. The input is the displayed menu and list, and the output is the user's final shopping plan.

[0663] (Application example 1)

[0664] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] Today's busy consumers are seeking systems that can efficiently manage the items in their closets and refrigerators and suggest optimal outfits and meal plans in their daily lives. They also have numerous needs for virtual store shopping experiences, such as suggestions for coordinating items with items they plan to purchase and smooth food inventory management. Conventional systems have difficulty meeting these needs in an integrated manner. Therefore, a new system that can comprehensively and efficiently support users' lives is needed.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0667] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, means for suggesting coordination with products that the user is considering purchasing in a virtual store, and means for managing information about purchased products and keeping track of the user's inventory. This allows the user to efficiently manage their daily closet and refrigerator, and enables optimal coordination suggestions and inventory management through a shopping experience in a virtual store.

[0668] "User" means an individual or corporation that uses this system.

[0669] "Clothing feature information" is data such as clothing type, color, and design extracted using image recognition technology.

[0670] "Food life data" refers to information on the type, quantity, and date of consumption of food purchased and consumed.

[0671] A "menu" is a meal plan suggested to a user, including the types and amounts of ingredients used.

[0672] A "candidate product for purchase" is a product that the user is considering purchasing in the virtual store.

[0673] "Temperature data" is current temperature information obtained from an external weather information service.

[0674] "Coordination" is a suggestion on how to combine and apply clothing.

[0675] A "receipt image" is an image of a receipt that lists details of the products purchased by the user.

[0676] "Inventory" refers to the quantity and type of items, such as food and clothing, that a user has already purchased and possesses.

[0677] A "learning algorithm" is a computational method for analyzing past data and predicting future trends and patterns.

[0678] A "virtual store" is a virtual shopping mall where users can browse and purchase products online.

[0679] The "server" is a computer device that serves as the center of the system and stores, analyzes, and makes recommendations on data.

[0680] This invention is a system designed to enable users to efficiently manage their lifestyles. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[0681] First, a user uses a smartphone application to take a photo of their clothing and upload it to the application. The device analyzes the uploaded image and uses an image recognition algorithm (e.g., Google Cloud Vision API) to extract feature information such as the type, color, and design of the clothing. The extracted feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0682] Next, the user takes a photo of the receipt for the purchased food and uploads it to the application. The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows the server to keep track of the inventory status of the ingredients purchased by the user.

[0683] In addition, the server obtains real-time temperature data from external weather information services (e.g., OpenWeather API), compares this data with the user's clothing characteristics, and suggests appropriate outfits for the user. The suggested outfits are then displayed to the user on their device.

[0684] The server also uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is then sent to the device and displayed to the user.

[0685] In particular, in the virtual store, users can receive suggestions for coordinating products with potential purchases. Purchased product information is managed by the server and added to the user's inventory database. This function enhances the virtual shopping experience.

[0686] Specific examples include suggestions for morning outfits and dinner menus. For example, when a user wakes up in the morning and opens a smartphone application, the device requests temperature data from the server, and the server compares the real-time temperature data with the user's closet information to select the optimal outfit and display it on the device.

[0687] When a user returns home from the supermarket and uploads their purchase receipt to the application, the device analyzes the received receipt image using OCR technology and sends the purchased food information to the server, which then analyzes the data and presents the user with the next day's menu and a list of any missing foods.

[0688] An example prompt is, "The user has provided an image of a dress they would like to purchase. Please suggest outfits that would go well with this dress. The temperature is 20 degrees and the user prefers a casual style. The user has the following items in their closet: a red jacket, black leather pants, and a white T-shirt. Which would go well together?"

[0689] In this way, this system efficiently manages the user's daily life and supports a more comfortable life.

[0690] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0691] Step 1:

[0692] Users use a smartphone application to take a photo of their clothing and upload it to the application. This input data is an image file, which is received by the device.

[0693] Step 2:

[0694] The device analyzes the uploaded image file using an image recognition algorithm (e.g., Google Cloud Vision API) and extracts characteristic information such as the type of clothing, color, and design. The extracted characteristic information is output as data and sent to the server.

[0695] Step 3:

[0696] The server stores the received feature information in a database for each user, which is later used to suggest outfits.

[0697] Step 4:

[0698] The user takes a photo of the receipt for the food they purchased and uploads it to the application. This input data is also an image file, and the device receives it.

[0699] Step 5:

[0700] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted food information is output as data and sent to the server.

[0701] Step 6:

[0702] The server stores the received food information in a designated database, allowing the user to keep track of the inventory of ingredients purchased.

[0703] Step 7:

[0704] The server obtains real-time temperature data from an external weather information service (e.g., OpenWeather API). This temperature data is input data and is received by the server.

[0705] Step 8:

[0706] The server compares the temperature data with the user's clothing characteristics information and uses a generative AI model to suggest appropriate outfits. This output data is the outfit suggestion and is sent to the device.

[0707] Step 9:

[0708] The terminal displays the coordinated outfit suggestions received from the server to the user, who can then check the suggested outfits and either accept them or make minor adjustments.

[0709] Step 10:

[0710] The server uses the refrigerator's inventory data, past consumption data, and a learning algorithm to propose a menu for the next day. This output data is the menu proposal, and is sent to the terminal along with a list of missing ingredients.

[0711] Step 11:

[0712] The terminal displays the menu suggestions and the list of missing foods received from the server to the user, who then checks the suggested menu and list and makes a plan to purchase the necessary ingredients.

[0713] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0714] MODE FOR CARRYING OUT THE INVENTION

[0715] The present invention is a system for efficiently managing a user's daily life, saving time and effort. In particular, it has the function of recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions.

[0716] Closet clothing management

[0717] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0718] Management of purchased ingredients

[0719] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0720] Coordination suggestions

[0721] The server obtains temperature data from an external source in real time. It compares the temperature data with the user's clothing characteristics information and proposes appropriate outfits to the user. In addition, it uses an emotion engine to obtain the user's emotional information and integrates the temperature data with the emotional information to optimize the outfit proposals. This allows the device to display the optimal outfits to the user. The user can review the proposed outfits and make fine adjustments if necessary.

[0722] Menu suggestions and shopping list generation

[0723] The server references the user's past consumption data based on refrigerator inventory data. Using a learning algorithm, the server takes into account the user's emotional information obtained by the emotion engine when generating the next day's menu. The proposed menu maximizes the use of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the device and displayed to the user. The user can check the proposed menu and the list of missing foods to efficiently plan their shopping.

[0724] Specific examples

[0725] Example 1: Morning Outfit

[0726] 1. The user wakes up in the morning and opens the app on their smartphone.

[0727] 2. The device requests the current temperature data and the user's emotion information from the server.

[0728] 3. The server obtains real-time temperature data from an external API and simultaneously analyzes the user's emotional information using an emotion engine.

[0729] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[0730] 5. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[0731] 6. The user checks the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0732] Example 2: Dinner menu suggestions

[0733] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0734] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0735] 3. The server adds the food information to the database and updates the refrigerator inventory information. It uses an emotion engine to obtain the user's emotional information and uses a learning algorithm to generate the next day's menu.

[0736] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[0737] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0738] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions tailored to their emotions, and live a richer life.

[0739] The processing flow will be explained below.

[0740] Closet clothing management

[0741] Step 1:

[0742] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[0743] Step 2:

[0744] The terminal receives the uploaded image.

[0745] Step 3:

[0746] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[0747] Step 4:

[0748] The terminal transmits the analysis results to the server.

[0749] Step 5:

[0750] The server stores the received clothing data in a database for each user.

[0751] Management of purchased ingredients

[0752] Step 1:

[0753] Users take a photo of the receipt for the food they purchased and upload it to the app.

[0754] Step 2:

[0755] The terminal receives the uploaded receipt image.

[0756] Step 3:

[0757] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[0758] Step 4:

[0759] The terminal transmits the analyzed food information to the server.

[0760] Step 5:

[0761] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[0762] Coordination suggestions

[0763] Step 1:

[0764] The server obtains temperature data for the user's location in real time (using an external API).

[0765] Step 2:

[0766] The server uses an emotion engine to acquire and analyze the user's emotion information.

[0767] Step 3:

[0768] The server compares the acquired temperature data and emotion information with the clothing information recorded in the user's closet database.

[0769] Step 4:

[0770] The server selects the best outfit that matches the temperature and your emotions.

[0771] Step 5:

[0772] The terminal displays the coordination suggestions received from the server on the user's app.

[0773] Step 6:

[0774] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[0775] Menu suggestions and shopping list generation

[0776] Step 1:

[0777] The server refers to the user's past consumption data based on the refrigerator inventory data.

[0778] Step 2:

[0779] The server uses an emotion engine to acquire and analyze the user's emotion information.

[0780] Step 3:

[0781] The server uses a learning algorithm to generate the next day's menu, taking emotional information into account.

[0782] Step 4:

[0783] The server creates a list of missing foods based on the generated menu.

[0784] Step 5:

[0785] The device displays the menu suggestions and shopping list received from the server in the user's app.

[0786] Step 6:

[0787] Users can plan their shopping by reviewing suggested menus and a list of missing foods.

[0788] This system allows users to efficiently manage the items in their closets and refrigerators and receive optimal suggestions tailored to their mood.

[0789] Example 2

[0790] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0791] In modern life, many users need to efficiently manage their daily lives to save time and effort. However, current systems have difficulty consistently managing clothing and food inventory, and suggesting appropriate outfits and menus. Furthermore, suggestions rarely reflect the user's mood or emotions, which prevents users from increasing their satisfaction. Therefore, there is a need for a system that can comprehensively support users' daily lives and make optimal suggestions based on their emotions.

[0792] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0793] In this invention, the server includes means for acquiring emotional information of a user and optimizing clothing coordination suggestions based on the emotional information, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, and means for optimizing the menus using a learning algorithm based on the emotional information and the user's past consumption data and current inventory data, thereby enabling the user to efficiently manage the items in their closet and refrigerator and receive optimal suggestions based on their emotions.

[0794] "User" means an individual or organization that uses this system.

[0795] "Clothing images" refers to photographs or image data of clothing owned by the user.

[0796] "Means for analyzing images and extracting characteristic information about clothing" refers to algorithms and technologies that analyze images of clothing and extract information such as color, type, and design.

[0797] "Food information" refers to data such as the type, quantity, and price of food purchased by the user.

[0798] The "predetermined database" refers to a database system for storing information about a user's clothing and purchased food.

[0799] "Temperature Data" refers to data regarding local temperatures obtained from external weather information services.

[0800] "User emotion information" refers to data that shows the results of analyzing the user's emotions and moods.

[0801] "Specified food life data" refers to data regarding food storage conditions, expiration dates, etc.

[0802] "Means for suggesting menus and generating a list of missing foods" refers to algorithms and technologies that suggest appropriate cooking menus based on the ingredients in the refrigerator and the user's past consumption patterns, and that create a list of missing foods for that purpose.

[0803] A "learning algorithm" is an algorithm that learns patterns from past data and makes inferences and predictions about new data.

[0804] MODE FOR CARRYING OUT THE INVENTION

[0805] The present invention is a system for efficiently managing a user's daily life and saving time and effort. In particular, it has a function for recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions. Specific embodiments of this system will be described in detail below.

[0806] Closet clothing management

[0807] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms (such as TensorFlow or OpenCV) to analyze feature information such as the type, color, and design of the clothing. The analyzed feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0808] For example, if a user uploads a photo of a blue shirt, the device analyzes the image and sends the characteristic information that the shirt is blue to the server, which stores this information in the user's database.

[0809] Management of purchased ingredients

[0810] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0811] For example, when a user uploads a receipt, the device uses OCR technology to send the information about "milk," "eggs," and "bread" to the server, which adds this information to the database and updates the user's refrigerator inventory.

[0812] Coordination suggestions

[0813] The server acquires real-time temperature data using an external API (such as the OpenWeatherMap API). It then acquires the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The server compares the acquired temperature data and emotional information with the user's closet database and generates an appropriate outfit. The generated outfit suggestions are sent to the device and displayed to the user.

[0814] For example, if the server receives data indicating that the temperature is 28 degrees and the user is in good spirits, the server will suggest a white shirt and blue jeans. This suggestion will be sent to the device and displayed to the user.

[0815] Menu suggestions and shopping list generation

[0816] The server generates a menu based on refrigerator inventory data and the user's past consumption data. It also takes into account the user's emotional information using an emotion engine. A learning algorithm (such as Scikit-learn or TensorFlow) is used in the generation process. The proposed menu makes the most of existing ingredients, and any ingredients that are missing are generated as a list. This list is sent to the device and displayed to the user.

[0817] For example, if the server recognizes that the user has chicken, onions, and potatoes in the refrigerator and considers that the user feels like refreshing, it will suggest chicken curry as a menu item for the next day. It will also add missing spices to the shopping list. This list is sent to the terminal and displayed to the user.

[0818] Prompt Sentence Examples

[0819] "Write a Python program that uses temperature data and user emotion information to generate suggestions for morning outfits."

[0820] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions based on their emotions, and live a richer life.

[0821] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0822] Step 1:

[0823] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The input is the image of the clothing taken. The device receives the uploaded image and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze the image for feature information such as the type, color, and design of the clothing. This analysis extracts feature information such as color, type, and design. The extracted feature information is sent to the server and stored in a database for each user. As an output, the feature information of the clothing is saved in a database for each user. For example, if a user uploads an image of a blue shirt, the device analyzes the image and extracts the feature information "blue shirt, long sleeves, size M" and sends it to the server.

[0824] Step 2:

[0825] The user takes a photo of the receipt for the food they purchased and uploads it to the app. The input is an image of the receipt for the purchased food. The device analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. Specifically, it extracts the product name, quantity, price, and other information listed on the receipt as text data. The extracted information is sent to the server and stored in a database. As an output, the user's food purchase information is added to the database. For example, if a user uploads a receipt that lists "Milk 200 yen," "Eggs 150 yen," and "Bread 300 yen," the device analyzes it using OCR technology and sends this information to the server. The server adds this information to the database.

[0826] Step 3:

[0827] The server obtains real-time temperature data using an external API (such as the OpenWeatherMap API). The input is the temperature data from the external API. The obtained temperature data is analyzed and the results are stored on the server. Next, the server obtains the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The input is the user's emotional data obtained by the emotion engine. This emotional data is analyzed and stored on the server. Once the temperature data and emotional information are collected, the server compares this information with the user's closet database and generates an appropriate outfit. As an output, the server generates an optimal outfit suggestion and sends it to the device. For example, if the server receives data that the temperature is 28 degrees Celsius and sunny, and information that the user is feeling cheerful, the server will suggest an outfit consisting of a white shirt and blue jeans.

[0828] Step 4:

[0829] The terminal displays the outfit suggestions received from the server to the user. The displayed suggestions are presented visually on the screen. The input is the outfit suggestions sent from the server. The user can review the displayed outfit and either accept it as is or fine-tune it to suit their own preferences. The output is the user's final outfit selection. For example, the user can review the suggested white shirt and blue jeans and choose to accept them as is or select different pants.

[0830] Step 5:

[0831] The server generates the next day's menu based on refrigerator inventory data and the user's past consumption data. The inputs are refrigerator inventory data and past consumption data. In addition, the user's emotional information is also obtained (input) using an emotion engine and taken into consideration. A learning algorithm (such as Scikit-learn or TensorFlow) is used to make the most of ingredients and identify any missing ingredients. The output is a menu suggestion for the user and a list of missing foods. For example, if the server recognizes that there is chicken, onions, and potatoes in the refrigerator, it will suggest chicken curry as the next day's menu based on the user's emotional data and add any missing spices to the list.

[0832] Step 6:

[0833] The terminal displays the menu suggestions and the list of missing foods received from the server to the user. The input is the menu suggestions and ingredient list sent from the server. The user checks this and makes a shopping plan. For example, the terminal displays chicken curry and a list of missing spices, and the user uses the list to plan the necessary shopping. The output is an efficient shopping list.

[0834] (Application example 2)

[0835] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0836] Conventional systems were unable to analyze the user's emotions and make suggestions based on them, making it difficult to propose coordination and menus that matched the user's mood and emotions. Furthermore, because suggestions were not based on the user's emotions, it was not possible to expect an improvement in satisfaction.

[0837] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0838] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store it in a predetermined database, means for analyzing the user's emotions using an emotion recognition model and acquiring emotional information, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and the characteristic information and emotional information about the user's clothes, and means for proposing a predetermined menu based on predetermined food inventory data and the user's emotional information and generating a list of foods that are in short supply based on the menu, thereby enabling optimal coordination and menu proposals that take into account the user's emotions and real-time temperature data.

[0839] A "user" is someone who utilizes the system to effectively manage their life.

[0840] "Clothing image" is photographic data of clothing owned by the user.

[0841] "Characteristic information" is information such as the type, color, and design of clothing extracted through image analysis.

[0842] "Information about food" is data including the name, quantity, price, etc. of the food purchased by the user.

[0843] The "database" is a storage device for storing analyzed characteristic information and information about food.

[0844] An "emotion recognition model" is an algorithm for analyzing a user's emotions and acquiring emotional information.

[0845] "Emotion information" is data that represents the user's current emotion.

[0846] "Temperature data" is information about current and forecast temperatures.

[0847] "Coordination suggestions" are optimal clothing combinations suggested based on the user's clothing characteristics information and temperature data.

[0848] "Food inventory data" is data that indicates the inventory status of food items owned by the user.

[0849] "Menu Suggestion" is a menu of dishes suggested based on food inventory data and emotional information.

[0850] The "missing food list" is a list of foods that the user does not currently own that are required to create the proposed menu.

[0851] "Analysis" refers to processing the received images and information to extract the necessary data.

[0852] This invention is a system for efficiently managing a user's daily life and saving time and effort. This system improves the user experience by recognizing the user's emotions and suggesting outfits and menus that suit those emotions. Specific embodiments for implementing the invention are described below.

[0853] Clothing Management

[0854] Users use their smartphones to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze the clothing's characteristics, such as type, color, and design. This analyzed information is sent to a server and stored in a database for each user. This database is then used to suggest outfits.

[0855] Food Management

[0856] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[0857] emotion recognition

[0858] The device captures the user's facial image in real time using a camera built into a smartphone or smart glasses. It then analyzes the user's emotions using an emotion recognition model (e.g., a deep learning model trained using Keras). The analyzed emotion information is sent to the server and used to suggest outfits and menus.

[0859] Coordination suggestions

[0860] The server obtains real-time temperature data using an external API. It integrates the temperature data with the user's clothing characteristics and emotional information to suggest appropriate outfits to the user. The device displays the outfit suggestions received from the server to the user. For example, it may suggest, "If today's weather is sunny and you are feeling happy, we recommend a white shirt and blue jeans."

[0861] Menu suggestions and shopping list generation

[0862] The server proposes a predetermined menu based on refrigerator inventory data and the user's emotional information. The proposed menu makes the most of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the terminal and displayed to the user. The user can check the proposed menu and the list of missing foods to plan their shopping efficiently.

[0863] Specific examples

[0864] Example 1: Morning outfit suggestions

[0865] 1. The user wakes up in the morning and opens the app on their smartphone.

[0866] 2. The device requests the current temperature data and the user's emotion information from the server.

[0867] 3. The server obtains real-time temperature data from an external API and analyzes the user's emotions using an emotion recognition model.

[0868] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[0869] 5. The device displays the outfit suggestions received from the server to the user. For example, "If the weather is sunny today and you're feeling happy, I recommend a white shirt and blue jeans."

[0870] Example prompt sentence:

[0871] "To suggest a new outfit, please tell us what you would wear if you were feeling happy and the weather today was sunny."

[0872] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0873] Step 1:

[0874] The user opens the smartphone app, takes a picture of the clothing they own, and uploads it to the app.

[0875] Input: Photographed image of clothing

[0876] Output: Image upload

[0877] Specific operation: The user takes a photo of their clothing using their smartphone and taps the upload button in the app to send the image to the server.

[0878] Step 2:

[0879] The terminal receives the uploaded image and analyzes the clothing's characteristic information using an image recognition algorithm.

[0880] Input: Uploaded clothing image

[0881] Output: Analyzed feature information (type, color, design, etc.)

[0882] Specific operation: The device preprocesses the received image and uses an image recognition algorithm (e.g., a CNN model) to identify the type, color, and design of the clothing.

[0883] Step 3:

[0884] The terminal transmits the analyzed feature information to the server and stores it in a database.

[0885] Input: Analyzed clothing feature information

[0886] Output: Feature information sent to the server and stored in the database

[0887] Specific operation: The device converts the identified feature information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[0888] Step 4:

[0889] The user takes a photo of the receipt for the food they purchased and uploads it to the app.

[0890] Input: Image of the receipt

[0891] Output: Upload receipt image

[0892] Specific operation: The user takes a photo of the receipt for the purchased food using their smartphone and taps the upload button in the app to send the image to the server.

[0893] Step 5:

[0894] The device analyzes the uploaded receipt image using OCR technology to extract information about the food purchased.

[0895] Input: Uploaded receipt image

[0896] Output: Extracted food information (name, quantity, price, etc.)

[0897] Specific operation: The terminal passes the received receipt image through an OCR (optical character recognition) engine, extracts character data, and analyzes food information.

[0898] Step 6:

[0899] The terminal transmits the extracted food information to the server and stores it in a database.

[0900] Input: Extracted food information

[0901] Output: Food information sent to the server and stored in the database

[0902] Specific operation: The terminal converts the extracted food information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[0903] Step 7:

[0904] The terminal uses a camera built into a smartphone or smart glasses to capture a user's facial image in real time.

[0905] Input: Real-time captured face image

[0906] Output: Get face image

[0907] Specific operation: While the user is using the app, the device activates the camera and periodically captures images of the user's face.

[0908] Step 8:

[0909] The terminal analyzes the user's emotions using the emotion recognition model and acquires emotion information.

[0910] Input: Real-time captured face image

[0911] Output: Analyzed emotion information (e.g., happiness, sadness)

[0912] Specific operation: The device inputs the acquired facial image into an emotion recognition model (e.g., a deep learning model trained with Keras) to identify the emotion.

[0913] Step 9:

[0914] The terminal transmits the acquired emotion information to the server and uses it to propose coordination and menu items.

[0915] Input: Parsed emotion information

[0916] Output: Emotion information sent to the server

[0917] Specific operation: The terminal sends the identified emotion information to the server and adds it to the user profile.

[0918] Step 10:

[0919] The server uses an external API to obtain real-time temperature data.

[0920] Input: Temperature data obtained from an external API

[0921] Output: Real-time temperature data

[0922] Specific operation: The server sends a request to an external API that provides temperature data and obtains the current temperature.

[0923] Step 11:

[0924] The server integrates the temperature data with the user's clothing characteristics information and emotion information to suggest appropriate outfits to the user.

[0925] Input: Temperature data, clothing feature information, emotion information

[0926] Output: The optimal outfit suggested to the user

[0927] Specific operation: The server compares the temperature data with the emotional information, and refers to the clothing feature database to select the optimal outfit.

[0928] Step 12:

[0929] The terminal displays the coordination proposal received from the server to the user.

[0930] Input: Coordinate proposal from the server

[0931] Output: Coordination suggestions displayed on the user's smartphone or smart glasses

[0932] Specific operation: The device displays the coordination suggestions received from the server within the app, allowing the user to easily check them.

[0933] Step 13:

[0934] The server proposes a predetermined menu based on the refrigerator's inventory data and the user's emotional information.

[0935] Input: refrigerator inventory data, emotional information

[0936] Output: Suggested menu

[0937] Specific operation: The server compares the refrigerator inventory data with the user's emotional information to generate the optimal menu.

[0938] Step 14:

[0939] The server generates a list of missing foods based on the proposed menu and sends it to the terminal.

[0940] Input: Suggested menu

[0941] Output: List of missing foods

[0942] Specific operation: The server creates a list of ingredients required based on the proposed menu, compares it with existing stock, and identifies any ingredients that are in short supply. It then generates a list of ingredients that are in short supply and sends it to the user's device.

[0943] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0944] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0945] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0946] [Third embodiment]

[0947] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0948] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0949] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0950] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0951] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0952] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0953] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0954] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0955] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0956] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0957] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0958] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0959] MODE FOR CARRYING OUT THE INVENTION

[0960] This invention is a system for efficiently managing a user's daily life, saving time and effort. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[0961] Closet clothing management

[0962] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[0963] Management of purchased ingredients

[0964] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the received receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[0965] Coordination suggestions

[0966] The server receives temperature data from an external device in real time. It compares the temperature data with the user's clothing characteristics information and suggests appropriate outfits to the user. The device then displays the optimal outfits to the user. The user can then review the suggested outfits and make fine adjustments if necessary.

[0967] Menu suggestions and shopping list generation

[0968] The server uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to plan their shopping efficiently.

[0969] Specific examples

[0970] Example 1: Morning Outfit

[0971] 1. The user wakes up in the morning and opens the app on their smartphone.

[0972] 2. The device requests temperature data from the server.

[0973] 3. The server obtains real-time temperature data from an external API and compares it with the user's closet database to select the optimal outfit.

[0974] 4. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[0975] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[0976] Example 2: Dinner menu suggestions

[0977] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[0978] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[0979] 3. The server adds the food information to the database, updates the refrigerator inventory, and uses a learning algorithm to generate data to suggest menus for the next day.

[0980] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[0981] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[0982] This system allows users to efficiently manage the items in their closets and refrigerators, freeing them from small everyday worries and allowing them to live a richer life.

[0983] The processing flow will be explained below.

[0984] Closet clothing management

[0985] Step 1:

[0986] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[0987] Step 2:

[0988] The terminal receives the uploaded image.

[0989] Step 3:

[0990] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[0991] Step 4:

[0992] The terminal transmits the analysis results to the server.

[0993] Step 5:

[0994] The server stores the received clothing data in a database for each user.

[0995] Management of purchased ingredients

[0996] Step 1:

[0997] Users take a photo of the receipt for the food they purchased and upload it to the app.

[0998] Step 2:

[0999] The terminal receives the uploaded receipt image.

[1000] Step 3:

[1001] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[1002] Step 4:

[1003] The terminal transmits the analyzed food information to the server.

[1004] Step 5:

[1005] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[1006] Coordination suggestions

[1007] Step 1:

[1008] The server obtains temperature data for the user's location in real time (using an external API).

[1009] Step 2:

[1010] The server compares the acquired temperature data with the clothing information recorded in the user's closet database.

[1011] Step 3:

[1012] The server selects the best outfit for the temperature.

[1013] Step 4:

[1014] The terminal displays the coordination suggestions received from the server on the user's app.

[1015] Step 5:

[1016] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[1017] Menu suggestions and shopping list generation

[1018] Step 1:

[1019] The server refers to the user's past consumption data based on the refrigerator inventory data.

[1020] Step 2:

[1021] The server uses a learning algorithm to generate the next day's menu.

[1022] Step 3:

[1023] The server creates a list of missing foods based on the generated menu.

[1024] Step 4:

[1025] The device displays the menu suggestions and shopping list received from the server in the user's app.

[1026] Step 5:

[1027] The user reviews the suggested menu and plans their shopping based on the list of missing foods.

[1028] Through the above steps, the system can efficiently assist users in their daily lives and save them a lot of time and effort.

[1029] Example 1

[1030] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1031] In modern life, managing clothing and food inventory, as well as suggesting clothing coordination according to the weather and planning daily meal menus, are complicated tasks that require time and effort. Furthermore, managing these pieces of information individually is inefficient, so a system that can manage them all comprehensively is needed.

[1032] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1033] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and suggesting appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for suggesting predetermined menus based on predetermined food life data and generating a list of missing foods based on the menus, means for receiving receipt images photographed by the user and extracting purchased food information using OCR technology, and means for suggesting menus using a learning algorithm based on the user's past consumption data and current inventory data and generating a list of missing ingredients, thereby enabling users to comprehensively and efficiently manage various aspects of their daily lives.

[1034] "User" refers to any individual or organization that uses this system.

[1035] "Clothing image" refers to image data of clothing owned by the user.

[1036] "Feature information" refers to attribute information such as clothing type, color, and design extracted through image analysis.

[1037] "Food information" refers to detailed information about the food purchased by the user, such as the food name, quantity, purchase date and time, etc.

[1038] A "database" refers to an information management system for systematically storing analyzed information.

[1039] "Temperature data" refers to information about the current temperature obtained from an external weather data provider.

[1040] "Coordination suggestions" refers to a function that suggests optimal clothing combinations based on the user's clothing characteristics information and temperature data.

[1041] "Food life data" refers to life cycle data about food, such as expiration dates and inventory information.

[1042] "Menu suggestion" refers to a function that suggests optimal meal menus based on the user's inventory information and consumption data.

[1043] "List of missing foods" refers to a list of additional foods that need to be purchased in order to implement the proposed menu.

[1044] "Receipt image" refers to image data of the receipt for the purchased product photographed by the user.

[1045] "OCR technology" is an abbreviation for optical character recognition technology, and refers to the technology of extracting character data from images.

[1046] "Learning algorithm" refers to a machine learning technique that learns from a user's past consumption data and current inventory data to make predictions and suggestions.

[1047] This system efficiently manages users' daily lives, saving them time and effort. This system has four main functions: managing clothes in the closet, managing purchased ingredients, suggesting outfits based on temperature data, and creating menus and shopping lists.

[1048] Closet clothing management

[1049] Users take photos of their clothing using their smartphones or computers and upload them to a dedicated app. The app uses well-known image processing technologies such as TensorFlow and OpenCV. The device receives the uploaded image and uses an image recognition algorithm to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is then sent to a server and stored in a database for each user.

[1050] Management of purchased ingredients

[1051] Users take a photo of the receipt for the food they purchased and upload it to a dedicated app. The device then analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[1052] Coordination suggestions

[1053] The server obtains temperature data in real time from an external weather data provider (e.g., OpenWeatherMap API). It compares this temperature data with clothing data stored in the user's closet and suggests optimal outfits for the user. The device displays the outfit suggestions received from the server to the user. The user can review the suggested outfits and make fine adjustments as necessary.

[1054] Menu suggestions and shopping list generation

[1055] After returning home from the supermarket, the user takes a photo of their purchase receipt and uploads it to a dedicated app. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server. The server adds the food information to a database and updates the refrigerator inventory information. Using a learning algorithm (e.g., Scikit-learn), the system suggests a menu for the next day based on the user's past consumption data and current inventory data. The suggested menu makes the most of existing ingredients and generates a list of any missing ingredients. The generated list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to efficiently plan their shopping.

[1056] Specific examples

[1057] Example 1: Morning Outfit

[1058] 1. The user wakes up in the morning and opens the app on their smartphone.

[1059] 2. The device requests temperature data from the server.

[1060] 3. The server obtains real-time temperature data from an external weather data provider and compares it with the user's closet database to select the optimal outfit.

[1061] 4. The terminal displays the outfit suggestions received from the server to the user (e.g., "white shirt and blue jeans").

[1062] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[1063] Example 2: Dinner menu suggestions

[1064] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[1065] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[1066] 3. The server adds the food information to the database, updates the refrigerator inventory, and generates the next day's menu (e.g., "chicken curry") using a learning algorithm.

[1067] 4. The terminal displays the menu suggestions received from the server and the list of missing foods to the user.

[1068] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[1069] Prompt Sentence Examples

[1070] 1. "What's the temperature today?" → Server: "It's 20 degrees."

[1071] 2. "What is appropriate clothing for this temperature?" → Server: "I recommend a white shirt and blue jeans."

[1072] 3. "Based on what ingredients you have in the fridge, what would be a good menu item for tomorrow?" → Server: "I think it would be good to have the ingredients needed to make chicken curry."

[1073] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1074] Closet clothing management

[1075] Step 1:

[1076] Users take photos of their own clothes using a smartphone or computer and upload them to a dedicated app. The input is the photo of the clothes taken by the user, and the output is the image file uploaded to the app.

[1077] Step 2:

[1078] The terminal receives the uploaded image. The input is the image file from the user, and the output is the received image data. At this point, a communication protocol (e.g., HTTP) is used for receiving.

[1079] Step 3:

[1080] The device analyzes images using image recognition algorithms such as TensorFlow and OpenCV to extract feature information such as the type, color, and design of clothing. For example, it can obtain information such as "white shirt" or "blue jeans" from an image. The input is the received image data, and the output is the extracted feature information.

[1081] Step 4:

[1082] The terminal sends the analysis results to the server. The input is the feature information, and the output is the analyzed data sent via the network.

[1083] Step 5:

[1084] The server stores the transmitted feature information in a database for each user. For example, "User A's closet" is registered as "white shirt" and "blue jeans." The input is the transmitted analyzed data, and the output is the updated database.

[1085] Management of purchased ingredients

[1086] Step 1:

[1087] The user takes a photo of the receipt for the food they purchased and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[1088] Step 2:

[1089] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the received receipt image, and the output is the food information extracted through analysis. For example, information such as "milk" and "eggs" is extracted from the receipt.

[1090] Step 3:

[1091] The terminal sends the extracted information to the server. The input is the extracted food information, and the output is the food data transmitted via the network.

[1092] Step 4:

[1093] The server stores the food information in a database. The input is the food information sent, and the output is the updated database. For example, add "milk" and "eggs" to "User A's food inventory."

[1094] Coordination suggestions

[1095] Step 1:

[1096] The server sends a request to an external weather data provider (e.g., OpenWeatherMap API) to obtain real-time temperature data. The input is the weather data request, and the output is the obtained temperature data.

[1097] Step 2:

[1098] The server compares the acquired temperature data with the user's closet database. For example, it selects an outfit suitable for a temperature of 20 degrees. The input is temperature data and closet data, and the output is a suggestion of the optimal outfit.

[1099] Step 3:

[1100] The server sends the proposed coordinates to the terminal. The input is the proposed coordinates, and the output is the coordination proposal sent over the network.

[1101] Step 4:

[1102] The terminal displays the coordinate suggestion received from the server to the user. The input is the suggested coordinate information, and the output is the coordinate displayed on the terminal.

[1103] Step 5:

[1104] The user can check the suggested outfits and either accept them as they are or adjust them themselves to choose the outfit. The input is the displayed outfit, and the output is the final outfit decided by the user.

[1105] Menu suggestions and shopping list generation

[1106] Step 1:

[1107] After returning home from the supermarket, the user takes a photo of the receipt and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[1108] Step 2:

[1109] The device analyzes the received receipt image using OCR technology and extracts information about the purchased food. The input is the received receipt image, and the output is the food information extracted by the analysis.

[1110] Step 3:

[1111] The terminal sends the purchased food information to the server. The input is the extracted food information, and the output is the food data sent via the network.

[1112] Step 4:

[1113] The server adds the food information to the database and updates the refrigerator inventory information. The input is the submitted food information, and the output is the updated refrigerator inventory data.

[1114] Step 5:

[1115] The server uses a learning algorithm (e.g., Scikit-learn) to suggest the next day's menu based on the user's past consumption data and current inventory data. The input is consumption data and inventory data, and the output is the suggested menu.

[1116] Step 6:

[1117] The server generates a list of missing ingredients and sends it to the terminal. The input is the proposed menu and the output is the missing ingredients list.

[1118] Step 7:

[1119] The terminal displays the received menu suggestions and the list of missing foods to the user. The input is the suggested menu and the list of missing foods, and the output is the menu and list displayed on the terminal.

[1120] Step 8:

[1121] The user can then review the suggested menu and list of ingredients needed to efficiently plan their shopping. The input is the displayed menu and list, and the output is the user's final shopping plan.

[1122] (Application example 1)

[1123] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1124] Today's busy consumers are seeking systems that can efficiently manage the items in their closets and refrigerators and suggest optimal outfits and meal plans in their daily lives. They also have numerous needs for virtual store shopping experiences, such as suggestions for coordinating items with items they plan to purchase and smooth food inventory management. Conventional systems have difficulty meeting these needs in an integrated manner. Therefore, a new system that can comprehensively and efficiently support users' lives is needed.

[1125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1126] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, means for suggesting coordination with products that the user is considering purchasing in a virtual store, and means for managing information about purchased products and keeping track of the user's inventory. This allows the user to efficiently manage their daily closet and refrigerator, and enables optimal coordination suggestions and inventory management through a shopping experience in a virtual store.

[1127] "User" means an individual or corporation that uses this system.

[1128] "Clothing feature information" is data such as clothing type, color, and design extracted using image recognition technology.

[1129] "Food life data" refers to information on the type, quantity, and date of consumption of food purchased and consumed.

[1130] A "menu" is a meal plan suggested to a user, including the types and amounts of ingredients used.

[1131] A "candidate product for purchase" is a product that the user is considering purchasing in the virtual store.

[1132] "Temperature data" is current temperature information obtained from an external weather information service.

[1133] "Coordination" is a suggestion on how to combine and apply clothing.

[1134] A "receipt image" is an image of a receipt that lists details of the products purchased by the user.

[1135] "Inventory" refers to the quantity and type of items, such as food and clothing, that a user has already purchased and possesses.

[1136] A "learning algorithm" is a computational method for analyzing past data and predicting future trends and patterns.

[1137] A "virtual store" is a virtual shopping mall where users can browse and purchase products online.

[1138] The "server" is a computer device that serves as the center of the system and stores, analyzes, and makes recommendations on data.

[1139] This invention is a system designed to enable users to efficiently manage their lifestyles. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[1140] First, a user uses a smartphone application to take a photo of their clothing and upload it to the application. The device analyzes the uploaded image and uses an image recognition algorithm (e.g., Google Cloud Vision API) to extract feature information such as the type, color, and design of the clothing. The extracted feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1141] Next, the user takes a photo of the receipt for the purchased food and uploads it to the application. The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows the server to keep track of the inventory status of the ingredients purchased by the user.

[1142] In addition, the server obtains real-time temperature data from external weather information services (e.g., OpenWeather API), compares this data with the user's clothing characteristics, and suggests appropriate outfits for the user. The suggested outfits are then displayed to the user on their device.

[1143] The server also uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is then sent to the device and displayed to the user.

[1144] In particular, in the virtual store, users can receive suggestions for coordinating products with potential purchases. Purchased product information is managed by the server and added to the user's inventory database. This function enhances the virtual shopping experience.

[1145] Specific examples include suggestions for morning outfits and dinner menus. For example, when a user wakes up in the morning and opens a smartphone application, the device requests temperature data from the server, and the server compares the real-time temperature data with the user's closet information to select the optimal outfit and display it on the device.

[1146] When a user returns home from the supermarket and uploads their purchase receipt to the application, the device analyzes the received receipt image using OCR technology and sends the purchased food information to the server, which then analyzes the data and presents the user with the next day's menu and a list of any missing foods.

[1147] An example prompt is, "The user has provided an image of a dress they would like to purchase. Please suggest outfits that would go well with this dress. The temperature is 20 degrees and the user prefers a casual style. The user has the following items in their closet: a red jacket, black leather pants, and a white T-shirt. Which would go well together?"

[1148] In this way, this system efficiently manages the user's daily life and supports a more comfortable life.

[1149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1150] Step 1:

[1151] Users use a smartphone application to take a photo of their clothing and upload it to the application. This input data is an image file, which is received by the device.

[1152] Step 2:

[1153] The device analyzes the uploaded image file using an image recognition algorithm (e.g., Google Cloud Vision API) and extracts characteristic information such as the type of clothing, color, and design. The extracted characteristic information is output as data and sent to the server.

[1154] Step 3:

[1155] The server stores the received feature information in a database for each user, which is later used to suggest outfits.

[1156] Step 4:

[1157] The user takes a photo of the receipt for the food they purchased and uploads it to the application. This input data is also an image file, and the device receives it.

[1158] Step 5:

[1159] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted food information is output as data and sent to the server.

[1160] Step 6:

[1161] The server stores the received food information in a designated database, allowing the user to keep track of the inventory of ingredients purchased.

[1162] Step 7:

[1163] The server obtains real-time temperature data from an external weather information service (e.g., OpenWeather API). This temperature data is input data and is received by the server.

[1164] Step 8:

[1165] The server compares the temperature data with the user's clothing characteristics information and uses a generative AI model to suggest appropriate outfits. This output data is the outfit suggestion and is sent to the device.

[1166] Step 9:

[1167] The terminal displays the coordinated outfit suggestions received from the server to the user, who can then check the suggested outfits and either accept them or make minor adjustments.

[1168] Step 10:

[1169] The server uses the refrigerator's inventory data, past consumption data, and a learning algorithm to propose a menu for the next day. This output data is the menu proposal, and is sent to the terminal along with a list of missing ingredients.

[1170] Step 11:

[1171] The terminal displays the menu suggestions and the list of missing foods received from the server to the user, who then checks the suggested menu and list and makes a plan to purchase the necessary ingredients.

[1172] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1173] MODE FOR CARRYING OUT THE INVENTION

[1174] The present invention is a system for efficiently managing a user's daily life, saving time and effort. In particular, it has the function of recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions.

[1175] Closet clothing management

[1176] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1177] Management of purchased ingredients

[1178] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[1179] Coordination suggestions

[1180] The server obtains temperature data from an external source in real time. It compares the temperature data with the user's clothing characteristics information and proposes appropriate outfits to the user. In addition, it uses an emotion engine to obtain the user's emotional information and integrates the temperature data with the emotional information to optimize the outfit proposals. This allows the device to display the optimal outfits to the user. The user can review the proposed outfits and make fine adjustments if necessary.

[1181] Menu suggestions and shopping list generation

[1182] The server references the user's past consumption data based on refrigerator inventory data. Using a learning algorithm, the server takes into account the user's emotional information obtained by the emotion engine when generating the next day's menu. The proposed menu maximizes the use of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the device and displayed to the user. The user can check the proposed menu and the list of missing foods to efficiently plan their shopping.

[1183] Specific examples

[1184] Example 1: Morning Outfit

[1185] 1. The user wakes up in the morning and opens the app on their smartphone.

[1186] 2. The device requests the current temperature data and the user's emotion information from the server.

[1187] 3. The server obtains real-time temperature data from an external API and simultaneously analyzes the user's emotional information using an emotion engine.

[1188] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[1189] 5. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[1190] 6. The user checks the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[1191] Example 2: Dinner menu suggestions

[1192] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[1193] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[1194] 3. The server adds the food information to the database and updates the refrigerator inventory information. It uses an emotion engine to obtain the user's emotional information and uses a learning algorithm to generate the next day's menu.

[1195] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[1196] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[1197] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions tailored to their emotions, and live a richer life.

[1198] The processing flow will be explained below.

[1199] Closet clothing management

[1200] Step 1:

[1201] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[1202] Step 2:

[1203] The terminal receives the uploaded image.

[1204] Step 3:

[1205] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[1206] Step 4:

[1207] The terminal transmits the analysis results to the server.

[1208] Step 5:

[1209] The server stores the received clothing data in a database for each user.

[1210] Management of purchased ingredients

[1211] Step 1:

[1212] Users take a photo of the receipt for the food they purchased and upload it to the app.

[1213] Step 2:

[1214] The terminal receives the uploaded receipt image.

[1215] Step 3:

[1216] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[1217] Step 4:

[1218] The terminal transmits the analyzed food information to the server.

[1219] Step 5:

[1220] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[1221] Coordination suggestions

[1222] Step 1:

[1223] The server obtains temperature data for the user's location in real time (using an external API).

[1224] Step 2:

[1225] The server uses an emotion engine to acquire and analyze the user's emotion information.

[1226] Step 3:

[1227] The server compares the acquired temperature data and emotion information with the clothing information recorded in the user's closet database.

[1228] Step 4:

[1229] The server selects the best outfit that matches the temperature and your emotions.

[1230] Step 5:

[1231] The terminal displays the coordination suggestions received from the server on the user's app.

[1232] Step 6:

[1233] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[1234] Menu suggestions and shopping list generation

[1235] Step 1:

[1236] The server refers to the user's past consumption data based on the refrigerator inventory data.

[1237] Step 2:

[1238] The server uses an emotion engine to acquire and analyze the user's emotion information.

[1239] Step 3:

[1240] The server uses a learning algorithm to generate the next day's menu, taking emotional information into account.

[1241] Step 4:

[1242] The server creates a list of missing foods based on the generated menu.

[1243] Step 5:

[1244] The device displays the menu suggestions and shopping list received from the server in the user's app.

[1245] Step 6:

[1246] Users can plan their shopping by reviewing suggested menus and a list of missing foods.

[1247] This system allows users to efficiently manage the items in their closets and refrigerators and receive optimal suggestions tailored to their mood.

[1248] Example 2

[1249] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1250] In modern life, many users need to efficiently manage their daily lives to save time and effort. However, current systems have difficulty consistently managing clothing and food inventory, and suggesting appropriate outfits and menus. Furthermore, suggestions rarely reflect the user's mood or emotions, which prevents users from increasing their satisfaction. Therefore, there is a need for a system that can comprehensively support users' daily lives and make optimal suggestions based on their emotions.

[1251] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1252] In this invention, the server includes means for acquiring emotional information of a user and optimizing clothing coordination suggestions based on the emotional information, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, and means for optimizing the menus using a learning algorithm based on the emotional information and the user's past consumption data and current inventory data, thereby enabling the user to efficiently manage the items in their closet and refrigerator and receive optimal suggestions based on their emotions.

[1253] "User" means an individual or organization that uses this system.

[1254] "Clothing images" refers to photographs or image data of clothing owned by the user.

[1255] "Means for analyzing images and extracting characteristic information about clothing" refers to algorithms and technologies that analyze images of clothing and extract information such as color, type, and design.

[1256] "Food information" refers to data such as the type, quantity, and price of food purchased by the user.

[1257] The "predetermined database" refers to a database system for storing information about a user's clothing and purchased food.

[1258] "Temperature Data" refers to data regarding local temperatures obtained from external weather information services.

[1259] "User emotion information" refers to data that shows the results of analyzing the user's emotions and moods.

[1260] "Specified food life data" refers to data regarding food storage conditions, expiration dates, etc.

[1261] "Means for suggesting menus and generating a list of missing foods" refers to algorithms and technologies that suggest appropriate cooking menus based on the ingredients in the refrigerator and the user's past consumption patterns, and that create a list of missing foods for that purpose.

[1262] A "learning algorithm" is an algorithm that learns patterns from past data and makes inferences and predictions about new data.

[1263] MODE FOR CARRYING OUT THE INVENTION

[1264] The present invention is a system for efficiently managing a user's daily life and saving time and effort. In particular, it has a function for recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions. Specific embodiments of this system will be described in detail below.

[1265] Closet clothing management

[1266] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms (such as TensorFlow or OpenCV) to analyze feature information such as the type, color, and design of the clothing. The analyzed feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1267] For example, if a user uploads a photo of a blue shirt, the device analyzes the image and sends the characteristic information that the shirt is blue to the server, which stores this information in the user's database.

[1268] Management of purchased ingredients

[1269] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[1270] For example, when a user uploads a receipt, the device uses OCR technology to send the information about "milk," "eggs," and "bread" to the server, which adds this information to the database and updates the user's refrigerator inventory.

[1271] Coordination suggestions

[1272] The server acquires real-time temperature data using an external API (such as the OpenWeatherMap API). It then acquires the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The server compares the acquired temperature data and emotional information with the user's closet database and generates an appropriate outfit. The generated outfit suggestions are sent to the device and displayed to the user.

[1273] For example, if the server receives data indicating that the temperature is 28 degrees and the user is in good spirits, the server will suggest a white shirt and blue jeans. This suggestion will be sent to the device and displayed to the user.

[1274] Menu suggestions and shopping list generation

[1275] The server generates a menu based on refrigerator inventory data and the user's past consumption data. It also takes into account the user's emotional information using an emotion engine. A learning algorithm (such as Scikit-learn or TensorFlow) is used in the generation process. The proposed menu makes the most of existing ingredients, and any ingredients that are missing are generated as a list. This list is sent to the device and displayed to the user.

[1276] For example, if the server recognizes that the user has chicken, onions, and potatoes in the refrigerator and considers that the user feels like refreshing, it will suggest chicken curry as a menu item for the next day. It will also add missing spices to the shopping list. This list is sent to the terminal and displayed to the user.

[1277] Prompt Sentence Examples

[1278] "Write a Python program that uses temperature data and user emotion information to generate suggestions for morning outfits."

[1279] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions based on their emotions, and live a richer life.

[1280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1281] Step 1:

[1282] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The input is the image of the clothing taken. The device receives the uploaded image and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze the image for feature information such as the type, color, and design of the clothing. This analysis extracts feature information such as color, type, and design. The extracted feature information is sent to the server and stored in a database for each user. As an output, the feature information of the clothing is saved in a database for each user. For example, if a user uploads an image of a blue shirt, the device analyzes the image and extracts the feature information "blue shirt, long sleeves, size M" and sends it to the server.

[1283] Step 2:

[1284] The user takes a photo of the receipt for the food they purchased and uploads it to the app. The input is an image of the receipt for the purchased food. The device analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. Specifically, it extracts the product name, quantity, price, and other information listed on the receipt as text data. The extracted information is sent to the server and stored in a database. As an output, the user's food purchase information is added to the database. For example, if a user uploads a receipt that lists "Milk 200 yen," "Eggs 150 yen," and "Bread 300 yen," the device analyzes it using OCR technology and sends this information to the server. The server adds this information to the database.

[1285] Step 3:

[1286] The server obtains real-time temperature data using an external API (such as the OpenWeatherMap API). The input is the temperature data from the external API. The obtained temperature data is analyzed and the results are stored on the server. Next, the server obtains the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The input is the user's emotional data obtained by the emotion engine. This emotional data is analyzed and stored on the server. Once the temperature data and emotional information are collected, the server compares this information with the user's closet database and generates an appropriate outfit. As an output, the server generates an optimal outfit suggestion and sends it to the device. For example, if the server receives data that the temperature is 28 degrees Celsius and sunny, and information that the user is feeling cheerful, the server will suggest an outfit consisting of a white shirt and blue jeans.

[1287] Step 4:

[1288] The terminal displays the outfit suggestions received from the server to the user. The displayed suggestions are presented visually on the screen. The input is the outfit suggestions sent from the server. The user can review the displayed outfit and either accept it as is or fine-tune it to suit their own preferences. The output is the user's final outfit selection. For example, the user can review the suggested white shirt and blue jeans and choose to accept them as is or select different pants.

[1289] Step 5:

[1290] The server generates the next day's menu based on refrigerator inventory data and the user's past consumption data. The inputs are refrigerator inventory data and past consumption data. In addition, the user's emotional information is also obtained (input) using an emotion engine and taken into consideration. A learning algorithm (such as Scikit-learn or TensorFlow) is used to make the most of ingredients and identify any missing ingredients. The output is a menu suggestion for the user and a list of missing foods. For example, if the server recognizes that there is chicken, onions, and potatoes in the refrigerator, it will suggest chicken curry as the next day's menu based on the user's emotional data and add any missing spices to the list.

[1291] Step 6:

[1292] The terminal displays the menu suggestions and the list of missing foods received from the server to the user. The input is the menu suggestions and ingredient list sent from the server. The user checks this and makes a shopping plan. For example, the terminal displays chicken curry and a list of missing spices, and the user uses the list to plan the necessary shopping. The output is an efficient shopping list.

[1293] (Application example 2)

[1294] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1295] Conventional systems were unable to analyze the user's emotions and make suggestions based on them, making it difficult to propose coordination and menus that matched the user's mood and emotions. Furthermore, because suggestions were not based on the user's emotions, it was not possible to expect an improvement in satisfaction.

[1296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1297] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store it in a predetermined database, means for analyzing the user's emotions using an emotion recognition model and acquiring emotional information, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and the characteristic information and emotional information about the user's clothes, and means for proposing a predetermined menu based on predetermined food inventory data and the user's emotional information and generating a list of foods that are in short supply based on the menu, thereby enabling optimal coordination and menu proposals that take into account the user's emotions and real-time temperature data.

[1298] A "user" is someone who utilizes the system to effectively manage their life.

[1299] "Clothing image" is photographic data of clothing owned by the user.

[1300] "Characteristic information" is information such as the type, color, and design of clothing extracted through image analysis.

[1301] "Information about food" is data including the name, quantity, price, etc. of the food purchased by the user.

[1302] The "database" is a storage device for storing analyzed characteristic information and information about food.

[1303] An "emotion recognition model" is an algorithm for analyzing a user's emotions and acquiring emotional information.

[1304] "Emotion information" is data that represents the user's current emotion.

[1305] "Temperature data" is information about current and forecast temperatures.

[1306] "Coordination suggestions" are optimal clothing combinations suggested based on the user's clothing characteristics information and temperature data.

[1307] "Food inventory data" is data that indicates the inventory status of food items owned by the user.

[1308] "Menu Suggestion" is a menu of dishes suggested based on food inventory data and emotional information.

[1309] The "missing food list" is a list of foods that the user does not currently own that are required to create the proposed menu.

[1310] "Analysis" refers to processing the received images and information to extract the necessary data.

[1311] This invention is a system for efficiently managing a user's daily life and saving time and effort. This system improves the user experience by recognizing the user's emotions and suggesting outfits and menus that suit those emotions. Specific embodiments for implementing the invention are described below.

[1312] Clothing Management

[1313] Users use their smartphones to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze the clothing's characteristics, such as type, color, and design. This analyzed information is sent to a server and stored in a database for each user. This database is then used to suggest outfits.

[1314] Food Management

[1315] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[1316] emotion recognition

[1317] The device captures the user's facial image in real time using a camera built into a smartphone or smart glasses. It then analyzes the user's emotions using an emotion recognition model (e.g., a deep learning model trained using Keras). The analyzed emotion information is sent to the server and used to suggest outfits and menus.

[1318] Coordination suggestions

[1319] The server obtains real-time temperature data using an external API. It integrates the temperature data with the user's clothing characteristics and emotional information to suggest appropriate outfits to the user. The device displays the outfit suggestions received from the server to the user. For example, it may suggest, "If today's weather is sunny and you are feeling happy, we recommend a white shirt and blue jeans."

[1320] Menu suggestions and shopping list generation

[1321] The server proposes a predetermined menu based on refrigerator inventory data and the user's emotional information. The proposed menu makes the most of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the terminal and displayed to the user. The user can check the proposed menu and the list of missing foods to plan their shopping efficiently.

[1322] Specific examples

[1323] Example 1: Morning outfit suggestions

[1324] 1. The user wakes up in the morning and opens the app on their smartphone.

[1325] 2. The device requests the current temperature data and the user's emotion information from the server.

[1326] 3. The server obtains real-time temperature data from an external API and analyzes the user's emotions using an emotion recognition model.

[1327] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[1328] 5. The device displays the outfit suggestions received from the server to the user. For example, "If the weather is sunny today and you're feeling happy, I recommend a white shirt and blue jeans."

[1329] Example prompt sentence:

[1330] "To suggest a new outfit, please tell us what you would wear if you were feeling happy and the weather today was sunny."

[1331] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1332] Step 1:

[1333] The user opens the smartphone app, takes a picture of the clothing they own, and uploads it to the app.

[1334] Input: Photographed image of clothing

[1335] Output: Image upload

[1336] Specific operation: The user takes a photo of their clothing using their smartphone and taps the upload button in the app to send the image to the server.

[1337] Step 2:

[1338] The terminal receives the uploaded image and analyzes the clothing's characteristic information using an image recognition algorithm.

[1339] Input: Uploaded clothing image

[1340] Output: Analyzed feature information (type, color, design, etc.)

[1341] Specific operation: The device preprocesses the received image and uses an image recognition algorithm (e.g., a CNN model) to identify the type, color, and design of the clothing.

[1342] Step 3:

[1343] The terminal transmits the analyzed feature information to the server and stores it in a database.

[1344] Input: Analyzed clothing feature information

[1345] Output: Feature information sent to the server and stored in the database

[1346] Specific operation: The device converts the identified feature information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[1347] Step 4:

[1348] The user takes a photo of the receipt for the food they purchased and uploads it to the app.

[1349] Input: Image of the receipt

[1350] Output: Upload receipt image

[1351] Specific operation: The user takes a photo of the receipt for the purchased food using their smartphone and taps the upload button in the app to send the image to the server.

[1352] Step 5:

[1353] The device analyzes the uploaded receipt image using OCR technology to extract information about the food purchased.

[1354] Input: Uploaded receipt image

[1355] Output: Extracted food information (name, quantity, price, etc.)

[1356] Specific operation: The terminal passes the received receipt image through an OCR (optical character recognition) engine, extracts character data, and analyzes food information.

[1357] Step 6:

[1358] The terminal transmits the extracted food information to the server and stores it in a database.

[1359] Input: Extracted food information

[1360] Output: Food information sent to the server and stored in the database

[1361] Specific operation: The terminal converts the extracted food information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[1362] Step 7:

[1363] The terminal uses a camera built into a smartphone or smart glasses to capture a user's facial image in real time.

[1364] Input: Real-time captured face image

[1365] Output: Get face image

[1366] Specific operation: While the user is using the app, the device activates the camera and periodically captures images of the user's face.

[1367] Step 8:

[1368] The terminal analyzes the user's emotions using the emotion recognition model and acquires emotion information.

[1369] Input: Real-time captured face image

[1370] Output: Analyzed emotion information (e.g., happiness, sadness)

[1371] Specific operation: The device inputs the acquired facial image into an emotion recognition model (e.g., a deep learning model trained with Keras) to identify the emotion.

[1372] Step 9:

[1373] The terminal transmits the acquired emotion information to the server and uses it to propose coordination and menu items.

[1374] Input: Parsed emotion information

[1375] Output: Emotion information sent to the server

[1376] Specific operation: The terminal sends the identified emotion information to the server and adds it to the user profile.

[1377] Step 10:

[1378] The server uses an external API to obtain real-time temperature data.

[1379] Input: Temperature data obtained from an external API

[1380] Output: Real-time temperature data

[1381] Specific operation: The server sends a request to an external API that provides temperature data and obtains the current temperature.

[1382] Step 11:

[1383] The server integrates the temperature data with the user's clothing characteristics information and emotion information to suggest appropriate outfits to the user.

[1384] Input: Temperature data, clothing feature information, emotion information

[1385] Output: The optimal outfit suggested to the user

[1386] Specific operation: The server compares the temperature data with the emotional information, and refers to the clothing feature database to select the optimal outfit.

[1387] Step 12:

[1388] The terminal displays the coordination proposal received from the server to the user.

[1389] Input: Coordinate proposal from the server

[1390] Output: Coordination suggestions displayed on the user's smartphone or smart glasses

[1391] Specific operation: The device displays the coordination suggestions received from the server within the app, allowing the user to easily check them.

[1392] Step 13:

[1393] The server proposes a predetermined menu based on the refrigerator's inventory data and the user's emotional information.

[1394] Input: refrigerator inventory data, emotional information

[1395] Output: Suggested menu

[1396] Specific operation: The server compares the refrigerator inventory data with the user's emotional information to generate the optimal menu.

[1397] Step 14:

[1398] The server generates a list of missing foods based on the proposed menu and sends it to the terminal.

[1399] Input: Suggested menu

[1400] Output: List of missing foods

[1401] Specific operation: The server creates a list of ingredients required based on the proposed menu, compares it with existing stock, and identifies any ingredients that are in short supply. It then generates a list of ingredients that are in short supply and sends it to the user's device.

[1402] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1404] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1405] [Fourth embodiment]

[1406] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1407] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1409] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1413] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1414] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1415] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1416] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1417] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1418] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] MODE FOR CARRYING OUT THE INVENTION

[1420] This invention is a system for efficiently managing a user's daily life, saving time and effort. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[1421] Closet clothing management

[1422] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1423] Management of purchased ingredients

[1424] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the received receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[1425] Coordination suggestions

[1426] The server receives temperature data from an external device in real time. It compares the temperature data with the user's clothing characteristics information and suggests appropriate outfits to the user. The device then displays the optimal outfits to the user. The user can then review the suggested outfits and make fine adjustments if necessary.

[1427] Menu suggestions and shopping list generation

[1428] The server uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to plan their shopping efficiently.

[1429] Specific examples

[1430] Example 1: Morning Outfit

[1431] 1. The user wakes up in the morning and opens the app on their smartphone.

[1432] 2. The device requests temperature data from the server.

[1433] 3. The server obtains real-time temperature data from an external API and compares it with the user's closet database to select the optimal outfit.

[1434] 4. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[1435] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[1436] Example 2: Dinner menu suggestions

[1437] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[1438] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[1439] 3. The server adds the food information to the database, updates the refrigerator inventory, and uses a learning algorithm to generate data to suggest menus for the next day.

[1440] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[1441] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[1442] This system allows users to efficiently manage the items in their closets and refrigerators, freeing them from small everyday worries and allowing them to live a richer life.

[1443] The processing flow will be explained below.

[1444] Closet clothing management

[1445] Step 1:

[1446] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[1447] Step 2:

[1448] The terminal receives the uploaded image.

[1449] Step 3:

[1450] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[1451] Step 4:

[1452] The terminal transmits the analysis results to the server.

[1453] Step 5:

[1454] The server stores the received clothing data in a database for each user.

[1455] Management of purchased ingredients

[1456] Step 1:

[1457] Users take a photo of the receipt for the food they purchased and upload it to the app.

[1458] Step 2:

[1459] The terminal receives the uploaded receipt image.

[1460] Step 3:

[1461] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[1462] Step 4:

[1463] The terminal transmits the analyzed food information to the server.

[1464] Step 5:

[1465] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[1466] Coordination suggestions

[1467] Step 1:

[1468] The server obtains temperature data for the user's location in real time (using an external API).

[1469] Step 2:

[1470] The server compares the acquired temperature data with the clothing information recorded in the user's closet database.

[1471] Step 3:

[1472] The server selects the best outfit for the temperature.

[1473] Step 4:

[1474] The terminal displays the coordination suggestions received from the server on the user's app.

[1475] Step 5:

[1476] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[1477] Menu suggestions and shopping list generation

[1478] Step 1:

[1479] The server refers to the user's past consumption data based on the refrigerator inventory data.

[1480] Step 2:

[1481] The server uses a learning algorithm to generate the next day's menu.

[1482] Step 3:

[1483] The server creates a list of missing foods based on the generated menu.

[1484] Step 4:

[1485] The device displays the menu suggestions and shopping list received from the server in the user's app.

[1486] Step 5:

[1487] The user reviews the suggested menu and plans their shopping based on the list of missing foods.

[1488] Through the above steps, the system can efficiently assist users in their daily lives and save them a lot of time and effort.

[1489] Example 1

[1490] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1491] In modern life, managing clothing and food inventory, as well as suggesting clothing coordination according to the weather and planning daily meal menus, are complicated tasks that require time and effort. Furthermore, managing these pieces of information individually is inefficient, so a system that can manage them all comprehensively is needed.

[1492] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1493] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and suggesting appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for suggesting predetermined menus based on predetermined food life data and generating a list of missing foods based on the menus, means for receiving receipt images photographed by the user and extracting purchased food information using OCR technology, and means for suggesting menus using a learning algorithm based on the user's past consumption data and current inventory data and generating a list of missing ingredients, thereby enabling users to comprehensively and efficiently manage various aspects of their daily lives.

[1494] "User" refers to any individual or organization that uses this system.

[1495] "Clothing image" refers to image data of clothing owned by the user.

[1496] "Feature information" refers to attribute information such as clothing type, color, and design extracted through image analysis.

[1497] "Food information" refers to detailed information about the food purchased by the user, such as the food name, quantity, purchase date and time, etc.

[1498] A "database" refers to an information management system for systematically storing analyzed information.

[1499] "Temperature data" refers to information about the current temperature obtained from an external weather data provider.

[1500] "Coordination suggestions" refers to a function that suggests optimal clothing combinations based on the user's clothing characteristics information and temperature data.

[1501] "Food life data" refers to life cycle data about food, such as expiration dates and inventory information.

[1502] "Menu suggestion" refers to a function that suggests optimal meal menus based on the user's inventory information and consumption data.

[1503] "List of missing foods" refers to a list of additional foods that need to be purchased in order to implement the proposed menu.

[1504] "Receipt image" refers to image data of the receipt for the purchased product photographed by the user.

[1505] "OCR technology" is an abbreviation for optical character recognition technology, and refers to the technology of extracting character data from images.

[1506] "Learning algorithm" refers to a machine learning technique that learns from a user's past consumption data and current inventory data to make predictions and suggestions.

[1507] This system efficiently manages users' daily lives, saving them time and effort. This system has four main functions: managing clothes in the closet, managing purchased ingredients, suggesting outfits based on temperature data, and creating menus and shopping lists.

[1508] Closet clothing management

[1509] Users take photos of their clothing using their smartphones or computers and upload them to a dedicated app. The app uses well-known image processing technologies such as TensorFlow and OpenCV. The device receives the uploaded image and uses an image recognition algorithm to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is then sent to a server and stored in a database for each user.

[1510] Management of purchased ingredients

[1511] Users take a photo of the receipt for the food they purchased and upload it to a dedicated app. The device then analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[1512] Coordination suggestions

[1513] The server obtains temperature data in real time from an external weather data provider (e.g., OpenWeatherMap API). It compares this temperature data with clothing data stored in the user's closet and suggests optimal outfits for the user. The device displays the outfit suggestions received from the server to the user. The user can review the suggested outfits and make fine adjustments as necessary.

[1514] Menu suggestions and shopping list generation

[1515] After returning home from the supermarket, the user takes a photo of their purchase receipt and uploads it to a dedicated app. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server. The server adds the food information to a database and updates the refrigerator inventory information. Using a learning algorithm (e.g., Scikit-learn), the system suggests a menu for the next day based on the user's past consumption data and current inventory data. The suggested menu makes the most of existing ingredients and generates a list of any missing ingredients. The generated list is sent to the device and displayed to the user. The user can check the suggested menu and the list of missing foods to efficiently plan their shopping.

[1516] Specific examples

[1517] Example 1: Morning Outfit

[1518] 1. The user wakes up in the morning and opens the app on their smartphone.

[1519] 2. The device requests temperature data from the server.

[1520] 3. The server obtains real-time temperature data from an external weather data provider and compares it with the user's closet database to select the optimal outfit.

[1521] 4. The terminal displays the outfit suggestions received from the server to the user (e.g., "white shirt and blue jeans").

[1522] 5. The user reviews the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[1523] Example 2: Dinner menu suggestions

[1524] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[1525] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[1526] 3. The server adds the food information to the database, updates the refrigerator inventory, and generates the next day's menu (e.g., "chicken curry") using a learning algorithm.

[1527] 4. The terminal displays the menu suggestions received from the server and the list of missing foods to the user.

[1528] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[1529] Prompt Sentence Examples

[1530] 1. "What's the temperature today?" → Server: "It's 20 degrees."

[1531] 2. "What is appropriate clothing for this temperature?" → Server: "I recommend a white shirt and blue jeans."

[1532] 3. "Based on what ingredients you have in the fridge, what would be a good menu item for tomorrow?" → Server: "I think it would be good to have the ingredients needed to make chicken curry."

[1533] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1534] Closet clothing management

[1535] Step 1:

[1536] Users take photos of their own clothes using a smartphone or computer and upload them to a dedicated app. The input is the photo of the clothes taken by the user, and the output is the image file uploaded to the app.

[1537] Step 2:

[1538] The terminal receives the uploaded image. The input is the image file from the user, and the output is the received image data. At this point, a communication protocol (e.g., HTTP) is used for receiving.

[1539] Step 3:

[1540] The device analyzes images using image recognition algorithms such as TensorFlow and OpenCV to extract feature information such as the type, color, and design of clothing. For example, it can obtain information such as "white shirt" or "blue jeans" from an image. The input is the received image data, and the output is the extracted feature information.

[1541] Step 4:

[1542] The terminal sends the analysis results to the server. The input is the feature information, and the output is the analyzed data sent via the network.

[1543] Step 5:

[1544] The server stores the transmitted feature information in a database for each user. For example, "User A's closet" is registered as "white shirt" and "blue jeans." The input is the transmitted analyzed data, and the output is the updated database.

[1545] Management of purchased ingredients

[1546] Step 1:

[1547] The user takes a photo of the receipt for the food they purchased and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[1548] Step 2:

[1549] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the received receipt image, and the output is the food information extracted through analysis. For example, information such as "milk" and "eggs" is extracted from the receipt.

[1550] Step 3:

[1551] The terminal sends the extracted information to the server. The input is the extracted food information, and the output is the food data transmitted via the network.

[1552] Step 4:

[1553] The server stores the food information in a database. The input is the food information sent, and the output is the updated database. For example, add "milk" and "eggs" to "User A's food inventory."

[1554] Coordination suggestions

[1555] Step 1:

[1556] The server sends a request to an external weather data provider (e.g., OpenWeatherMap API) to obtain real-time temperature data. The input is the weather data request, and the output is the obtained temperature data.

[1557] Step 2:

[1558] The server compares the acquired temperature data with the user's closet database. For example, it selects an outfit suitable for a temperature of 20 degrees. The input is temperature data and closet data, and the output is a suggestion of the optimal outfit.

[1559] Step 3:

[1560] The server sends the proposed coordinates to the terminal. The input is the proposed coordinates, and the output is the coordination proposal sent over the network.

[1561] Step 4:

[1562] The terminal displays the coordinate suggestion received from the server to the user. The input is the suggested coordinate information, and the output is the coordinate displayed on the terminal.

[1563] Step 5:

[1564] The user can check the suggested outfits and either accept them as they are or adjust them themselves to choose the outfit. The input is the displayed outfit, and the output is the final outfit decided by the user.

[1565] Menu suggestions and shopping list generation

[1566] Step 1:

[1567] After returning home from the supermarket, the user takes a photo of the receipt and uploads it to a dedicated app. The input is the receipt image taken by the user, and the output is the receipt image file uploaded to the app.

[1568] Step 2:

[1569] The device analyzes the received receipt image using OCR technology and extracts information about the purchased food. The input is the received receipt image, and the output is the food information extracted by the analysis.

[1570] Step 3:

[1571] The terminal sends the purchased food information to the server. The input is the extracted food information, and the output is the food data sent via the network.

[1572] Step 4:

[1573] The server adds the food information to the database and updates the refrigerator inventory information. The input is the submitted food information, and the output is the updated refrigerator inventory data.

[1574] Step 5:

[1575] The server uses a learning algorithm (e.g., Scikit-learn) to suggest the next day's menu based on the user's past consumption data and current inventory data. The input is consumption data and inventory data, and the output is the suggested menu.

[1576] Step 6:

[1577] The server generates a list of missing ingredients and sends it to the terminal. The input is the proposed menu and the output is the missing ingredients list.

[1578] Step 7:

[1579] The terminal displays the received menu suggestions and the list of missing foods to the user. The input is the suggested menu and the list of missing foods, and the output is the menu and list displayed on the terminal.

[1580] Step 8:

[1581] The user can then review the suggested menu and list of ingredients needed to efficiently plan their shopping. The input is the displayed menu and list, and the output is the user's final shopping plan.

[1582] (Application example 1)

[1583] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1584] Today's busy consumers are seeking systems that can efficiently manage the items in their closets and refrigerators and suggest optimal outfits and meal plans in their daily lives. They also have numerous needs for virtual store shopping experiences, such as suggestions for coordinating items with items they plan to purchase and smooth food inventory management. Conventional systems have difficulty meeting these needs in an integrated manner. Therefore, a new system that can comprehensively and efficiently support users' lives is needed.

[1585] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1586] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store in a predetermined database, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information about the user's clothes, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, means for suggesting coordination with products that the user is considering purchasing in a virtual store, and means for managing information about purchased products and keeping track of the user's inventory. This allows the user to efficiently manage their daily closet and refrigerator, and enables optimal coordination suggestions and inventory management through a shopping experience in a virtual store.

[1587] "User" means an individual or corporation that uses this system.

[1588] "Clothing feature information" is data such as clothing type, color, and design extracted using image recognition technology.

[1589] "Food life data" refers to information on the type, quantity, and date of consumption of food purchased and consumed.

[1590] A "menu" is a meal plan suggested to a user, including the types and amounts of ingredients used.

[1591] A "candidate product for purchase" is a product that the user is considering purchasing in the virtual store.

[1592] "Temperature data" is current temperature information obtained from an external weather information service.

[1593] "Coordination" is a suggestion on how to combine and apply clothing.

[1594] A "receipt image" is an image of a receipt that lists details of the products purchased by the user.

[1595] "Inventory" refers to the quantity and type of items, such as food and clothing, that a user has already purchased and possesses.

[1596] A "learning algorithm" is a computational method for analyzing past data and predicting future trends and patterns.

[1597] A "virtual store" is a virtual shopping mall where users can browse and purchase products online.

[1598] The "server" is a computer device that serves as the center of the system and stores, analyzes, and makes recommendations on data.

[1599] This invention is a system designed to enable users to efficiently manage their lifestyles. This system manages clothes in the closet, manages purchased ingredients, suggests outfits based on temperature data, suggests menus, and generates shopping lists.

[1600] First, a user uses a smartphone application to take a photo of their clothing and upload it to the application. The device analyzes the uploaded image and uses an image recognition algorithm (e.g., Google Cloud Vision API) to extract feature information such as the type, color, and design of the clothing. The extracted feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1601] Next, the user takes a photo of the receipt for the purchased food and uploads it to the application. The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows the server to keep track of the inventory status of the ingredients purchased by the user.

[1602] In addition, the server obtains real-time temperature data from external weather information services (e.g., OpenWeather API), compares this data with the user's clothing characteristics, and suggests appropriate outfits for the user. The suggested outfits are then displayed to the user on their device.

[1603] The server also uses consumption data and learning algorithms to suggest a menu for the next day based on refrigerator inventory data. The suggested menu makes the most of existing ingredients and generates a list of ingredients that are missing. The list is then sent to the device and displayed to the user.

[1604] In particular, in the virtual store, users can receive suggestions for coordinating products with potential purchases. Purchased product information is managed by the server and added to the user's inventory database. This function enhances the virtual shopping experience.

[1605] Specific examples include suggestions for morning outfits and dinner menus. For example, when a user wakes up in the morning and opens a smartphone application, the device requests temperature data from the server, and the server compares the real-time temperature data with the user's closet information to select the optimal outfit and display it on the device.

[1606] When a user returns home from the supermarket and uploads their purchase receipt to the application, the device analyzes the received receipt image using OCR technology and sends the purchased food information to the server, which then analyzes the data and presents the user with the next day's menu and a list of any missing foods.

[1607] An example prompt is, "The user has provided an image of a dress they would like to purchase. Please suggest outfits that would go well with this dress. The temperature is 20 degrees and the user prefers a casual style. The user has the following items in their closet: a red jacket, black leather pants, and a white T-shirt. Which would go well together?"

[1608] In this way, this system efficiently manages the user's daily life and supports a more comfortable life.

[1609] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1610] Step 1:

[1611] Users use a smartphone application to take a photo of their clothing and upload it to the application. This input data is an image file, which is received by the device.

[1612] Step 2:

[1613] The device analyzes the uploaded image file using an image recognition algorithm (e.g., Google Cloud Vision API) and extracts characteristic information such as the type of clothing, color, and design. The extracted characteristic information is output as data and sent to the server.

[1614] Step 3:

[1615] The server stores the received feature information in a database for each user, which is later used to suggest outfits.

[1616] Step 4:

[1617] The user takes a photo of the receipt for the food they purchased and uploads it to the application. This input data is also an image file, and the device receives it.

[1618] Step 5:

[1619] The device analyzes the received receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted food information is output as data and sent to the server.

[1620] Step 6:

[1621] The server stores the received food information in a designated database, allowing the user to keep track of the inventory of ingredients purchased.

[1622] Step 7:

[1623] The server obtains real-time temperature data from an external weather information service (e.g., OpenWeather API). This temperature data is input data and is received by the server.

[1624] Step 8:

[1625] The server compares the temperature data with the user's clothing characteristics information and uses a generative AI model to suggest appropriate outfits. This output data is the outfit suggestion and is sent to the device.

[1626] Step 9:

[1627] The terminal displays the coordinated outfit suggestions received from the server to the user, who can then check the suggested outfits and either accept them or make minor adjustments.

[1628] Step 10:

[1629] The server uses the refrigerator's inventory data, past consumption data, and a learning algorithm to propose a menu for the next day. This output data is the menu proposal, and is sent to the terminal along with a list of missing ingredients.

[1630] Step 11:

[1631] The terminal displays the menu suggestions and the list of missing foods received from the server to the user, who then checks the suggested menu and list and makes a plan to purchase the necessary ingredients.

[1632] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1633] MODE FOR CARRYING OUT THE INVENTION

[1634] The present invention is a system for efficiently managing a user's daily life, saving time and effort. In particular, it has the function of recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions.

[1635] Closet clothing management

[1636] Users use their smartphones or computers to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze characteristic information such as the type, color, and design of the clothing. The analyzed characteristic information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1637] Management of purchased ingredients

[1638] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[1639] Coordination suggestions

[1640] The server obtains temperature data from an external source in real time. It compares the temperature data with the user's clothing characteristics information and proposes appropriate outfits to the user. In addition, it uses an emotion engine to obtain the user's emotional information and integrates the temperature data with the emotional information to optimize the outfit proposals. This allows the device to display the optimal outfits to the user. The user can review the proposed outfits and make fine adjustments if necessary.

[1641] Menu suggestions and shopping list generation

[1642] The server references the user's past consumption data based on refrigerator inventory data. Using a learning algorithm, the server takes into account the user's emotional information obtained by the emotion engine when generating the next day's menu. The proposed menu maximizes the use of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the device and displayed to the user. The user can check the proposed menu and the list of missing foods to efficiently plan their shopping.

[1643] Specific examples

[1644] Example 1: Morning Outfit

[1645] 1. The user wakes up in the morning and opens the app on their smartphone.

[1646] 2. The device requests the current temperature data and the user's emotion information from the server.

[1647] 3. The server obtains real-time temperature data from an external API and simultaneously analyzes the user's emotional information using an emotion engine.

[1648] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[1649] 5. The terminal displays the outfit suggestions received from the server to the user. For example, it suggests "a white shirt and blue jeans."

[1650] 6. The user checks the suggested outfits and either adopts them as is or adjusts them themselves to choose their own outfits.

[1651] Example 2: Dinner menu suggestions

[1652] 1. After returning home from the supermarket, the user takes a photo of the receipt and uploads it to the app.

[1653] 2. The device analyzes the received receipt image using OCR technology and sends the purchased food information to the server.

[1654] 3. The server adds the food information to the database and updates the refrigerator inventory information. It uses an emotion engine to obtain the user's emotional information and uses a learning algorithm to generate the next day's menu.

[1655] 4. The device displays the menu suggestions received from the server (e.g., chicken curry) and a list of missing foods to the user.

[1656] 5. The user reviews the menu and shopping list and plans to purchase any missing ingredients.

[1657] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions tailored to their emotions, and live a richer life.

[1658] The processing flow will be explained below.

[1659] Closet clothing management

[1660] Step 1:

[1661] Users take a photo of the clothing using their smartphone camera and upload it to a dedicated app.

[1662] Step 2:

[1663] The terminal receives the uploaded image.

[1664] Step 3:

[1665] The device uses image recognition algorithms to analyze characteristic information such as the type of clothing (e.g., shirt, jeans), color, and design.

[1666] Step 4:

[1667] The terminal transmits the analysis results to the server.

[1668] Step 5:

[1669] The server stores the received clothing data in a database for each user.

[1670] Management of purchased ingredients

[1671] Step 1:

[1672] Users take a photo of the receipt for the food they purchased and upload it to the app.

[1673] Step 2:

[1674] The terminal receives the uploaded receipt image.

[1675] Step 3:

[1676] The terminal analyzes the purchased food information listed on the receipt using OCR (optical character recognition) technology.

[1677] Step 4:

[1678] The terminal transmits the analyzed food information to the server.

[1679] Step 5:

[1680] The server stores the received food data in a database and updates refrigerator inventory information for each user.

[1681] Coordination suggestions

[1682] Step 1:

[1683] The server obtains temperature data for the user's location in real time (using an external API).

[1684] Step 2:

[1685] The server uses an emotion engine to acquire and analyze the user's emotion information.

[1686] Step 3:

[1687] The server compares the acquired temperature data and emotion information with the clothing information recorded in the user's closet database.

[1688] Step 4:

[1689] The server selects the best outfit that matches the temperature and your emotions.

[1690] Step 5:

[1691] The terminal displays the coordination suggestions received from the server on the user's app.

[1692] Step 6:

[1693] The user can review the proposed coordination and implement it, or make minor adjustments if necessary.

[1694] Menu suggestions and shopping list generation

[1695] Step 1:

[1696] The server refers to the user's past consumption data based on the refrigerator inventory data.

[1697] Step 2:

[1698] The server uses an emotion engine to acquire and analyze the user's emotion information.

[1699] Step 3:

[1700] The server uses a learning algorithm to generate the next day's menu, taking emotional information into account.

[1701] Step 4:

[1702] The server creates a list of missing foods based on the generated menu.

[1703] Step 5:

[1704] The device displays the menu suggestions and shopping list received from the server in the user's app.

[1705] Step 6:

[1706] Users can plan their shopping by reviewing suggested menus and a list of missing foods.

[1707] This system allows users to efficiently manage the items in their closets and refrigerators and receive optimal suggestions tailored to their mood.

[1708] Example 2

[1709] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1710] In modern life, many users need to efficiently manage their daily lives to save time and effort. However, current systems have difficulty consistently managing clothing and food inventory, and suggesting appropriate outfits and menus. Furthermore, suggestions rarely reflect the user's mood or emotions, which prevents users from increasing their satisfaction. Therefore, there is a need for a system that can comprehensively support users' daily lives and make optimal suggestions based on their emotions.

[1711] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1712] In this invention, the server includes means for acquiring emotional information of a user and optimizing clothing coordination suggestions based on the emotional information, means for proposing predetermined menus based on predetermined food life data and generating a list of foods that are in short supply based on the menus, and means for optimizing the menus using a learning algorithm based on the emotional information and the user's past consumption data and current inventory data, thereby enabling the user to efficiently manage the items in their closet and refrigerator and receive optimal suggestions based on their emotions.

[1713] "User" means an individual or organization that uses this system.

[1714] "Clothing images" refers to photographs or image data of clothing owned by the user.

[1715] "Means for analyzing images and extracting characteristic information about clothing" refers to algorithms and technologies that analyze images of clothing and extract information such as color, type, and design.

[1716] "Food information" refers to data such as the type, quantity, and price of food purchased by the user.

[1717] The "predetermined database" refers to a database system for storing information about a user's clothing and purchased food.

[1718] "Temperature Data" refers to data regarding local temperatures obtained from external weather information services.

[1719] "User emotion information" refers to data that shows the results of analyzing the user's emotions and moods.

[1720] "Specified food life data" refers to data regarding food storage conditions, expiration dates, etc.

[1721] "Means for suggesting menus and generating a list of missing foods" refers to algorithms and technologies that suggest appropriate cooking menus based on the ingredients in the refrigerator and the user's past consumption patterns, and that create a list of missing foods for that purpose.

[1722] A "learning algorithm" is an algorithm that learns patterns from past data and makes inferences and predictions about new data.

[1723] MODE FOR CARRYING OUT THE INVENTION

[1724] The present invention is a system for efficiently managing a user's daily life and saving time and effort. In particular, it has a function for recognizing the user's emotions and reflecting them in coordination suggestions and menu suggestions. Specific embodiments of this system will be described in detail below.

[1725] Closet clothing management

[1726] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms (such as TensorFlow or OpenCV) to analyze feature information such as the type, color, and design of the clothing. The analyzed feature information is sent to a server and stored in a database for each user. This database is later used to suggest outfits.

[1727] For example, if a user uploads a photo of a blue shirt, the device analyzes the image and sends the characteristic information that the shirt is blue to the server, which stores this information in the user's database.

[1728] Management of purchased ingredients

[1729] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. The extracted information is sent to a server and stored in a database. This allows users to keep track of the inventory status of the ingredients they have purchased.

[1730] For example, when a user uploads a receipt, the device uses OCR technology to send the information about "milk," "eggs," and "bread" to the server, which adds this information to the database and updates the user's refrigerator inventory.

[1731] Coordination suggestions

[1732] The server acquires real-time temperature data using an external API (such as the OpenWeatherMap API). It then acquires the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The server compares the acquired temperature data and emotional information with the user's closet database and generates an appropriate outfit. The generated outfit suggestions are sent to the device and displayed to the user.

[1733] For example, if the server receives data indicating that the temperature is 28 degrees and the user is in good spirits, the server will suggest a white shirt and blue jeans. This suggestion will be sent to the device and displayed to the user.

[1734] Menu suggestions and shopping list generation

[1735] The server generates a menu based on refrigerator inventory data and the user's past consumption data. It also takes into account the user's emotional information using an emotion engine. A learning algorithm (such as Scikit-learn or TensorFlow) is used in the generation process. The proposed menu makes the most of existing ingredients, and any ingredients that are missing are generated as a list. This list is sent to the device and displayed to the user.

[1736] For example, if the server recognizes that the user has chicken, onions, and potatoes in the refrigerator and considers that the user feels like refreshing, it will suggest chicken curry as a menu item for the next day. It will also add missing spices to the shopping list. This list is sent to the terminal and displayed to the user.

[1737] Prompt Sentence Examples

[1738] "Write a Python program that uses temperature data and user emotion information to generate suggestions for morning outfits."

[1739] This system allows users to efficiently manage the items in their closets and refrigerators, receive optimal suggestions based on their emotions, and live a richer life.

[1740] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1741] Step 1:

[1742] Users use their smartphones or computers to take photos of their own clothing and upload them to the app. The input is the image of the clothing taken. The device receives the uploaded image and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze the image for feature information such as the type, color, and design of the clothing. This analysis extracts feature information such as color, type, and design. The extracted feature information is sent to the server and stored in a database for each user. As an output, the feature information of the clothing is saved in a database for each user. For example, if a user uploads an image of a blue shirt, the device analyzes the image and extracts the feature information "blue shirt, long sleeves, size M" and sends it to the server.

[1743] Step 2:

[1744] The user takes a photo of the receipt for the food they purchased and uploads it to the app. The input is an image of the receipt for the purchased food. The device analyzes the uploaded receipt image using OCR (optical character recognition) technology (such as Tesseract OCR) to extract information about the purchased food. Specifically, it extracts the product name, quantity, price, and other information listed on the receipt as text data. The extracted information is sent to the server and stored in a database. As an output, the user's food purchase information is added to the database. For example, if a user uploads a receipt that lists "Milk 200 yen," "Eggs 150 yen," and "Bread 300 yen," the device analyzes it using OCR technology and sends this information to the server. The server adds this information to the database.

[1745] Step 3:

[1746] The server obtains real-time temperature data using an external API (such as the OpenWeatherMap API). The input is the temperature data from the external API. The obtained temperature data is analyzed and the results are stored on the server. Next, the server obtains the user's emotional information using an emotion engine (such as IBM Watson Emotion Analysis). The input is the user's emotional data obtained by the emotion engine. This emotional data is analyzed and stored on the server. Once the temperature data and emotional information are collected, the server compares this information with the user's closet database and generates an appropriate outfit. As an output, the server generates an optimal outfit suggestion and sends it to the device. For example, if the server receives data that the temperature is 28 degrees Celsius and sunny, and information that the user is feeling cheerful, the server will suggest an outfit consisting of a white shirt and blue jeans.

[1747] Step 4:

[1748] The terminal displays the outfit suggestions received from the server to the user. The displayed suggestions are presented visually on the screen. The input is the outfit suggestions sent from the server. The user can review the displayed outfit and either accept it as is or fine-tune it to suit their own preferences. The output is the user's final outfit selection. For example, the user can review the suggested white shirt and blue jeans and choose to accept them as is or select different pants.

[1749] Step 5:

[1750] The server generates the next day's menu based on refrigerator inventory data and the user's past consumption data. The inputs are refrigerator inventory data and past consumption data. In addition, the user's emotional information is also obtained (input) using an emotion engine and taken into consideration. A learning algorithm (such as Scikit-learn or TensorFlow) is used to make the most of ingredients and identify any missing ingredients. The output is a menu suggestion for the user and a list of missing foods. For example, if the server recognizes that there is chicken, onions, and potatoes in the refrigerator, it will suggest chicken curry as the next day's menu based on the user's emotional data and add any missing spices to the list.

[1751] Step 6:

[1752] The terminal displays the menu suggestions and the list of missing foods received from the server to the user. The input is the menu suggestions and ingredient list sent from the server. The user checks this and makes a shopping plan. For example, the terminal displays chicken curry and a list of missing spices, and the user uses the list to plan the necessary shopping. The output is an efficient shopping list.

[1753] (Application example 2)

[1754] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1755] Conventional systems were unable to analyze the user's emotions and make suggestions based on them, making it difficult to propose coordination and menus that matched the user's mood and emotions. Furthermore, because suggestions were not based on the user's emotions, it was not possible to expect an improvement in satisfaction.

[1756] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1757] In this invention, the server includes means for receiving images of clothes owned by the user and analyzing the images to extract characteristic information about the clothes, means for receiving information about foods purchased by the user and analyzing the information to store it in a predetermined database, means for analyzing the user's emotions using an emotion recognition model and acquiring emotional information, means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and the characteristic information and emotional information about the user's clothes, and means for proposing a predetermined menu based on predetermined food inventory data and the user's emotional information and generating a list of foods that are in short supply based on the menu, thereby enabling optimal coordination and menu proposals that take into account the user's emotions and real-time temperature data.

[1758] A "user" is someone who utilizes the system to effectively manage their life.

[1759] "Clothing image" is photographic data of clothing owned by the user.

[1760] "Characteristic information" is information such as the type, color, and design of clothing extracted through image analysis.

[1761] "Information about food" is data including the name, quantity, price, etc. of the food purchased by the user.

[1762] The "database" is a storage device for storing analyzed characteristic information and information about food.

[1763] An "emotion recognition model" is an algorithm for analyzing a user's emotions and acquiring emotional information.

[1764] "Emotion information" is data that represents the user's current emotion.

[1765] "Temperature data" is information about current and forecast temperatures.

[1766] "Coordination suggestions" are optimal clothing combinations suggested based on the user's clothing characteristics information and temperature data.

[1767] "Food inventory data" is data that indicates the inventory status of food items owned by the user.

[1768] "Menu Suggestion" is a menu of dishes suggested based on food inventory data and emotional information.

[1769] The "missing food list" is a list of foods that the user does not currently own that are required to create the proposed menu.

[1770] "Analysis" refers to processing the received images and information to extract the necessary data.

[1771] This invention is a system for efficiently managing a user's daily life and saving time and effort. This system improves the user experience by recognizing the user's emotions and suggesting outfits and menus that suit those emotions. Specific embodiments for implementing the invention are described below.

[1772] Clothing Management

[1773] Users use their smartphones to take photos of their clothing and upload them to the app. The device receives the uploaded images and uses image recognition algorithms to analyze the clothing's characteristics, such as type, color, and design. This analyzed information is sent to a server and stored in a database for each user. This database is then used to suggest outfits.

[1774] Food Management

[1775] Users take a photo of the receipt for the food they purchased and upload it to the app. The device then analyzes the uploaded receipt image using OCR technology (e.g., Tesseract OCR) to extract information about the purchased food. The extracted information is sent to the server and stored in a database. This allows users to keep track of the inventory status of the ingredients they purchased.

[1776] emotion recognition

[1777] The device captures the user's facial image in real time using a camera built into a smartphone or smart glasses. It then analyzes the user's emotions using an emotion recognition model (e.g., a deep learning model trained using Keras). The analyzed emotion information is sent to the server and used to suggest outfits and menus.

[1778] Coordination suggestions

[1779] The server obtains real-time temperature data using an external API. It integrates the temperature data with the user's clothing characteristics and emotional information to suggest appropriate outfits to the user. The device displays the outfit suggestions received from the server to the user. For example, it may suggest, "If today's weather is sunny and you are feeling happy, we recommend a white shirt and blue jeans."

[1780] Menu suggestions and shopping list generation

[1781] The server proposes a predetermined menu based on refrigerator inventory data and the user's emotional information. The proposed menu makes the most of existing ingredients and generates a list of ingredients that are missing. The generated list is sent to the terminal and displayed to the user. The user can check the proposed menu and the list of missing foods to plan their shopping efficiently.

[1782] Specific examples

[1783] Example 1: Morning outfit suggestions

[1784] 1. The user wakes up in the morning and opens the app on their smartphone.

[1785] 2. The device requests the current temperature data and the user's emotion information from the server.

[1786] 3. The server obtains real-time temperature data from an external API and analyzes the user's emotions using an emotion recognition model.

[1787] 4. The server integrates the temperature data and emotional information and compares it with the user's closet database to select the optimal outfit.

[1788] 5. The device displays the outfit suggestions received from the server to the user. For example, "If the weather is sunny today and you're feeling happy, I recommend a white shirt and blue jeans."

[1789] Example prompt sentence:

[1790] "To suggest a new outfit, please tell us what you would wear if you were feeling happy and the weather today was sunny."

[1791] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1792] Step 1:

[1793] The user opens the smartphone app, takes a picture of the clothing they own, and uploads it to the app.

[1794] Input: Photographed image of clothing

[1795] Output: Image upload

[1796] Specific operation: The user takes a photo of their clothing using their smartphone and taps the upload button in the app to send the image to the server.

[1797] Step 2:

[1798] The terminal receives the uploaded image and analyzes the clothing's characteristic information using an image recognition algorithm.

[1799] Input: Uploaded clothing image

[1800] Output: Analyzed feature information (type, color, design, etc.)

[1801] Specific operation: The device preprocesses the received image and uses an image recognition algorithm (e.g., a CNN model) to identify the type, color, and design of the clothing.

[1802] Step 3:

[1803] The terminal transmits the analyzed feature information to the server and stores it in a database.

[1804] Input: Analyzed clothing feature information

[1805] Output: Feature information sent to the server and stored in the database

[1806] Specific operation: The device converts the identified feature information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[1807] Step 4:

[1808] The user takes a photo of the receipt for the food they purchased and uploads it to the app.

[1809] Input: Image of the receipt

[1810] Output: Upload receipt image

[1811] Specific operation: The user takes a photo of the receipt for the purchased food using their smartphone and taps the upload button in the app to send the image to the server.

[1812] Step 5:

[1813] The device analyzes the uploaded receipt image using OCR technology to extract information about the food purchased.

[1814] Input: Uploaded receipt image

[1815] Output: Extracted food information (name, quantity, price, etc.)

[1816] Specific operation: The terminal passes the received receipt image through an OCR (optical character recognition) engine, extracts character data, and analyzes food information.

[1817] Step 6:

[1818] The terminal transmits the extracted food information to the server and stores it in a database.

[1819] Input: Extracted food information

[1820] Output: Food information sent to the server and stored in the database

[1821] Specific operation: The terminal converts the extracted food information into a format suitable for the database and sends it to the server. The server stores the received data in the database.

[1822] Step 7:

[1823] The terminal uses a camera built into a smartphone or smart glasses to capture a user's facial image in real time.

[1824] Input: Real-time captured face image

[1825] Output: Get face image

[1826] Specific operation: While the user is using the app, the device activates the camera and periodically captures images of the user's face.

[1827] Step 8:

[1828] The terminal analyzes the user's emotions using the emotion recognition model and acquires emotion information.

[1829] Input: Real-time captured face image

[1830] Output: Analyzed emotion information (e.g., happiness, sadness)

[1831] Specific operation: The device inputs the acquired facial image into an emotion recognition model (e.g., a deep learning model trained with Keras) to identify the emotion.

[1832] Step 9:

[1833] The terminal transmits the acquired emotion information to the server and uses it to propose coordination and menu items.

[1834] Input: Parsed emotion information

[1835] Output: Emotion information sent to the server

[1836] Specific operation: The terminal sends the identified emotion information to the server and adds it to the user profile.

[1837] Step 10:

[1838] The server uses an external API to obtain real-time temperature data.

[1839] Input: Temperature data obtained from an external API

[1840] Output: Real-time temperature data

[1841] Specific operation: The server sends a request to an external API that provides temperature data and obtains the current temperature.

[1842] Step 11:

[1843] The server integrates the temperature data with the user's clothing characteristics information and emotion information to suggest appropriate outfits to the user.

[1844] Input: Temperature data, clothing feature information, emotion information

[1845] Output: The optimal outfit suggested to the user

[1846] Specific operation: The server compares the temperature data with the emotional information, and refers to the clothing feature database to select the optimal outfit.

[1847] Step 12:

[1848] The terminal displays the coordination proposal received from the server to the user.

[1849] Input: Coordinate proposal from the server

[1850] Output: Coordination suggestions displayed on the user's smartphone or smart glasses

[1851] Specific operation: The device displays the coordination suggestions received from the server within the app, allowing the user to easily check them.

[1852] Step 13:

[1853] The server proposes a predetermined menu based on the refrigerator's inventory data and the user's emotional information.

[1854] Input: refrigerator inventory data, emotional information

[1855] Output: Suggested menu

[1856] Specific operation: The server compares the refrigerator inventory data with the user's emotional information to generate the optimal menu.

[1857] Step 14:

[1858] The server generates a list of missing foods based on the proposed menu and sends it to the terminal.

[1859] Input: Suggested menu

[1860] Output: List of missing foods

[1861] Specific operation: The server creates a list of ingredients required based on the proposed menu, compares it with existing stock, and identifies any ingredients that are in short supply. It then generates a list of ingredients that are in short supply and sends it to the user's device.

[1862] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1863] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1864] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1865] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1866] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1867] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1868] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1869] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1870] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1871] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1872] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1873] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1874] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1875] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1876] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1877] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1878] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1879] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1880] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1881] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1882] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1883] The following is further disclosed regarding the above embodiment.

[1884] (Claim 1)

[1885] means for receiving an image of clothing owned by a user and analyzing the image to extract characteristic information of the clothing;

[1886] means for receiving information about food purchased by a user, analyzing the information, and storing the information in a predetermined database;

[1887] A means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information of the user's clothing;

[1888] a means for proposing a predetermined menu based on predetermined food life data and generating a list of foods that are in short supply based on the menu;

[1889] A system including:

[1890] (Claim 2)

[1891] 10. The system of claim 1, further comprising means for analyzing a given receipt image to extract information about food products purchased by a user.

[1892] (Claim 3)

[1893] 10. The system of claim 1, further comprising means for using a learning algorithm in suggesting predetermined menu items based on a user's past consumption data and current inventory data.

[1894] "Example 1"

[1895] (Claim 1)

[1896] means for receiving an image of clothing owned by a user and analyzing the image to extract characteristic information of the clothing;

[1897] means for receiving information about food purchased by a user, analyzing the information, and storing the information in a predetermined database;

[1898] A means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information of the user's clothing;

[1899] a means for proposing a predetermined menu based on predetermined food life data and generating a list of foods that are in short supply based on the menu;

[1900] A means for receiving a receipt image taken by a user and extracting purchased food information using OCR technology;

[1901] a means for suggesting menu items using a learning algorithm based on the user's past consumption data and current inventory data, and generating a list of ingredients that are in short supply;

[1902] A system including:

[1903] (Claim 2)

[1904] 10. The system of claim 1, further comprising means for extracting information about the food products purchased by the user from a given receipt image using OCR technology.

[1905] (Claim 3)

[1906] 10. The system of claim 1, further comprising means for using a learning algorithm to suggest predetermined menu items based on a user's past consumption data and current inventory data.

[1907] "Application Example 1"

[1908] (Claim 1)

[1909] means for receiving an image of clothing owned by a user and analyzing the image to extract characteristic information of the clothing;

[1910] means for receiving information about food purchased by a user, analyzing the information, and storing the information in a predetermined database;

[1911] A means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information of the user's clothing;

[1912] a means for proposing a predetermined menu based on predetermined food life data and generating a list of foods that are in short supply based on the menu;

[1913] A means for suggesting coordination with products that the user is considering purchasing in the virtual store;

[1914] A means for managing information on purchased items and understanding user inventory;

[1915] A system including:

[1916] (Claim 2)

[1917] 10. The system of claim 1, further comprising means for analyzing a given receipt image to extract information about food products purchased by a user.

[1918] (Claim 3)

[1919] 10. The system of claim 1, further compris...

Claims

1. means for receiving an image of clothing owned by a user and analyzing the image to extract characteristic information of the clothing; means for receiving information about food purchased by a user, analyzing the information, and storing the information in a predetermined database; A means for acquiring temperature data and proposing appropriate clothing coordination based on the temperature data and characteristic information of the user's clothing; a means for proposing a predetermined menu based on predetermined food life data and generating a list of foods that are in short supply based on the menu; A system including:

2. The system of claim 1 , further comprising means for analyzing a given receipt image to extract information about food products purchased by a user.

3. 10. The system of claim 1, further comprising means for using a learning algorithm in suggesting predetermined menu items based on a user's past consumption data and current inventory data.

Images