system

Abstract

A system is provided.SOLUTION: A system comprising: image recognition means disposed in a refrigerator; barcode scanning means or RFID reading means; means for recording ingredient information in the refrigerator in a database; means for receiving health data of a user from a smart device and updating to a user profile; means for matching the profile data of the user with the ingredient information in the refrigerator and generating an appropriate recipe; and means for automatically ordering missing ingredients at an online store.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, many people are seeking to reduce the time they spend on meal planning and shopping at home while still managing their health. However, managing the right ingredients, choosing nutritionally balanced recipes, and quickly replenishing missing ingredients can be challenging. This process can be particularly cumbersome for people with health conditions, specific nutritional requirements, or allergies, making it difficult to manage efficiently and consistently. [Means for solving the problem]

[0005] This invention provides a system that automatically acquires information about ingredients stored in a refrigerator, suggests recipes based on the user's health condition and nutritional requirements, and automatically orders missing ingredients online. Specifically, the system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. Furthermore, the system receives the user's health data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients stored in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[0006] "Inside the refrigerator" refers to the internal space of a cooling device used to store food.

[0007] "Image recognition means" is a system that includes an image capture device such as a camera and an algorithm that analyzes the image and identifies objects.

[0008] A "barcode scanning means" is a device that reads a barcode and converts the information into digital data.

[0009] "RFID reading means" is a device that wirelessly reads information from an RFID tag and acquires that data.

[0010] "Means of recording in a database" refers to an information system for storing and managing acquired data.

[0011] A "smart device" is a portable electronic device with internet connectivity that has the ability to collect and transmit a user's health data.

[0012] A "user profile" is a collection of personal information, including health data and preference information, about an individual user.

[0013] "Means for updating the profile" refers to a mechanism that reflects newly acquired data in the user profile, keeping it up to date at all times.

[0014] The "means for generating recipes" is an algorithm that suggests cooking methods suitable for the user based on the user's health data and information about ingredients in the refrigerator.

[0015] "Online Store" means an e-commerce website that allows the purchase of products via the Internet.

[0016] "Automatic ordering" is a system that automatically processes the purchase of required products at an online store based on specific conditions.

[0017] A "camera" is an optical device for taking still or video images.

[0018] "Nutrient requirements" are the amounts and types of nutrients needed based on an individual's health and lifestyle.

[0019] An "allergy" is an exaggerated immune response to a particular substance and is a factor that influences a user's dietary choices. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that automatically acquires information about ingredients stored in a refrigerator, suggests recipes based on the user's health condition and nutritional requirements, and automatically orders missing ingredients online. The system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients stored in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[0042] A natural language description of the program's operation

[0043] Acquiring ingredient information

[0044] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[0045] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[0046] Health monitoring and profile updates

[0047] The user's health information is sent from the smart device, which serves as the terminal, to the server.

[0048] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. The received data is updated based on the user's profile. The server also analyzes the user's health information and notifies them of any significant changes.

[0049] Recipe Suggestions

[0050] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[0051] The server compares the user's health information (e.g., nutritional requirements and allergies) with the ingredients available in the refrigerator and generates recipes suited to the user's health condition. The generated recipes are prioritized based on the user's preferences and health condition and are notified to the user via their smart device.

[0052] Automatic ordering of missing ingredients

[0053] The server automatically orders missing ingredients online for the suggested recipe.

[0054] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[0055] Specific examples

[0056] Adding ingredients and updating information

[0057] 1. Suppose you add a new carton of milk to the refrigerator.

[0058] 2. The server uses an RFID reader to read the information on the new milk carton (e.g. type, quantity, expiration date).

[0059] 3. The server records the obtained information in a database.

[0060] 4. The device then notifies the user that "New milk has been added."

[0061] Recipe suggestions and auto-ordering

[0062] 1. The server analyzes the user's profile data and selects low-carb dinner recipes that take blood sugar levels into consideration.

[0063] 2. Check the ingredients in the refrigerator and confirm that you are missing tomatoes and lettuce, which are ingredients needed for the recipe.

[0064] 3. The server automatically orders the missing ingredients from the online store and notifies the user that "Some of the ingredients required for the specified recipe have been ordered."

[0065] 4. The user checks the notification on their smart device and approves or changes the order as needed.

[0066] This system allows users to efficiently manage ingredients in their refrigerators and automatically order and suggest recipes based on health information, reducing the hassle of everyday shopping and meal planning and supporting a healthy lifestyle.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[0070] Step 2:

[0071] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[0072] Step 3:

[0073] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[0074] Step 4:

[0075] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[0076] Step 5:

[0077] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[0078] Step 6:

[0079] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies and nutritional requirements, and prioritizes the recipes based on the user's health status and preferences.

[0080] Step 7:

[0081] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[0082] Step 8:

[0083] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[0084] Step 9:

[0085] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[0086] Step 10:

[0087] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In today's busy lifestyles, managing ingredients and planning meals based on health status takes a lot of time and effort. Furthermore, if necessary ingredients are in short supply, users are forced to go shopping every time, wasting time and effort. There is a need for a system that can efficiently solve these issues and simultaneously improve users' health and convenience.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes an image recognition means, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for updating the database and notifying the user when an ingredient is added, and a means for notifying the user's smart device of the generated recipe. This allows the user to efficiently manage ingredients, monitor health information, suggest recipes, and automatically order missing ingredients.

[0093] The "image recognition means" is a device that uses a camera placed inside the refrigerator to take images of ingredients and includes an algorithm that identifies the type and quantity of ingredients.

[0094] A "barcode scanning means" is a device that reads the barcode printed on the food package.

[0095] The "RFID reading means" is a device that acquires information from the RFID tag attached to the food material.

[0096] The "database" is a system for storing and managing collected food ingredient information and user health information.

[0097] A "smart device" is a terminal that has the function of collecting a user's health data and transmitting it to a server via a communication means.

[0098] A "user profile" is a set of information that includes personal data such as a user's health status, nutritional requirements, and allergy information.

[0099] The "recipe generation means" is an algorithm that generates appropriate cooking recipes based on the user's profile data and information about ingredients in the refrigerator.

[0100] An "online store" is an online sales site where you can purchase products via the Internet.

[0101] The "online ordering method" is a system that calls the API of affiliated online stores and automatically orders ingredients that are in short supply.

[0102] "Notification means" is a function for sending notifications to users, such as new information, recipe suggestions, and automatic order confirmations.

[0103] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and nutritional requirements, and automatically orders missing ingredients online. This system integrates various hardware and software to enable efficient ingredient management and healthy meal planning.

[0104] Hardware and software used

[0105] Image recognition means: A camera placed inside the refrigerator is used to take images of the interior and run image recognition algorithms to identify the type and quantity of ingredients.

[0106] Barcode scanning means: A device that reads the barcode printed on food packaging.

[0107] RFID reading means: A device that acquires information from the RFID tag attached to the food ingredient.

[0108] Database: A system that stores and manages collected food ingredient information and user health information.

[0109] Smart device: A device (e.g., smartwatch, fitness tracker) that has the ability to collect user health data and transmit it to a server via a communication means.

[0110] Server: A computer system that matches the user's profile data with the information about ingredients in the refrigerator and generates appropriate recipes.

[0111] Program processing

[0112] Acquiring ingredient information

[0113] The server periodically activates the refrigerator's camera to take pictures of the interior. These pictures are then analyzed using image recognition algorithms to identify the type and quantity of ingredients. Barcode scanners and RFID readers are also used to collect detailed information about the ingredients, which is then recorded in a database.

[0114] Health monitoring and profile updates

[0115] Smart devices send user health data (weight, blood pressure, heart rate, blood sugar level, etc.) to a server, which updates the user profile based on the received data, analyzes it as needed, and notifies users of any significant changes.

[0116] Recipe Suggestions

[0117] The server compares the user's profile data with the ingredients in the refrigerator and generates recipes suited to the user's health and nutritional requirements. The generated recipes are then sent to the user via their smart device.

[0118] Automatic ordering of missing ingredients

[0119] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If missing ingredients are identified, the server calls the API of a partner online store to automatically place an order. The user receives an order confirmation notification via their smart device.

[0120] Specific examples

[0121] 1. Add ingredients and update information

[0122] When a user adds a new carton of milk to the refrigerator, the server uses an RFID reader to read the information about the new carton (type, quantity, expiration date), then records the information in the database and notifies the user that "new milk has been added."

[0123] 2. Recipe suggestions and automatic ordering

[0124] The server analyzes the user's profile data and selects a low-carb dinner recipe for a user who wants to manage their blood sugar levels, for example. After checking the information about ingredients in the refrigerator, it is determined that the tomato and lettuce needed for the recipe are missing. The server automatically orders the missing ingredients from the online store and notifies the user that "some of the ingredients needed for the specified recipe have been ordered." The user checks the notification on their smart device and approves or changes the order as necessary.

[0125] This system allows users to efficiently manage ingredients, suggest recipes based on health information, and automatically order, reducing the hassle of everyday shopping and meal planning while supporting a healthy lifestyle.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1: Photographing ingredients and analyzing the images

[0128] The server periodically takes pictures of the inside of the refrigerator using a camera inside the refrigerator. It takes the image of the inside of the refrigerator as input and sends the image data to an image recognition algorithm. The server then analyzes the image using the image recognition algorithm and identifies the type and quantity of ingredients as output. Specifically, it identifies three apples and one carton of eggs.

[0129] Step 2: Obtain barcode or RFID information

[0130] When the server detects that a newly added ingredient has been added, it uses a barcode scanner or RFID reader to read the ingredient's information. The barcode or RFID tag information is taken as input, and the server analyzes it to extract data such as type, quantity, and expiration date. The output is to record the specific ingredient information in the database. A specific example of its operation is when a user adds a new carton of milk, recording it as "1L milk, expiration date: October 30, 2023."

[0131] Step 3: Receive health information and update your profile

[0132] The terminal (smart device) collects the user's health data (weight, blood pressure, heart rate, blood glucose level, etc.) on a daily basis and sends the data to a server. The server receives the health data as input, analyzes the data, and updates it with the latest information to update the user profile. The latest user profile is updated as an output. Specifically, the smartwatch sends the nighttime sleep data and morning blood pressure measurement.

[0133] Step 4: Recipe generation and notification

[0134] The server compares the user's profile data with the information about ingredients in the refrigerator and generates an appropriate recipe. It takes the profile data and ingredient information as input, and uses a generative AI model to generate an appropriate recipe. The generated recipe is then sent to the smart device as output. Specifically, it generates a low-carb dinner recipe and sends a notification to the smartphone saying, "Would you like to make a chicken and vegetable salad?"

[0135] Step 5: Auto-order missing ingredients

[0136] The server checks the ingredients needed for the proposed recipe and identifies any missing ingredients by comparing them with the inventory information in the refrigerator. The server receives the list of ingredients needed for the recipe and inventory information as input, and automatically orders the missing ingredients via the online store API. The output is an order confirmation notification sent to the user. Specifically, it identifies that two tomatoes and one lettuce are missing, and completes the automatic ordering process at the online store.

[0137] Step 6: Order confirmation and user notification

[0138] The server sends an automated order confirmation to the user's smart device and provides an interface that allows the user to confirm or change the order details. The server takes the order details as input, generates a confirmation notice, and sends it to the user. The output is a notification that is received by the user. Specifically, the user's smartphone displays "You have ordered two tomatoes and one lettuce," allowing the user to confirm the order details.

[0139] (Application example 1)

[0140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0141] In today's busy lifestyles, managing food and planning healthy meals has become increasingly difficult. Furthermore, even when shopping in brick-and-mortar stores, managing ingredient inventory and planning purchases can be a time-consuming process. However, due to the lack of appropriate systems to solve these problems, consumers find it difficult to plan healthy meals efficiently. In addition, there is a demand for personalized recipe suggestions based on health data and the ability to automatically order ingredients when they are in short supply. The present invention aims to solve these problems.

[0142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0143] In this invention, the server includes an image recognition device installed in the refrigerator, a barcode scanning device or an RFID reader, a means for recording information about ingredients in the refrigerator in a database, a means for receiving user health data from a smart device and updating it in a user profile, a means for comparing the user profile data with the information about ingredients in the refrigerator and generating appropriate recipes, a means for automatically ordering missing ingredients from an online store, a means for collecting ingredient information using an image recognition device installed in smart shelves or smart refrigerators in physical stores, and a means for generating recipes suited to the customer's health condition based on the information and automating food ordering. This enables the creation of efficient and healthy meal plans and reduces the hassle of shopping through the automatic ordering function.

[0144] The "image recognition means" is a device that analyzes images taken with a camera and identifies the type and quantity of ingredients.

[0145] The "barcode scanning means" is a device that reads the barcode printed on the food package and obtains the ingredient information.

[0146] The "RFID reading means" is a device that reads the information attached to the RFID tag and acquires the ingredient information.

[0147] The "means for recording in a database" is a system for storing the acquired ingredient information in a database.

[0148] A "smart device" is a portable electronic device for managing health data and user profiles.

[0149] "Means for receiving user health data and updating the user profile" refers to a system in which a server receives the user's health information sent from a smart device and reflects it in the user profile as the latest information.

[0150] "Means for matching user profile data with information on ingredients in the refrigerator and generating appropriate recipes" refers to a system that suggests recipes using ingredients in the refrigerator based on the user's health condition and nutritional requirements.

[0151] The "means for automatically ordering missing ingredients from an online store" is a system that checks the stock of ingredients required for a proposed recipe and automatically orders the missing ingredients from an online store.

[0152] "Image recognition devices installed on smart shelves and smart refrigerators in physical stores" are devices equipped with cameras and analysis systems that automatically grasp the types of ingredients and inventory in physical stores.

[0153] The "means for automating food ordering" is a system that automatically orders missing ingredients, allowing users to obtain the ingredients they need without any hassle.

[0154] This invention is a system that uses smart shelves and smart refrigerators in brick-and-mortar stores to automatically obtain information about ingredients, suggest recipes based on the user's health status and nutritional requirements, and automatically order missing ingredients online. This system is realized using multiple hardware and software components.

[0155] System configuration

[0156] How to obtain food information

[0157] The server uses image recognition, barcode scanning, or RFID reading methods built into smart refrigerators and smart shelves in physical stores. This allows the server to identify the type, quantity, and expiration date of ingredients and record the obtained information in a database. TensorFlow is used for image recognition, and common scanners and readers are used for barcode scanning and RFID reading.

[0158] For example, when organic vegetables are added as new ingredients to a physical store, the server uses an RFID reader to obtain this information and stores it in a database, which also stores the type, quantity, and expiration date of the ingredients.

[0159] Health information monitoring and profile updating methods

[0160] The server receives health data from smart devices such as smartwatches and fitness trackers. Received data includes the user's weight, blood pressure, heart rate, blood sugar level, etc. This received data is automatically updated to the user profile on the server. Python's Pandas and NumPy are used to manage and analyze the data.

[0161] For example, a user's new weight measurement on their smartwatch is sent to the server, and the data is updated in the user profile.

[0162] Recipe suggestion method

[0163] The server generates appropriate recipes based on the user's health information and the ingredients in the refrigerator. This recipe is generated using the machine learning model scikit-learn. The generated recipe is then sent to the user's smart device.

[0164] For example, the server selects a low-carb dinner recipe, compares the ingredients in the refrigerator to generate the recipe, and notifies the user.

[0165] Automatic ordering of ingredients that are in short supply

[0166] The server checks the refrigerator stock of ingredients needed for the proposed recipe and automatically orders any missing ingredients from the online store. The order process calls the API of the affiliated online store. If the order details need to be confirmed or changed, a notification is sent to the user's smart device, allowing them to make changes via the interface.

[0167] If the missing ingredients are identified as tomatoes and lettuce, the server uses the online store's ordering API to order these ingredients and notifies the user that "some of the ingredients required for the specified recipe have been ordered."

[0168] Prompt Sentence Examples

[0169] "Automatically order missing ingredients online based on your desired recipe."

[0170] As described above, the present invention provides a system that allows for efficient and healthy meal planning and significantly reduces the effort required for shopping.

[0171] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0172] Step 1: Obtaining ingredient information

[0173] The server acquires ingredient information using cameras, barcode scanners, or RFID readers built into smart shelves and smart refrigerators in physical stores. Specifically, the camera takes an image of the ingredient, and an image recognition algorithm (e.g., TensorFlow) is used to identify the type and quantity of the ingredient. The acquired information is recorded in a database. The input is an image of the ingredient or barcode / RFID label data, and the output is ingredient information including the ingredient name, quantity, and expiration date.

[0174] Step 2: Monitor your health information and update your profile

[0175] The server receives health data sent from the user's smart device (smartwatch or fitness tracker). The received data includes weight, blood pressure, heart rate, blood glucose level, etc. The server analyzes the data and updates the user profile. The input is the health data sent from the smart device, and the output is the updated user profile.

[0176] Step 3: Recipe suggestions

[0177] The server generates recipes based on the user's health information and information about ingredients in the refrigerator. A machine learning model (scikit-learn) is used for generation. The input is the user's health data and information about ingredients in the refrigerator, and the output is a recipe appropriate for the user's health condition. The generated recipe is then sent to the user's smart device.

[0178] Step 4: Identify missing ingredients

[0179] The server creates a list of ingredients needed based on the generated recipe and compares it with the refrigerator's inventory information. It identifies missing ingredients and compiles that information. The input is the list of ingredients needed for the recipe and the refrigerator's inventory information, and the output is the list of missing ingredients.

[0180] Step 5: Automated ordering of ingredients

[0181] The server automatically orders the identified missing ingredients from the online store. The ordering is done using the API of the partner online store. The input is the list of missing ingredients, and the output is order confirmation information. The user is notified of the automatic order via their smart device.

[0182] Step 6: Order confirmation and receipt notification

[0183] Users can confirm and change their order details through their smart device. After placing an order, they receive a notification about where to pick up the ingredients in the store. The input is order confirmation information, and the output is a notification to the user and instructions on where to pick up the ingredients.

[0184] Through the above processing flow, the present invention can support efficient and healthy meal planning in physical stores and provide an automatic ordering function for missing ingredients.

[0185] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0186] This invention is a system that automatically acquires information about ingredients in a refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health and emotional data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[0187] A natural language description of the program's operation

[0188] Acquiring ingredient information

[0189] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[0190] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[0191] Health and emotional monitoring and profile updates

[0192] The user's health and emotional information is sent from the smart device to the server.

[0193] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. It then uses an emotion engine to analyze the user's voice and facial expressions to generate emotion data. This data is updated based on the user profile. The server also analyzes the user's health and emotion information and notifies the user of any significant changes.

[0194] Recipe Suggestions

[0195] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[0196] The server compares the user's health information and emotional state (e.g., stress level and mood) with the ingredients available in the refrigerator and generates recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences, health, and emotional state, and are notified to the user via their smart device.

[0197] Automatic ordering of missing ingredients

[0198] The server automatically orders missing ingredients online for the suggested recipe.

[0199] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[0200] Specific examples

[0201] Recipe suggestions based on emotional data

[0202] 1. Add ingredients and update information

[0203] When a new milk carton is added to the refrigerator, the server uses an RFID reader to read the information about the new milk carton (e.g., type, quantity, expiration date).

[0204] The server records the obtained information in a database.

[0205] The device then notifies the user that "New milk has been added."

[0206] 2. Collecting emotional data using an emotion engine

[0207] When a user uses a smart device, the emotion engine analyzes facial expressions and voice to identify the user's emotional state (e.g., stress level and mood).

[0208] The emotion data is sent to the server and updated in the user profile.

[0209] 3. Recipe suggestions based on emotional and health data

[0210] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective in reducing stress) based on the user's current health and emotional state.

[0211] The server checks the information about ingredients in the refrigerator and identifies the ingredients needed for the recipe.

[0212] Additionally, if necessary, any missing ingredients will be automatically ordered from the online store.

[0213] 4. Check recipe notifications and auto-orders

[0214] The generated recipe is notified to the user via their device, and a message is displayed saying, "A salad recipe that is effective in reducing stress has been suggested."

[0215] The user selects a salad recipe and the server automatically orders any missing ingredients.

[0216] The terminal will send the user an automated order confirmation and allow the user to review and modify the order.

[0217] This system allows users to efficiently manage ingredients in their refrigerator, and offers recipe suggestions and automatic ordering based on health and emotional information, thereby reducing the hassle of everyday shopping and meal planning and supporting a lifestyle that is healthy and meets emotional needs.

[0218] The processing flow will be explained below.

[0219] Step 1:

[0220] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[0221] Step 2:

[0222] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[0223] Step 3:

[0224] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[0225] Step 4:

[0226] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[0227] Step 5:

[0228] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[0229] Step 6:

[0230] The terminal (smart device) uses an emotion engine to analyze the user's facial expressions and voice to recognize emotions. The recognized emotion data is sent to the server and updated in the user profile.

[0231] Step 7:

[0232] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies, nutritional requirements, and emotional state. The generated recipes are prioritized based on the user's preferences, health status, and emotional state.

[0233] Step 8:

[0234] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[0235] Step 9:

[0236] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[0237] Step 10:

[0238] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[0239] Step 11:

[0240] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[0241] Specific examples

[0242] Recipe suggestions based on emotional data

[0243] Step 1:

[0244] The server uses the refrigerator's camera to take pictures of the inside of the refrigerator.

[0245] The captured image is analyzed to identify information about ingredients such as apples and milk.

[0246] The analysis results are recorded in a database.

[0247] Step 2:

[0248] The server scans the barcode of the milk using a barcode scanner in the refrigerator.

[0249] Detailed information about the milk (type, quantity, expiration date) is recorded in a database from the barcode information.

[0250] Step 3:

[0251] When a user adds a new yogurt to the refrigerator, the barcode is read using the barcode scanner function of the terminal (smart device).

[0252] The information read is sent to a server, and the yogurt information is recorded in a database.

[0253] Step 4:

[0254] The terminal (smart device) periodically measures the user's weight, blood pressure, heart rate, blood sugar level, etc. and sends the data to the server.

[0255] The server updates the user profile with the received data.

[0256] Step 5:

[0257] The server analyzes health data and detects high blood pressure.

[0258] Based on this information, recipes are generated that take into account appropriate nutritional balance.

[0259] Step 6:

[0260] The emotion engine in the device (smart device) analyzes the user's facial expressions and voice and recognizes that the user is currently feeling stressed.

[0261] The emotion data is sent to the server and updated in the user profile.

[0262] Step 7:

[0263] The server compares the user's health and emotional data and suggests recipes that are effective in reducing stress.

[0264] The generated recipe is notified to the user as an "herbal tea recipe effective for stress relief."

[0265] Step 8:

[0266] The user sees the notification and selects the suggested recipe.

[0267] The server double-checks the ingredients needed for the recipe and compares them with the stock in the refrigerator.

[0268] Identify missing herbal tea ingredients.

[0269] Step 9:

[0270] The server calls the API of a partner online store and automatically orders the missing herbs for herbal tea.

[0271] The order process is completed and the details are recorded in a database.

[0272] Step 10:

[0273] The terminal (smart device) sends the user a confirmation of the automatic order.

[0274] The user can review the notification and approve or change the order.

[0275] In this way, a system that combines an emotion engine can suggest individual recipes based on the user's health condition and emotions, providing more effective support for daily stress management and health management.

[0276] Example 2

[0277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0278] In today's lifestyle, managing the ingredients in the refrigerator and creating meal plans based on the user's health and emotional information is extremely time-consuming. Furthermore, checking and purchasing ingredients that are lacking also increases the time required. A system that solves these problems and supports an efficient and healthy lifestyle is needed.

[0279] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a photographing means, an identification means, a means for recording information about ingredients in the refrigerator, a means for receiving the user's health data from the smart device and updating it in the user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating recipes, a means for ordering missing ingredients online, a means for collecting the user's emotion data and updating the profile, a means for notifying the user of significant changes in the health data and emotion data, and a means for sending a confirmation notice of the automatic order. This automates a series of processes, from ingredient management to health information monitoring, recipe suggestions, and automatic ordering of missing ingredients, enabling the user to maintain an efficient and healthy lifestyle.

[0280] The "photography means" is a tool that is installed inside the refrigerator and is used to take pictures of ingredients.

[0281] "Identification means" refers to a method for analyzing the captured image and read information to identify the type and quantity of ingredients.

[0282] The "recording means" is a method for storing the collected information on ingredients in a database.

[0283] A "smart device" is a terminal that collects a user's health information and emotional data and transmits it to a server.

[0284] A "user profile" is a database entry that records and updates a user's health and emotional state.

[0285] The "matching means" is a method for comparing user profile data with information about ingredients in the refrigerator and generating an appropriate recipe based on that information.

[0286] A "generator" is a method for creating a recipe based on the collated data.

[0287] An "online ordering method" is a method for automatically purchasing missing ingredients required for a suggested recipe.

[0288] "Emotional data" is data on the emotional state of a user obtained by analyzing their voice and facial expressions.

[0289] "Means for notifying significant changes" refers to a method for notifying a user when there is a significant change in the user's health data or emotional data.

[0290] "Confirmation notification means" refers to a method for informing users of the contents of an online order after it has been completed and prompting them to confirm or change the order.

[0291] This invention relates to a system that automatically acquires information about ingredients in a refrigerator based on the user's health and emotional state, suggests appropriate recipes, and orders missing ingredients online.

[0292] Specifically, this system is implemented using the following hardware and software.

[0293] Hardware Configuration

[0294] 1. Photographing means inside the refrigerator: A camera placed inside the refrigerator is a device that takes pictures of ingredients.

[0295] 2. Identification means: A device that uses a barcode scanner or RFID reader to identify the type and quantity of ingredients.

[0296] 3. Smart devices: These are devices such as smartwatches and fitness trackers used by users that collect health and emotional data and send it to a server.

[0297] Software Configuration

[0298] 1. Image recognition algorithm: Software that runs on the server and analyzes the captured image to identify the type and quantity of ingredients.

[0299] 2. Database management system: A system for storing and managing collected food ingredient information, health data, and emotional data.

[0300] 3. Recipe generation engine: Software that generates appropriate recipes based on the user's health and emotional state.

[0301] 4. Order management system: A system that identifies shortages of ingredients and automatically places orders by calling the API of affiliated online stores.

[0302] 5. Notification Engine: This software notifies users with recipe suggestions and automatic ordering confirmations.

[0303] Example of operation

[0304] Example 1: Collecting and recording information about ingredients in the refrigerator

[0305] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[0306] The server uses an image recognition algorithm to analyze the captured image and identify the type and quantity of ingredients.

[0307] For newly added ingredients, the server uses a barcode scanner or RFID reader in the refrigerator to read the identifying information.

[0308] The server records the obtained information in a database, saving the names, quantities, expiration dates, etc. of ingredients.

[0309] Example 2: Monitoring health and emotional information and updating profiles

[0310] Users wear smart devices and collect health data (e.g., weight, blood pressure, heart rate, blood glucose levels) on a daily basis.

[0311] Smart devices periodically send the collected data to a server.

[0312] The server uses an emotion engine to analyze the user's voice and facial expressions and generate emotion data.

[0313] The collected health and emotional data is updated by the server into a user profile.

[0314] For example, if a user's stress level increases, the server uses a notification engine to notify the user of important fluctuations.

[0315] Example 3: Recipe suggestions

[0316] The server matches the user profile data with the ingredients in the refrigerator and generates appropriate recipes based on the user's health and emotional state.

[0317] For example, if a user's stress level is high, the app will generate recipes using ingredients that are effective in reducing stress.

[0318] The server notifies the smart device of the generated recipe so that the user can check it.

[0319] Example 4: Automatic ordering and notification of missing ingredients

[0320] The server checks the ingredients needed for the generated recipe against the inventory information in the refrigerator.

[0321] When a missing ingredient is identified, the server calls the API of a partner online store to automatically place the order.

[0322] After the order is completed, the server uses a notification engine to send a confirmation to the user, who can then review the order details through their smart device and make any necessary changes.

[0323] This system allows users to efficiently carry out all processes, from managing ingredients in the refrigerator to monitoring health information, collecting emotional data, suggesting recipes, and automatically ordering ingredients when they are running low.

[0324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0325] Step 1: Collect images of the inside of the refrigerator

[0326] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[0327] Input: An image of the inside of a refrigerator taken by a camera.

[0328] Output: Captured image data.

[0329] Specific operation: The server activates the camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added.

[0330] Step 2: Analyze with image recognition algorithm

[0331] The server uses image recognition algorithms to analyze the images sent to it.

[0332] Input: Captured image data.

[0333] Output: Data on the type and quantity of ingredients analyzed.

[0334] What it does: The server analyzes the received image and runs an image recognition algorithm to identify the type and quantity of ingredients.

[0335] Step 3: Barcode scanning or RFID reading

[0336] The server activates the barcode scanner and RFID reader inside the refrigerator.

[0337] Input: Barcode or RFID tag information of the newly added ingredient.

[0338] Output: Detailed information about the ingredients read (e.g. name, quantity, expiration date).

[0339] Specific operation: The server activates a barcode scanner or RFID reader to read the information about the new ingredients.

[0340] Step 4: Record in the database

[0341] The server records the analysis results and the barcode / RFID tag information read in a database.

[0342] Input: Parsed ingredient information, read ingredient information.

[0343] Output: Detailed information such as ingredient name, quantity, expiration date, etc. recorded in the database.

[0344] Specific operation: The server stores the obtained ingredient information in a database and manages the ingredients.

[0345] Step 5: Collecting health data

[0346] The smart device that serves as the terminal collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.).

[0347] Input: User's everyday health data.

[0348] Output: Collected health data.

[0349] How it works: Smart devices such as smartwatches and fitness trackers continuously collect biometric data from their users.

[0350] Step 6: Collect emotion data

[0351] The smart device analyzes the user's voice and facial expressions.

[0352] Input: User voice and facial expression data.

[0353] Output: Parsed emotion data.

[0354] What it does: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state.

[0355] Step 7: Update your user profile

[0356] The server updates the user profile based on the health and emotional data.

[0357] Input: Collected health and emotion data.

[0358] Output: Updated user profile.

[0359] Specific operation: The server updates the profile with newly acquired data, maintaining the latest health and emotional state.

[0360] Step 8: Notification of significant changes

[0361] The server analyzes the user's health and emotional information and notifies them if there are any significant changes.

[0362] Input: Continuously updated health and emotion data.

[0363] Output: Notification message of significant fluctuations.

[0364] Specific operation: The server will send an alert to the user if there is a fluctuation that exceeds a certain standard.

[0365] Step 9: Matching profile data with ingredient information

[0366] The server matches the user profile with the information about ingredients in the refrigerator.

[0367] Input: User profile data, refrigerator food information.

[0368] Output: Matching results.

[0369] Specific operation: The server compares the profile data with the ingredient information and obtains the data necessary to suggest appropriate recipes.

[0370] Step 10: Recipe Generation

[0371] The server generates an appropriate recipe based on the matching results.

[0372] Input: Matching result.

[0373] Output: The generated recipe.

[0374] How it works: The server uses a generative AI model to create a recipe tailored to the user.

[0375] Step 11: Prioritize and notify

[0376] The server prioritizes the generated recipes based on the user's preferences and health status and notifies them.

[0377] Input: The generated recipe.

[0378] Output: Prioritized recipe notifications.

[0379] Specific operation: The server evaluates the importance of the recipe and notifies the user via the smart device.

[0380] Step 12: Identify missing ingredients

[0381] The server checks the ingredients needed for the proposed recipe against the inventory information in the refrigerator.

[0382] Inputs: Suggested recipes, refrigerator inventory.

[0383] Output: A list of missing ingredients.

[0384] What happens: The server compares the ingredients required for the recipe with the current inventory and identifies any missing ingredients.

[0385] Step 13: Order online

[0386] The server will then process the missing ingredients for ordering online.

[0387] Input: Missing ingredient list.

[0388] Output: Order information from the online store.

[0389] Specific operation: The server calls the API of the affiliated online store and automatically orders the necessary ingredients.

[0390] Step 14: Order confirmation

[0391] The server will send a confirmation to the user after the order is completed.

[0392] Input: Order checkout completion information.

[0393] Output: Order confirmation notification.

[0394] Specific operation: Once the order is completed, the server notifies the user of the order details via the smart device, allowing them to confirm or change the order.

[0395] (Application example 2)

[0396] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] Conventional refrigerator management systems only managed food inventory and suggested recipes, but were unable to provide suggestions that took into account the user's health or emotional state. They also lacked the ability to automatically order ingredients when they were running low or display recipes in real time, making them inconvenient for users. Furthermore, the recipe options were limited, leaving a need for a more flexible system that could adapt to changes in the user's health and emotions.

[0398] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means arranged in the refrigerator, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for receiving audio or video from the smart device and determining the user's emotional state, and a means for displaying the recipe on the smart glasses. This enables the system to propose appropriate recipes taking into account the user's health and emotional state, automatically order missing ingredients, and display the recipe on the smart glasses in real time.

[0399] The "image recognition means" is a means for analyzing an image captured using a camera and identifying an object contained in the image.

[0400] "Barcode scanning means" refers to means for reading a barcode and obtaining the information written thereon.

[0401] "RFID reading means" refers to a means for reading data from an RFID tag using wireless communication.

[0402] The "means for recording in a database" refers to a means for recording in a database for managing and storing the acquired data.

[0403] The "means for updating the user profile" refers to a means for updating the user profile with the latest information based on the health data and emotional data received from the user.

[0404] The "means for generating an appropriate recipe" is a means for creating an optimal recipe based on the user's profile data and information about ingredients in the refrigerator.

[0405] "Means for automatic ordering from online store" refers to a means for automatically ordering missing ingredients from an online store.

[0406] The "means for displaying a recipe on smart glasses" refers to a means for displaying the generated recipe on the display of smart glasses.

[0407] "Means for receiving audio or video and determining the emotional state of a user" refers to means for analyzing audio or video from a smart device and identifying and determining the emotional state of a user.

[0408] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system works in conjunction with smart glasses and smart devices to support efficient and healthy meal planning.

[0409] System configuration

[0410] Hardware Configuration

[0411] 1. Refrigerator

[0412] It is equipped with a built-in camera, barcode scanning means, and RFID reading means, which are used to collect information about ingredients in the refrigerator.

[0413] 2. Smart Glasses

[0414] Equipped with a display, camera, and microphone, it monitors the user's health and emotional state while displaying generated recipes.

[0415] 3. Smart Devices

[0416] Devices such as smartwatches that collect health data such as weight, blood pressure, heart rate, and blood sugar levels.

[0417] Software Configuration

[0418] 1. Image recognition algorithms (e.g., OpenCV)

[0419] The system analyzes images taken by a camera inside the refrigerator to determine the type, quantity, and expiration date of ingredients.

[0420] 2. Sentiment analysis engine (e.g., Emotion API)

[0421] It analyzes data from the smart glasses' camera or microphone to identify the user's emotional state.

[0422] 3. Data Management System

[0423] A database that manages ingredient information and user profile data.

[0424] 4. Recipe Generation Algorithm

[0425] An algorithm that generates appropriate recipes based on the user's health and emotional data.

[0426] 5. Online Store API (e.g., API for online retailers)

[0427] An API for automatically ordering missing ingredients.

[0428] System Operation

[0429] Acquiring ingredient information

[0430] The server uses the refrigerator's built-in camera, barcode scanner, and RFID reader to take pictures of the refrigerator's interior and collect information about ingredients. It uses image recognition algorithms to identify the type and quantity of ingredients and records this information in a database. The information is updated periodically or when new ingredients are added.

[0431] Health and emotional monitoring and profile updates

[0432] The server receives user health and emotional data from smart glasses and smart devices. Data such as weight, blood pressure, heart rate, and blood sugar levels are collected from smartwatches and other devices, and an emotional analysis engine is used to analyze voice and facial expressions to generate emotional data. This information is updated based on the user profile.

[0433] Recipe Suggestions

[0434] The server generates optimal recipes based on the user profile data and the ingredients in the refrigerator. The generated recipes are prioritized and notified to the user through the smart glasses display. The user can then view the recipes on the smart glasses.

[0435] Automatic ordering of missing ingredients

[0436] The server checks the ingredients required for the proposed recipe and automatically orders any missing ingredients from the online store. The order is processed through the online store API, and a confirmation of the automatic order is sent to the user via the smart glasses. The user can then confirm and change the order details.

[0437] Specific examples

[0438] Recipe suggestions based on emotional data

[0439] 1. Add ingredients and update information

[0440] When a user adds a new ingredient (e.g., a carton of milk) to the refrigerator, the built-in RFID reader reads the new ingredient information, and the server records it in the database. The terminal notifies the user that "a new ingredient has been added."

[0441] 2. Collecting emotional data using an emotion engine

[0442] When a user wears the smart glasses, the emotion engine analyzes their facial expressions and voice to identify their emotional state, and the emotion data is sent to the server and updated in the user profile.

[0443] 3. Recipe suggestions based on emotional and health data

[0444] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective for stress reduction) based on the user's health and emotional state. If necessary, missing ingredients are automatically ordered from the online store.

[0445] 4. Check recipe notifications and auto-orders

[0446] The generated recipe is sent to the user through the smart glasses. The user selects the recipe and the server automatically orders any missing ingredients. The order details are displayed on the smart glasses display, and the user can check and change them.

[0447] Prompt Sentence Examples

[0448] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[0449] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0450] Step 1:

[0451] The server uses a camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added. This obtains image data of the inside of the refrigerator. The server then uses an image recognition algorithm (e.g., OpenCV) to identify the type, quantity, and expiration date of each ingredient in the image. This information is recorded in a database. The input is the image data of the inside of the refrigerator, and the output is ingredient information (type, quantity, expiration date).

[0452] Step 2:

[0453] The server receives the user's health data from the smart device (e.g., smart watch). This data includes weight, blood pressure, heart rate, blood glucose level, etc. The server updates this data to the user profile. It also collects emotion data from the user's voice and video using the camera and microphone of the smart glasses, and analyzes it using an emotion analysis engine (e.g., Emotion API). The input is the audio and video data from the smart device and smart glasses, and the output is the updated health data and emotion data.

[0454] Step 3:

[0455] The server compares the collected health and emotional data with the information about ingredients in the refrigerator. It then runs a recipe generation algorithm to generate optimal recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences and current health and emotional state. The inputs are the health data, emotional data, and ingredient information, and the output is the generated recipe.

[0456] Step 4:

[0457] The device displays the generated recipe on the smart glasses display, allowing the user to check and select the recipe on the smart glasses. The input is the generated recipe, and the output is the recipe display on the smart glasses.

[0458] Step 5:

[0459] The server checks whether the refrigerator contains all the necessary ingredients based on the selected recipe. If any ingredients are missing, it identifies them and calls the online store API to automatically place an order. The order details are displayed on the smart glasses and can be confirmed and changed by the user. The input is the information about ingredients in the refrigerator and the selected recipe, and the output is the ordered ingredients and their confirmation information.

[0460] Specific examples of processing

[0461] In step 1, when new milk is added to the refrigerator, an image of the inside of the refrigerator is taken with a camera, and the type, quantity, and expiration date of the milk are identified using OpenCV, and this is recorded in the database.

[0462] In step 2, the server receives weight and heart rate data from the smartwatch, and sends voice data from the smartglasses to an emotion analysis engine to determine the user's emotional state and update the profile.

[0463] In step 3, a recipe generation algorithm uses this data to generate and prioritize "stress-reducing salads."

[0464] In step 4, the recipe is displayed on the smart glasses display for the user to review.

[0465] In step 5, if it is determined that the refrigerator is low on necessary vegetables, an automatic order is placed using the online store API, and the order details are displayed on the smart glass.

[0466] Prompt Sentence Examples

[0467] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[0468] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0469] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0470] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0471] [Second embodiment]

[0472] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0473] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0474] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0475] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0476] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0477] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0478] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0479] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0480] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0481] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0482] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0483] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0484] This invention is a system that automatically acquires information about ingredients stored in a refrigerator, suggests recipes based on the user's health condition and nutritional requirements, and automatically orders missing ingredients online. The system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients stored in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[0485] A natural language description of the program's operation

[0486] Acquiring ingredient information

[0487] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[0488] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[0489] Health monitoring and profile updates

[0490] The user's health information is sent from the smart device, which serves as the terminal, to the server.

[0491] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. The received data is updated based on the user's profile. The server also analyzes the user's health information and notifies them of any significant changes.

[0492] Recipe Suggestions

[0493] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[0494] The server compares the user's health information (e.g., nutritional requirements and allergies) with the ingredients available in the refrigerator and generates recipes suited to the user's health condition. The generated recipes are prioritized based on the user's preferences and health condition and are notified to the user via their smart device.

[0495] Automatic ordering of missing ingredients

[0496] The server automatically orders missing ingredients online for the suggested recipe.

[0497] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[0498] Specific examples

[0499] Adding ingredients and updating information

[0500] 1. Suppose you add a new carton of milk to the refrigerator.

[0501] 2. The server uses an RFID reader to read the information on the new milk carton (e.g. type, quantity, expiration date).

[0502] 3. The server records the obtained information in a database.

[0503] 4. The device then notifies the user that "New milk has been added."

[0504] Recipe suggestions and auto-ordering

[0505] 1. The server analyzes the user's profile data and selects low-carb dinner recipes that take blood sugar levels into consideration.

[0506] 2. Check the ingredients in the refrigerator and confirm that you are missing tomatoes and lettuce, which are ingredients needed for the recipe.

[0507] 3. The server automatically orders the missing ingredients from the online store and notifies the user that "Some of the ingredients required for the specified recipe have been ordered."

[0508] 4. The user checks the notification on their smart device and approves or changes the order as needed.

[0509] This system allows users to efficiently manage ingredients in their refrigerators and automatically order and suggest recipes based on health information, reducing the hassle of everyday shopping and meal planning and supporting a healthy lifestyle.

[0510] The processing flow will be explained below.

[0511] Step 1:

[0512] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[0513] Step 2:

[0514] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[0515] Step 3:

[0516] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[0517] Step 4:

[0518] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[0519] Step 5:

[0520] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[0521] Step 6:

[0522] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies and nutritional requirements, and prioritizes the recipes based on the user's health status and preferences.

[0523] Step 7:

[0524] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[0525] Step 8:

[0526] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[0527] Step 9:

[0528] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[0529] Step 10:

[0530] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[0531] Example 1

[0532] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] In today's busy lifestyles, managing ingredients and planning meals based on health status takes a lot of time and effort. Furthermore, if necessary ingredients are in short supply, users are forced to go shopping every time, wasting time and effort. There is a need for a system that can efficiently solve these issues and simultaneously improve users' health and convenience.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0535] In this invention, the server includes an image recognition means, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for updating the database and notifying the user when an ingredient is added, and a means for notifying the user's smart device of the generated recipe. This allows the user to efficiently manage ingredients, monitor health information, suggest recipes, and automatically order missing ingredients.

[0536] The "image recognition means" is a device that uses a camera placed inside the refrigerator to take images of ingredients and includes an algorithm that identifies the type and quantity of ingredients.

[0537] A "barcode scanning means" is a device that reads the barcode printed on the food package.

[0538] The "RFID reading means" is a device that acquires information from the RFID tag attached to the food material.

[0539] The "database" is a system for storing and managing collected food ingredient information and user health information.

[0540] A "smart device" is a terminal that has the function of collecting a user's health data and transmitting it to a server via a communication means.

[0541] A "user profile" is a set of information that includes personal data such as a user's health status, nutritional requirements, and allergy information.

[0542] The "recipe generation means" is an algorithm that generates appropriate cooking recipes based on the user's profile data and information about ingredients in the refrigerator.

[0543] An "online store" is an online sales site where you can purchase products via the Internet.

[0544] The "online ordering method" is a system that calls the API of affiliated online stores and automatically orders ingredients that are in short supply.

[0545] "Notification means" is a function for sending notifications to users, such as new information, recipe suggestions, and automatic order confirmations.

[0546] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and nutritional requirements, and automatically orders missing ingredients online. This system integrates various hardware and software to enable efficient ingredient management and healthy meal planning.

[0547] Hardware and software used

[0548] Image recognition means: A camera placed inside the refrigerator is used to take images of the interior and run image recognition algorithms to identify the type and quantity of ingredients.

[0549] Barcode scanning means: A device that reads the barcode printed on food packaging.

[0550] RFID reading means: A device that acquires information from the RFID tag attached to the food ingredient.

[0551] Database: A system that stores and manages collected food ingredient information and user health information.

[0552] Smart device: A device (e.g., smartwatch, fitness tracker) that has the ability to collect user health data and transmit it to a server via a communication means.

[0553] Server: A computer system that matches the user's profile data with the information about ingredients in the refrigerator and generates appropriate recipes.

[0554] Program processing

[0555] Acquiring ingredient information

[0556] The server periodically activates the refrigerator's camera to take pictures of the interior. These pictures are then analyzed using image recognition algorithms to identify the type and quantity of ingredients. Barcode scanners and RFID readers are also used to collect detailed information about the ingredients, which is then recorded in a database.

[0557] Health monitoring and profile updates

[0558] Smart devices send user health data (weight, blood pressure, heart rate, blood sugar level, etc.) to a server, which updates the user profile based on the received data, analyzes it as needed, and notifies users of any significant changes.

[0559] Recipe Suggestions

[0560] The server compares the user's profile data with the ingredients in the refrigerator and generates recipes suited to the user's health and nutritional requirements. The generated recipes are then sent to the user via their smart device.

[0561] Automatic ordering of missing ingredients

[0562] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If missing ingredients are identified, the server calls the API of a partner online store to automatically place an order. The user receives an order confirmation notification via their smart device.

[0563] Specific examples

[0564] 1. Add ingredients and update information

[0565] When a user adds a new carton of milk to the refrigerator, the server uses an RFID reader to read the information about the new carton (type, quantity, expiration date), then records the information in the database and notifies the user that "new milk has been added."

[0566] 2. Recipe suggestions and automatic ordering

[0567] The server analyzes the user's profile data and selects a low-carb dinner recipe for a user who wants to manage their blood sugar levels, for example. After checking the information about ingredients in the refrigerator, it is determined that the tomato and lettuce needed for the recipe are missing. The server automatically orders the missing ingredients from the online store and notifies the user that "some of the ingredients needed for the specified recipe have been ordered." The user checks the notification on their smart device and approves or changes the order as necessary.

[0568] This system allows users to efficiently manage ingredients, suggest recipes based on health information, and automatically order, reducing the hassle of everyday shopping and meal planning while supporting a healthy lifestyle.

[0569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0570] Step 1: Photographing ingredients and analyzing the images

[0571] The server periodically takes pictures of the inside of the refrigerator using a camera inside the refrigerator. It takes the image of the inside of the refrigerator as input and sends the image data to an image recognition algorithm. The server then analyzes the image using the image recognition algorithm and identifies the type and quantity of ingredients as output. Specifically, it identifies three apples and one carton of eggs.

[0572] Step 2: Obtain barcode or RFID information

[0573] When the server detects that a newly added ingredient has been added, it uses a barcode scanner or RFID reader to read the ingredient's information. The barcode or RFID tag information is taken as input, and the server analyzes it to extract data such as type, quantity, and expiration date. The output is to record the specific ingredient information in the database. A specific example of its operation is when a user adds a new carton of milk, recording it as "1L milk, expiration date: October 30, 2023."

[0574] Step 3: Receive health information and update your profile

[0575] The terminal (smart device) collects the user's health data (weight, blood pressure, heart rate, blood glucose level, etc.) on a daily basis and sends the data to a server. The server receives the health data as input, analyzes the data, and updates it with the latest information to update the user profile. The latest user profile is updated as an output. Specifically, the smartwatch sends the nighttime sleep data and morning blood pressure measurement.

[0576] Step 4: Recipe generation and notification

[0577] The server compares the user's profile data with the information about ingredients in the refrigerator and generates an appropriate recipe. It takes the profile data and ingredient information as input, and uses a generative AI model to generate an appropriate recipe. The generated recipe is then sent to the smart device as output. Specifically, it generates a low-carb dinner recipe and sends a notification to the smartphone saying, "Would you like to make a chicken and vegetable salad?"

[0578] Step 5: Auto-order missing ingredients

[0579] The server checks the ingredients needed for the proposed recipe and identifies any missing ingredients by comparing them with the inventory information in the refrigerator. The server receives the list of ingredients needed for the recipe and inventory information as input, and automatically orders the missing ingredients via the online store API. The output is an order confirmation notification sent to the user. Specifically, it identifies that two tomatoes and one lettuce are missing, and completes the automatic ordering process at the online store.

[0580] Step 6: Order confirmation and user notification

[0581] The server sends an automated order confirmation to the user's smart device and provides an interface that allows the user to confirm or change the order details. The server takes the order details as input, generates a confirmation notice, and sends it to the user. The output is a notification that is received by the user. Specifically, the user's smartphone displays "You have ordered two tomatoes and one lettuce," allowing the user to confirm the order details.

[0582] (Application example 1)

[0583] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In today's busy lifestyles, managing food and planning healthy meals has become increasingly difficult. Furthermore, even when shopping in brick-and-mortar stores, managing ingredient inventory and planning purchases can be a time-consuming process. However, due to the lack of appropriate systems to solve these problems, consumers find it difficult to plan healthy meals efficiently. In addition, there is a demand for personalized recipe suggestions based on health data and the ability to automatically order ingredients when they are in short supply. The present invention aims to solve these problems.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0586] In this invention, the server includes an image recognition device installed in the refrigerator, a barcode scanning device or an RFID reader, a means for recording information about ingredients in the refrigerator in a database, a means for receiving user health data from a smart device and updating it in a user profile, a means for comparing the user profile data with the information about ingredients in the refrigerator and generating appropriate recipes, a means for automatically ordering missing ingredients from an online store, a means for collecting ingredient information using an image recognition device installed in smart shelves or smart refrigerators in physical stores, and a means for generating recipes suited to the customer's health condition based on the information and automating food ordering. This enables the creation of efficient and healthy meal plans and reduces the hassle of shopping through the automatic ordering function.

[0587] The "image recognition means" is a device that analyzes images taken with a camera and identifies the type and quantity of ingredients.

[0588] The "barcode scanning means" is a device that reads the barcode printed on the food package and obtains the ingredient information.

[0589] The "RFID reading means" is a device that reads the information attached to the RFID tag and acquires the ingredient information.

[0590] The "means for recording in a database" is a system for storing the acquired ingredient information in a database.

[0591] A "smart device" is a portable electronic device for managing health data and user profiles.

[0592] "Means for receiving user health data and updating the user profile" refers to a system in which a server receives the user's health information sent from a smart device and reflects it in the user profile as the latest information.

[0593] "Means for matching user profile data with information on ingredients in the refrigerator and generating appropriate recipes" refers to a system that suggests recipes using ingredients in the refrigerator based on the user's health condition and nutritional requirements.

[0594] The "means for automatically ordering missing ingredients from an online store" is a system that checks the stock of ingredients required for a proposed recipe and automatically orders the missing ingredients from an online store.

[0595] "Image recognition devices installed on smart shelves and smart refrigerators in physical stores" are devices equipped with cameras and analysis systems that automatically grasp the types of ingredients and inventory in physical stores.

[0596] The "means for automating food ordering" is a system that automatically orders missing ingredients, allowing users to obtain the ingredients they need without any hassle.

[0597] This invention is a system that uses smart shelves and smart refrigerators in brick-and-mortar stores to automatically obtain information about ingredients, suggest recipes based on the user's health status and nutritional requirements, and automatically order missing ingredients online. This system is realized using multiple hardware and software components.

[0598] System configuration

[0599] How to obtain food information

[0600] The server uses image recognition, barcode scanning, or RFID reading methods built into smart refrigerators and smart shelves in physical stores. This allows the server to identify the type, quantity, and expiration date of ingredients and record the obtained information in a database. TensorFlow is used for image recognition, and common scanners and readers are used for barcode scanning and RFID reading.

[0601] For example, when organic vegetables are added as new ingredients to a physical store, the server uses an RFID reader to obtain this information and stores it in a database, which also stores the type, quantity, and expiration date of the ingredients.

[0602] Health information monitoring and profile updating methods

[0603] The server receives health data from smart devices such as smartwatches and fitness trackers. Received data includes the user's weight, blood pressure, heart rate, blood sugar level, etc. This received data is automatically updated to the user profile on the server. Python's Pandas and NumPy are used to manage and analyze the data.

[0604] For example, a user's new weight measurement on their smartwatch is sent to the server, and the data is updated in the user profile.

[0605] Recipe suggestion method

[0606] The server generates appropriate recipes based on the user's health information and the ingredients in the refrigerator. This recipe is generated using the machine learning model scikit-learn. The generated recipe is then sent to the user's smart device.

[0607] For example, the server selects a low-carb dinner recipe, compares the ingredients in the refrigerator to generate the recipe, and notifies the user.

[0608] Automatic ordering of ingredients that are in short supply

[0609] The server checks the refrigerator stock of ingredients needed for the proposed recipe and automatically orders any missing ingredients from the online store. The order process calls the API of the affiliated online store. If the order details need to be confirmed or changed, a notification is sent to the user's smart device, allowing them to make changes via the interface.

[0610] If the missing ingredients are identified as tomatoes and lettuce, the server uses the online store's ordering API to order these ingredients and notifies the user that "some of the ingredients required for the specified recipe have been ordered."

[0611] Prompt Sentence Examples

[0612] "Automatically order missing ingredients online based on your desired recipe."

[0613] As described above, the present invention provides a system that allows for efficient and healthy meal planning and significantly reduces the effort required for shopping.

[0614] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0615] Step 1: Obtaining ingredient information

[0616] The server acquires ingredient information using cameras, barcode scanners, or RFID readers built into smart shelves and smart refrigerators in physical stores. Specifically, the camera takes an image of the ingredient, and an image recognition algorithm (e.g., TensorFlow) is used to identify the type and quantity of the ingredient. The acquired information is recorded in a database. The input is an image of the ingredient or barcode / RFID label data, and the output is ingredient information including the ingredient name, quantity, and expiration date.

[0617] Step 2: Monitor your health information and update your profile

[0618] The server receives health data sent from the user's smart device (smartwatch or fitness tracker). The received data includes weight, blood pressure, heart rate, blood glucose level, etc. The server analyzes the data and updates the user profile. The input is the health data sent from the smart device, and the output is the updated user profile.

[0619] Step 3: Recipe suggestions

[0620] The server generates recipes based on the user's health information and information about ingredients in the refrigerator. A machine learning model (scikit-learn) is used for generation. The input is the user's health data and information about ingredients in the refrigerator, and the output is a recipe appropriate for the user's health condition. The generated recipe is then sent to the user's smart device.

[0621] Step 4: Identify missing ingredients

[0622] The server creates a list of ingredients needed based on the generated recipe and compares it with the refrigerator's inventory information. It identifies missing ingredients and compiles that information. The input is the list of ingredients needed for the recipe and the refrigerator's inventory information, and the output is the list of missing ingredients.

[0623] Step 5: Automated ordering of ingredients

[0624] The server automatically orders the identified missing ingredients from the online store. The ordering is done using the API of the partner online store. The input is the list of missing ingredients, and the output is order confirmation information. The user is notified of the automatic order via their smart device.

[0625] Step 6: Order confirmation and receipt notification

[0626] Users can confirm and change their order details through their smart device. After placing an order, they receive a notification about where to pick up the ingredients in the store. The input is order confirmation information, and the output is a notification to the user and instructions on where to pick up the ingredients.

[0627] Through the above processing flow, the present invention can support efficient and healthy meal planning in physical stores and provide an automatic ordering function for missing ingredients.

[0628] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0629] This invention is a system that automatically acquires information about ingredients in a refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health and emotional data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[0630] A natural language description of the program's operation

[0631] Acquiring ingredient information

[0632] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[0633] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[0634] Health and emotional monitoring and profile updates

[0635] The user's health and emotional information is sent from the smart device to the server.

[0636] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. It then uses an emotion engine to analyze the user's voice and facial expressions to generate emotion data. This data is updated based on the user profile. The server also analyzes the user's health and emotion information and notifies the user of any significant changes.

[0637] Recipe Suggestions

[0638] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[0639] The server compares the user's health information and emotional state (e.g., stress level and mood) with the ingredients available in the refrigerator and generates recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences, health, and emotional state, and are notified to the user via their smart device.

[0640] Automatic ordering of missing ingredients

[0641] The server automatically orders missing ingredients online for the suggested recipe.

[0642] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[0643] Specific examples

[0644] Recipe suggestions based on emotional data

[0645] 1. Add ingredients and update information

[0646] When a new milk carton is added to the refrigerator, the server uses an RFID reader to read the information about the new milk carton (e.g., type, quantity, expiration date).

[0647] The server records the obtained information in a database.

[0648] The device then notifies the user that "New milk has been added."

[0649] 2. Collecting emotional data using an emotion engine

[0650] When a user uses a smart device, the emotion engine analyzes facial expressions and voice to identify the user's emotional state (e.g., stress level and mood).

[0651] The emotion data is sent to the server and updated in the user profile.

[0652] 3. Recipe suggestions based on emotional and health data

[0653] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective in reducing stress) based on the user's current health and emotional state.

[0654] The server checks the information about ingredients in the refrigerator and identifies the ingredients needed for the recipe.

[0655] Additionally, if necessary, any missing ingredients will be automatically ordered from the online store.

[0656] 4. Check recipe notifications and auto-orders

[0657] The generated recipe is notified to the user via their device, and a message is displayed saying, "A salad recipe that is effective in reducing stress has been suggested."

[0658] The user selects a salad recipe and the server automatically orders any missing ingredients.

[0659] The terminal will send the user an automated order confirmation and allow the user to review and modify the order.

[0660] This system allows users to efficiently manage ingredients in their refrigerator, and offers recipe suggestions and automatic ordering based on health and emotional information, thereby reducing the hassle of everyday shopping and meal planning and supporting a lifestyle that is healthy and meets emotional needs.

[0661] The processing flow will be explained below.

[0662] Step 1:

[0663] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[0664] Step 2:

[0665] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[0666] Step 3:

[0667] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[0668] Step 4:

[0669] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[0670] Step 5:

[0671] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[0672] Step 6:

[0673] The terminal (smart device) uses an emotion engine to analyze the user's facial expressions and voice to recognize emotions. The recognized emotion data is sent to the server and updated in the user profile.

[0674] Step 7:

[0675] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies, nutritional requirements, and emotional state. The generated recipes are prioritized based on the user's preferences, health status, and emotional state.

[0676] Step 8:

[0677] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[0678] Step 9:

[0679] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[0680] Step 10:

[0681] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[0682] Step 11:

[0683] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[0684] Specific examples

[0685] Recipe suggestions based on emotional data

[0686] Step 1:

[0687] The server uses the refrigerator's camera to take pictures of the inside of the refrigerator.

[0688] The captured image is analyzed to identify information about ingredients such as apples and milk.

[0689] The analysis results are recorded in a database.

[0690] Step 2:

[0691] The server scans the barcode of the milk using a barcode scanner in the refrigerator.

[0692] Detailed information about the milk (type, quantity, expiration date) is recorded in a database from the barcode information.

[0693] Step 3:

[0694] When a user adds a new yogurt to the refrigerator, the barcode is read using the barcode scanner function of the terminal (smart device).

[0695] The information read is sent to a server, and the yogurt information is recorded in a database.

[0696] Step 4:

[0697] The terminal (smart device) periodically measures the user's weight, blood pressure, heart rate, blood sugar level, etc. and sends the data to the server.

[0698] The server updates the user profile with the received data.

[0699] Step 5:

[0700] The server analyzes health data and detects high blood pressure.

[0701] Based on this information, recipes are generated that take into account appropriate nutritional balance.

[0702] Step 6:

[0703] The emotion engine in the device (smart device) analyzes the user's facial expressions and voice and recognizes that the user is currently feeling stressed.

[0704] The emotion data is sent to the server and updated in the user profile.

[0705] Step 7:

[0706] The server compares the user's health and emotional data and suggests recipes that are effective in reducing stress.

[0707] The generated recipe is notified to the user as an "herbal tea recipe effective for stress relief."

[0708] Step 8:

[0709] The user sees the notification and selects the suggested recipe.

[0710] The server double-checks the ingredients needed for the recipe and compares them with the stock in the refrigerator.

[0711] Identify missing herbal tea ingredients.

[0712] Step 9:

[0713] The server calls the API of a partner online store and automatically orders the missing herbs for herbal tea.

[0714] The order process is completed and the details are recorded in a database.

[0715] Step 10:

[0716] The terminal (smart device) sends the user a confirmation of the automatic order.

[0717] The user can review the notification and approve or change the order.

[0718] In this way, a system that combines an emotion engine can suggest individual recipes based on the user's health condition and emotions, providing more effective support for daily stress management and health management.

[0719] Example 2

[0720] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0721] In today's lifestyle, managing the ingredients in the refrigerator and creating meal plans based on the user's health and emotional information is extremely time-consuming. Furthermore, checking and purchasing ingredients that are lacking also increases the time required. A system that solves these problems and supports an efficient and healthy lifestyle is needed.

[0722] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a photographing means, an identification means, a means for recording information about ingredients in the refrigerator, a means for receiving the user's health data from the smart device and updating it in the user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating recipes, a means for ordering missing ingredients online, a means for collecting the user's emotion data and updating the profile, a means for notifying the user of significant changes in the health data and emotion data, and a means for sending a confirmation notice of the automatic order. This automates a series of processes, from ingredient management to health information monitoring, recipe suggestions, and automatic ordering of missing ingredients, enabling the user to maintain an efficient and healthy lifestyle.

[0723] The "photography means" is a tool that is installed inside the refrigerator and is used to take pictures of ingredients.

[0724] "Identification means" refers to a method for analyzing the captured image and read information to identify the type and quantity of ingredients.

[0725] The "recording means" is a method for storing the collected information on ingredients in a database.

[0726] A "smart device" is a terminal that collects a user's health information and emotional data and transmits it to a server.

[0727] A "user profile" is a database entry that records and updates a user's health and emotional state.

[0728] The "matching means" is a method for comparing user profile data with information about ingredients in the refrigerator and generating an appropriate recipe based on that information.

[0729] A "generator" is a method for creating a recipe based on the collated data.

[0730] An "online ordering method" is a method for automatically purchasing missing ingredients required for a suggested recipe.

[0731] "Emotional data" is data on the emotional state of a user obtained by analyzing their voice and facial expressions.

[0732] "Means for notifying significant changes" refers to a method for notifying a user when there is a significant change in the user's health data or emotional data.

[0733] "Confirmation notification means" refers to a method for informing users of the contents of an online order after it has been completed and prompting them to confirm or change the order.

[0734] This invention relates to a system that automatically acquires information about ingredients in a refrigerator based on the user's health and emotional state, suggests appropriate recipes, and orders missing ingredients online.

[0735] Specifically, this system is implemented using the following hardware and software.

[0736] Hardware Configuration

[0737] 1. Photographing means inside the refrigerator: A camera placed inside the refrigerator is a device that takes pictures of ingredients.

[0738] 2. Identification means: A device that uses a barcode scanner or RFID reader to identify the type and quantity of ingredients.

[0739] 3. Smart devices: These are devices such as smartwatches and fitness trackers used by users that collect health and emotional data and send it to a server.

[0740] Software Configuration

[0741] 1. Image recognition algorithm: Software that runs on the server and analyzes the captured image to identify the type and quantity of ingredients.

[0742] 2. Database management system: A system for storing and managing collected food ingredient information, health data, and emotional data.

[0743] 3. Recipe generation engine: Software that generates appropriate recipes based on the user's health and emotional state.

[0744] 4. Order management system: A system that identifies shortages of ingredients and automatically places orders by calling the API of affiliated online stores.

[0745] 5. Notification Engine: This software notifies users with recipe suggestions and automatic ordering confirmations.

[0746] Example of operation

[0747] Example 1: Collecting and recording information about ingredients in the refrigerator

[0748] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[0749] The server uses an image recognition algorithm to analyze the captured image and identify the type and quantity of ingredients.

[0750] For newly added ingredients, the server uses a barcode scanner or RFID reader in the refrigerator to read the identifying information.

[0751] The server records the obtained information in a database, saving the names, quantities, expiration dates, etc. of ingredients.

[0752] Example 2: Monitoring health and emotional information and updating profiles

[0753] Users wear smart devices and collect health data (e.g., weight, blood pressure, heart rate, blood glucose levels) on a daily basis.

[0754] Smart devices periodically send the collected data to a server.

[0755] The server uses an emotion engine to analyze the user's voice and facial expressions and generate emotion data.

[0756] The collected health and emotional data is updated by the server into a user profile.

[0757] For example, if a user's stress level increases, the server uses a notification engine to notify the user of important fluctuations.

[0758] Example 3: Recipe suggestions

[0759] The server matches the user profile data with the ingredients in the refrigerator and generates appropriate recipes based on the user's health and emotional state.

[0760] For example, if a user's stress level is high, the app will generate recipes using ingredients that are effective in reducing stress.

[0761] The server notifies the smart device of the generated recipe so that the user can check it.

[0762] Example 4: Automatic ordering and notification of missing ingredients

[0763] The server checks the ingredients needed for the generated recipe against the inventory information in the refrigerator.

[0764] When a missing ingredient is identified, the server calls the API of a partner online store to automatically place the order.

[0765] After the order is completed, the server uses a notification engine to send a confirmation to the user, who can then review the order details through their smart device and make any necessary changes.

[0766] This system allows users to efficiently carry out all processes, from managing ingredients in the refrigerator to monitoring health information, collecting emotional data, suggesting recipes, and automatically ordering ingredients when they are running low.

[0767] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0768] Step 1: Collect images of the inside of the refrigerator

[0769] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[0770] Input: An image of the inside of a refrigerator taken by a camera.

[0771] Output: Captured image data.

[0772] Specific operation: The server activates the camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added.

[0773] Step 2: Analyze with image recognition algorithm

[0774] The server uses image recognition algorithms to analyze the images sent to it.

[0775] Input: Captured image data.

[0776] Output: Data on the type and quantity of ingredients analyzed.

[0777] What it does: The server analyzes the received image and runs an image recognition algorithm to identify the type and quantity of ingredients.

[0778] Step 3: Barcode scanning or RFID reading

[0779] The server activates the barcode scanner and RFID reader inside the refrigerator.

[0780] Input: Barcode or RFID tag information of the newly added ingredient.

[0781] Output: Detailed information about the ingredients read (e.g. name, quantity, expiration date).

[0782] Specific operation: The server activates a barcode scanner or RFID reader to read the information about the new ingredients.

[0783] Step 4: Record in the database

[0784] The server records the analysis results and the barcode / RFID tag information read in a database.

[0785] Input: Parsed ingredient information, read ingredient information.

[0786] Output: Detailed information such as ingredient name, quantity, expiration date, etc. recorded in the database.

[0787] Specific operation: The server stores the obtained ingredient information in a database and manages the ingredients.

[0788] Step 5: Collecting health data

[0789] The smart device that serves as the terminal collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.).

[0790] Input: User's everyday health data.

[0791] Output: Collected health data.

[0792] How it works: Smart devices such as smartwatches and fitness trackers continuously collect biometric data from their users.

[0793] Step 6: Collect emotion data

[0794] The smart device analyzes the user's voice and facial expressions.

[0795] Input: User voice and facial expression data.

[0796] Output: Parsed emotion data.

[0797] What it does: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state.

[0798] Step 7: Update your user profile

[0799] The server updates the user profile based on the health and emotional data.

[0800] Input: Collected health and emotion data.

[0801] Output: Updated user profile.

[0802] Specific operation: The server updates the profile with newly acquired data, maintaining the latest health and emotional state.

[0803] Step 8: Notification of significant changes

[0804] The server analyzes the user's health and emotional information and notifies them if there are any significant changes.

[0805] Input: Continuously updated health and emotion data.

[0806] Output: Notification message of significant fluctuations.

[0807] Specific operation: The server will send an alert to the user if there is a fluctuation that exceeds a certain standard.

[0808] Step 9: Matching profile data with ingredient information

[0809] The server matches the user profile with the information about ingredients in the refrigerator.

[0810] Input: User profile data, refrigerator food information.

[0811] Output: Matching results.

[0812] Specific operation: The server compares the profile data with the ingredient information and obtains the data necessary to suggest appropriate recipes.

[0813] Step 10: Recipe Generation

[0814] The server generates an appropriate recipe based on the matching results.

[0815] Input: Matching result.

[0816] Output: The generated recipe.

[0817] How it works: The server uses a generative AI model to create a recipe tailored to the user.

[0818] Step 11: Prioritize and notify

[0819] The server prioritizes the generated recipes based on the user's preferences and health status and notifies them.

[0820] Input: The generated recipe.

[0821] Output: Prioritized recipe notifications.

[0822] Specific operation: The server evaluates the importance of the recipe and notifies the user via the smart device.

[0823] Step 12: Identify missing ingredients

[0824] The server checks the ingredients needed for the proposed recipe against the inventory information in the refrigerator.

[0825] Inputs: Suggested recipes, refrigerator inventory.

[0826] Output: A list of missing ingredients.

[0827] What happens: The server compares the ingredients required for the recipe with the current inventory and identifies any missing ingredients.

[0828] Step 13: Order online

[0829] The server will then process the missing ingredients for ordering online.

[0830] Input: Missing ingredient list.

[0831] Output: Order information from the online store.

[0832] Specific operation: The server calls the API of the affiliated online store and automatically orders the necessary ingredients.

[0833] Step 14: Order confirmation

[0834] The server will send a confirmation to the user after the order is completed.

[0835] Input: Order checkout completion information.

[0836] Output: Order confirmation notification.

[0837] Specific operation: Once the order is completed, the server notifies the user of the order details via the smart device, allowing them to confirm or change the order.

[0838] (Application example 2)

[0839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Conventional refrigerator management systems only managed food inventory and suggested recipes, but were unable to provide suggestions that took into account the user's health or emotional state. They also lacked the ability to automatically order ingredients when they were running low or display recipes in real time, making them inconvenient for users. Furthermore, the recipe options were limited, leaving a need for a more flexible system that could adapt to changes in the user's health and emotions.

[0841] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means arranged in the refrigerator, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for receiving audio or video from the smart device and determining the user's emotional state, and a means for displaying the recipe on the smart glasses. This enables the system to propose appropriate recipes taking into account the user's health and emotional state, automatically order missing ingredients, and display the recipe on the smart glasses in real time.

[0842] The "image recognition means" is a means for analyzing an image captured using a camera and identifying an object contained in the image.

[0843] "Barcode scanning means" refers to means for reading a barcode and obtaining the information written thereon.

[0844] "RFID reading means" refers to a means for reading data from an RFID tag using wireless communication.

[0845] The "means for recording in a database" refers to a means for recording in a database for managing and storing the acquired data.

[0846] The "means for updating the user profile" refers to a means for updating the user profile with the latest information based on the health data and emotional data received from the user.

[0847] The "means for generating an appropriate recipe" is a means for creating an optimal recipe based on the user's profile data and information about ingredients in the refrigerator.

[0848] "Means for automatic ordering from online store" refers to a means for automatically ordering missing ingredients from an online store.

[0849] The "means for displaying a recipe on smart glasses" refers to a means for displaying the generated recipe on the display of smart glasses.

[0850] "Means for receiving audio or video and determining the emotional state of a user" refers to means for analyzing audio or video from a smart device and identifying and determining the emotional state of a user.

[0851] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system works in conjunction with smart glasses and smart devices to support efficient and healthy meal planning.

[0852] System configuration

[0853] Hardware Configuration

[0854] 1. Refrigerator

[0855] It is equipped with a built-in camera, barcode scanning means, and RFID reading means, which are used to collect information about ingredients in the refrigerator.

[0856] 2. Smart Glasses

[0857] Equipped with a display, camera, and microphone, it monitors the user's health and emotional state while displaying generated recipes.

[0858] 3. Smart Devices

[0859] Devices such as smartwatches that collect health data such as weight, blood pressure, heart rate, and blood sugar levels.

[0860] Software Configuration

[0861] 1. Image recognition algorithms (e.g., OpenCV)

[0862] The system analyzes images taken by a camera inside the refrigerator to determine the type, quantity, and expiration date of ingredients.

[0863] 2. Sentiment analysis engine (e.g., Emotion API)

[0864] It analyzes data from the smart glasses' camera or microphone to identify the user's emotional state.

[0865] 3. Data Management System

[0866] A database that manages ingredient information and user profile data.

[0867] 4. Recipe Generation Algorithm

[0868] An algorithm that generates appropriate recipes based on the user's health and emotional data.

[0869] 5. Online Store API (e.g., API for online retailers)

[0870] An API for automatically ordering missing ingredients.

[0871] System Operation

[0872] Acquiring ingredient information

[0873] The server uses the refrigerator's built-in camera, barcode scanner, and RFID reader to take pictures of the refrigerator's interior and collect information about ingredients. It uses image recognition algorithms to identify the type and quantity of ingredients and records this information in a database. The information is updated periodically or when new ingredients are added.

[0874] Health and emotional monitoring and profile updates

[0875] The server receives user health and emotional data from smart glasses and smart devices. Data such as weight, blood pressure, heart rate, and blood sugar levels are collected from smartwatches and other devices, and an emotional analysis engine is used to analyze voice and facial expressions to generate emotional data. This information is updated based on the user profile.

[0876] Recipe Suggestions

[0877] The server generates optimal recipes based on the user profile data and the ingredients in the refrigerator. The generated recipes are prioritized and notified to the user through the smart glasses display. The user can then view the recipes on the smart glasses.

[0878] Automatic ordering of missing ingredients

[0879] The server checks the ingredients required for the proposed recipe and automatically orders any missing ingredients from the online store. The order is processed through the online store API, and a confirmation of the automatic order is sent to the user via the smart glasses. The user can then confirm and change the order details.

[0880] Specific examples

[0881] Recipe suggestions based on emotional data

[0882] 1. Add ingredients and update information

[0883] When a user adds a new ingredient (e.g., a carton of milk) to the refrigerator, the built-in RFID reader reads the new ingredient information, and the server records it in the database. The terminal notifies the user that "a new ingredient has been added."

[0884] 2. Collecting emotional data using an emotion engine

[0885] When a user wears the smart glasses, the emotion engine analyzes their facial expressions and voice to identify their emotional state, and the emotion data is sent to the server and updated in the user profile.

[0886] 3. Recipe suggestions based on emotional and health data

[0887] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective for stress reduction) based on the user's health and emotional state. If necessary, missing ingredients are automatically ordered from the online store.

[0888] 4. Check recipe notifications and auto-orders

[0889] The generated recipe is sent to the user through the smart glasses. The user selects the recipe and the server automatically orders any missing ingredients. The order details are displayed on the smart glasses display, and the user can check and change them.

[0890] Prompt Sentence Examples

[0891] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[0892] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0893] Step 1:

[0894] The server uses a camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added. This obtains image data of the inside of the refrigerator. The server then uses an image recognition algorithm (e.g., OpenCV) to identify the type, quantity, and expiration date of each ingredient in the image. This information is recorded in a database. The input is the image data of the inside of the refrigerator, and the output is ingredient information (type, quantity, expiration date).

[0895] Step 2:

[0896] The server receives the user's health data from the smart device (e.g., smart watch). This data includes weight, blood pressure, heart rate, blood glucose level, etc. The server updates this data to the user profile. It also collects emotion data from the user's voice and video using the camera and microphone of the smart glasses, and analyzes it using an emotion analysis engine (e.g., Emotion API). The input is the audio and video data from the smart device and smart glasses, and the output is the updated health data and emotion data.

[0897] Step 3:

[0898] The server compares the collected health and emotional data with the information about ingredients in the refrigerator. It then runs a recipe generation algorithm to generate optimal recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences and current health and emotional state. The inputs are the health data, emotional data, and ingredient information, and the output is the generated recipe.

[0899] Step 4:

[0900] The device displays the generated recipe on the smart glasses display, allowing the user to check and select the recipe on the smart glasses. The input is the generated recipe, and the output is the recipe display on the smart glasses.

[0901] Step 5:

[0902] The server checks whether the refrigerator contains all the necessary ingredients based on the selected recipe. If any ingredients are missing, it identifies them and calls the online store API to automatically place an order. The order details are displayed on the smart glasses and can be confirmed and changed by the user. The input is the information about ingredients in the refrigerator and the selected recipe, and the output is the ordered ingredients and their confirmation information.

[0903] Specific examples of processing

[0904] In step 1, when new milk is added to the refrigerator, an image of the inside of the refrigerator is taken with a camera, and the type, quantity, and expiration date of the milk are identified using OpenCV, and this is recorded in the database.

[0905] In step 2, the server receives weight and heart rate data from the smartwatch, and sends voice data from the smartglasses to an emotion analysis engine to determine the user's emotional state and update the profile.

[0906] In step 3, a recipe generation algorithm uses this data to generate and prioritize "stress-reducing salads."

[0907] In step 4, the recipe is displayed on the smart glasses display for the user to review.

[0908] In step 5, if it is determined that the refrigerator is low on necessary vegetables, an automatic order is placed using the online store API, and the order details are displayed on the smart glass.

[0909] Prompt Sentence Examples

[0910] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[0911] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0912] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0913] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0914] [Third embodiment]

[0915] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0916] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0917] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0918] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0919] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0920] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0921] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0922] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0923] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0924] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0925] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0926] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0927] This invention is a system that automatically acquires information about ingredients stored in a refrigerator, suggests recipes based on the user's health condition and nutritional requirements, and automatically orders missing ingredients online. The system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients stored in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[0928] A natural language description of the program's operation

[0929] Acquiring ingredient information

[0930] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[0931] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[0932] Health monitoring and profile updates

[0933] The user's health information is sent from the smart device, which serves as the terminal, to the server.

[0934] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. The received data is updated based on the user's profile. The server also analyzes the user's health information and notifies them of any significant changes.

[0935] Recipe Suggestions

[0936] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[0937] The server compares the user's health information (e.g., nutritional requirements and allergies) with the ingredients available in the refrigerator and generates recipes suited to the user's health condition. The generated recipes are prioritized based on the user's preferences and health condition and are notified to the user via their smart device.

[0938] Automatic ordering of missing ingredients

[0939] The server automatically orders missing ingredients online for the suggested recipe.

[0940] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[0941] Specific examples

[0942] Adding ingredients and updating information

[0943] 1. Suppose you add a new carton of milk to the refrigerator.

[0944] 2. The server uses an RFID reader to read the information on the new milk carton (e.g. type, quantity, expiration date).

[0945] 3. The server records the obtained information in a database.

[0946] 4. The device then notifies the user that "New milk has been added."

[0947] Recipe suggestions and auto-ordering

[0948] 1. The server analyzes the user's profile data and selects low-carb dinner recipes that take blood sugar levels into consideration.

[0949] 2. Check the ingredients in the refrigerator and confirm that you are missing tomatoes and lettuce, which are ingredients needed for the recipe.

[0950] 3. The server automatically orders the missing ingredients from the online store and notifies the user that "Some of the ingredients required for the specified recipe have been ordered."

[0951] 4. The user checks the notification on their smart device and approves or changes the order as needed.

[0952] This system allows users to efficiently manage ingredients in their refrigerators and automatically order and suggest recipes based on health information, reducing the hassle of everyday shopping and meal planning and supporting a healthy lifestyle.

[0953] The processing flow will be explained below.

[0954] Step 1:

[0955] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[0956] Step 2:

[0957] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[0958] Step 3:

[0959] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[0960] Step 4:

[0961] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[0962] Step 5:

[0963] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[0964] Step 6:

[0965] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies and nutritional requirements, and prioritizes the recipes based on the user's health status and preferences.

[0966] Step 7:

[0967] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[0968] Step 8:

[0969] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[0970] Step 9:

[0971] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[0972] Step 10:

[0973] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[0974] Example 1

[0975] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0976] In today's busy lifestyles, managing ingredients and planning meals based on health status takes a lot of time and effort. Furthermore, if necessary ingredients are in short supply, users are forced to go shopping every time, wasting time and effort. There is a need for a system that can efficiently solve these issues and simultaneously improve users' health and convenience.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0978] In this invention, the server includes an image recognition means, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for updating the database and notifying the user when an ingredient is added, and a means for notifying the user's smart device of the generated recipe. This allows the user to efficiently manage ingredients, monitor health information, suggest recipes, and automatically order missing ingredients.

[0979] The "image recognition means" is a device that uses a camera placed inside the refrigerator to take images of ingredients and includes an algorithm that identifies the type and quantity of ingredients.

[0980] A "barcode scanning means" is a device that reads the barcode printed on the food package.

[0981] The "RFID reading means" is a device that acquires information from the RFID tag attached to the food material.

[0982] The "database" is a system for storing and managing collected food ingredient information and user health information.

[0983] A "smart device" is a terminal that has the function of collecting a user's health data and transmitting it to a server via a communication means.

[0984] A "user profile" is a set of information that includes personal data such as a user's health status, nutritional requirements, and allergy information.

[0985] The "recipe generation means" is an algorithm that generates appropriate cooking recipes based on the user's profile data and information about ingredients in the refrigerator.

[0986] An "online store" is an online sales site where you can purchase products via the Internet.

[0987] The "online ordering method" is a system that calls the API of affiliated online stores and automatically orders ingredients that are in short supply.

[0988] "Notification means" is a function for sending notifications to users, such as new information, recipe suggestions, and automatic order confirmations.

[0989] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and nutritional requirements, and automatically orders missing ingredients online. This system integrates various hardware and software to enable efficient ingredient management and healthy meal planning.

[0990] Hardware and software used

[0991] Image recognition means: A camera placed inside the refrigerator is used to take images of the interior and run image recognition algorithms to identify the type and quantity of ingredients.

[0992] Barcode scanning means: A device that reads the barcode printed on food packaging.

[0993] RFID reading means: A device that acquires information from the RFID tag attached to the food ingredient.

[0994] Database: A system that stores and manages collected food ingredient information and user health information.

[0995] Smart device: A device (e.g., smartwatch, fitness tracker) that has the ability to collect user health data and transmit it to a server via a communication means.

[0996] Server: A computer system that matches the user's profile data with the information about ingredients in the refrigerator and generates appropriate recipes.

[0997] Program processing

[0998] Acquiring ingredient information

[0999] The server periodically activates the refrigerator's camera to take pictures of the interior. These pictures are then analyzed using image recognition algorithms to identify the type and quantity of ingredients. Barcode scanners and RFID readers are also used to collect detailed information about the ingredients, which is then recorded in a database.

[1000] Health monitoring and profile updates

[1001] Smart devices send user health data (weight, blood pressure, heart rate, blood sugar level, etc.) to a server, which updates the user profile based on the received data, analyzes it as needed, and notifies users of any significant changes.

[1002] Recipe Suggestions

[1003] The server compares the user's profile data with the ingredients in the refrigerator and generates recipes suited to the user's health and nutritional requirements. The generated recipes are then sent to the user via their smart device.

[1004] Automatic ordering of missing ingredients

[1005] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If missing ingredients are identified, the server calls the API of a partner online store to automatically place an order. The user receives an order confirmation notification via their smart device.

[1006] Specific examples

[1007] 1. Add ingredients and update information

[1008] When a user adds a new carton of milk to the refrigerator, the server uses an RFID reader to read the information about the new carton (type, quantity, expiration date), then records the information in the database and notifies the user that "new milk has been added."

[1009] 2. Recipe suggestions and automatic ordering

[1010] The server analyzes the user's profile data and selects a low-carb dinner recipe for a user who wants to manage their blood sugar levels, for example. After checking the information about ingredients in the refrigerator, it is determined that the tomato and lettuce needed for the recipe are missing. The server automatically orders the missing ingredients from the online store and notifies the user that "some of the ingredients needed for the specified recipe have been ordered." The user checks the notification on their smart device and approves or changes the order as necessary.

[1011] This system allows users to efficiently manage ingredients, suggest recipes based on health information, and automatically order, reducing the hassle of everyday shopping and meal planning while supporting a healthy lifestyle.

[1012] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1013] Step 1: Photographing ingredients and analyzing the images

[1014] The server periodically takes pictures of the inside of the refrigerator using a camera inside the refrigerator. It takes the image of the inside of the refrigerator as input and sends the image data to an image recognition algorithm. The server then analyzes the image using the image recognition algorithm and identifies the type and quantity of ingredients as output. Specifically, it identifies three apples and one carton of eggs.

[1015] Step 2: Obtain barcode or RFID information

[1016] When the server detects that a newly added ingredient has been added, it uses a barcode scanner or RFID reader to read the ingredient's information. The barcode or RFID tag information is taken as input, and the server analyzes it to extract data such as type, quantity, and expiration date. The output is to record the specific ingredient information in the database. A specific example of its operation is when a user adds a new carton of milk, recording it as "1L milk, expiration date: October 30, 2023."

[1017] Step 3: Receive health information and update your profile

[1018] The terminal (smart device) collects the user's health data (weight, blood pressure, heart rate, blood glucose level, etc.) on a daily basis and sends the data to a server. The server receives the health data as input, analyzes the data, and updates it with the latest information to update the user profile. The latest user profile is updated as an output. Specifically, the smartwatch sends the nighttime sleep data and morning blood pressure measurement.

[1019] Step 4: Recipe generation and notification

[1020] The server compares the user's profile data with the information about ingredients in the refrigerator and generates an appropriate recipe. It takes the profile data and ingredient information as input, and uses a generative AI model to generate an appropriate recipe. The generated recipe is then sent to the smart device as output. Specifically, it generates a low-carb dinner recipe and sends a notification to the smartphone saying, "Would you like to make a chicken and vegetable salad?"

[1021] Step 5: Auto-order missing ingredients

[1022] The server checks the ingredients needed for the proposed recipe and identifies any missing ingredients by comparing them with the inventory information in the refrigerator. The server receives the list of ingredients needed for the recipe and inventory information as input, and automatically orders the missing ingredients via the online store API. The output is an order confirmation notification sent to the user. Specifically, it identifies that two tomatoes and one lettuce are missing, and completes the automatic ordering process at the online store.

[1023] Step 6: Order confirmation and user notification

[1024] The server sends an automated order confirmation to the user's smart device and provides an interface that allows the user to confirm or change the order details. The server takes the order details as input, generates a confirmation notice, and sends it to the user. The output is a notification that is received by the user. Specifically, the user's smartphone displays "You have ordered two tomatoes and one lettuce," allowing the user to confirm the order details.

[1025] (Application example 1)

[1026] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1027] In today's busy lifestyles, managing food and planning healthy meals has become increasingly difficult. Furthermore, even when shopping in brick-and-mortar stores, managing ingredient inventory and planning purchases can be a time-consuming process. However, due to the lack of appropriate systems to solve these problems, consumers find it difficult to plan healthy meals efficiently. In addition, there is a demand for personalized recipe suggestions based on health data and the ability to automatically order ingredients when they are in short supply. The present invention aims to solve these problems.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1029] In this invention, the server includes an image recognition device installed in the refrigerator, a barcode scanning device or an RFID reader, a means for recording information about ingredients in the refrigerator in a database, a means for receiving user health data from a smart device and updating it in a user profile, a means for comparing the user profile data with the information about ingredients in the refrigerator and generating appropriate recipes, a means for automatically ordering missing ingredients from an online store, a means for collecting ingredient information using an image recognition device installed in smart shelves or smart refrigerators in physical stores, and a means for generating recipes suited to the customer's health condition based on the information and automating food ordering. This enables the creation of efficient and healthy meal plans and reduces the hassle of shopping through the automatic ordering function.

[1030] The "image recognition means" is a device that analyzes images taken with a camera and identifies the type and quantity of ingredients.

[1031] The "barcode scanning means" is a device that reads the barcode printed on the food package and obtains the ingredient information.

[1032] The "RFID reading means" is a device that reads the information attached to the RFID tag and acquires the ingredient information.

[1033] The "means for recording in a database" is a system for storing the acquired ingredient information in a database.

[1034] A "smart device" is a portable electronic device for managing health data and user profiles.

[1035] "Means for receiving user health data and updating the user profile" refers to a system in which a server receives the user's health information sent from a smart device and reflects it in the user profile as the latest information.

[1036] "Means for matching user profile data with information on ingredients in the refrigerator and generating appropriate recipes" refers to a system that suggests recipes using ingredients in the refrigerator based on the user's health condition and nutritional requirements.

[1037] The "means for automatically ordering missing ingredients from an online store" is a system that checks the stock of ingredients required for a proposed recipe and automatically orders the missing ingredients from an online store.

[1038] "Image recognition devices installed on smart shelves and smart refrigerators in physical stores" are devices equipped with cameras and analysis systems that automatically grasp the types of ingredients and inventory in physical stores.

[1039] The "means for automating food ordering" is a system that automatically orders missing ingredients, allowing users to obtain the ingredients they need without any hassle.

[1040] This invention is a system that uses smart shelves and smart refrigerators in brick-and-mortar stores to automatically obtain information about ingredients, suggest recipes based on the user's health status and nutritional requirements, and automatically order missing ingredients online. This system is realized using multiple hardware and software components.

[1041] System configuration

[1042] How to obtain food information

[1043] The server uses image recognition, barcode scanning, or RFID reading methods built into smart refrigerators and smart shelves in physical stores. This allows the server to identify the type, quantity, and expiration date of ingredients and record the obtained information in a database. TensorFlow is used for image recognition, and common scanners and readers are used for barcode scanning and RFID reading.

[1044] For example, when organic vegetables are added as new ingredients to a physical store, the server uses an RFID reader to obtain this information and stores it in a database, which also stores the type, quantity, and expiration date of the ingredients.

[1045] Health information monitoring and profile updating methods

[1046] The server receives health data from smart devices such as smartwatches and fitness trackers. Received data includes the user's weight, blood pressure, heart rate, blood sugar level, etc. This received data is automatically updated to the user profile on the server. Python's Pandas and NumPy are used to manage and analyze the data.

[1047] For example, a user's new weight measurement on their smartwatch is sent to the server, and the data is updated in the user profile.

[1048] Recipe suggestion method

[1049] The server generates appropriate recipes based on the user's health information and the ingredients in the refrigerator. This recipe is generated using the machine learning model scikit-learn. The generated recipe is then sent to the user's smart device.

[1050] For example, the server selects a low-carb dinner recipe, compares the ingredients in the refrigerator to generate the recipe, and notifies the user.

[1051] Automatic ordering of ingredients that are in short supply

[1052] The server checks the refrigerator stock of ingredients needed for the proposed recipe and automatically orders any missing ingredients from the online store. The order process calls the API of the affiliated online store. If the order details need to be confirmed or changed, a notification is sent to the user's smart device, allowing them to make changes via the interface.

[1053] If the missing ingredients are identified as tomatoes and lettuce, the server uses the online store's ordering API to order these ingredients and notifies the user that "some of the ingredients required for the specified recipe have been ordered."

[1054] Prompt Sentence Examples

[1055] "Automatically order missing ingredients online based on your desired recipe."

[1056] As described above, the present invention provides a system that allows for efficient and healthy meal planning and significantly reduces the effort required for shopping.

[1057] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1058] Step 1: Obtaining ingredient information

[1059] The server acquires ingredient information using cameras, barcode scanners, or RFID readers built into smart shelves and smart refrigerators in physical stores. Specifically, the camera takes an image of the ingredient, and an image recognition algorithm (e.g., TensorFlow) is used to identify the type and quantity of the ingredient. The acquired information is recorded in a database. The input is an image of the ingredient or barcode / RFID label data, and the output is ingredient information including the ingredient name, quantity, and expiration date.

[1060] Step 2: Monitor your health information and update your profile

[1061] The server receives health data sent from the user's smart device (smartwatch or fitness tracker). The received data includes weight, blood pressure, heart rate, blood glucose level, etc. The server analyzes the data and updates the user profile. The input is the health data sent from the smart device, and the output is the updated user profile.

[1062] Step 3: Recipe suggestions

[1063] The server generates recipes based on the user's health information and information about ingredients in the refrigerator. A machine learning model (scikit-learn) is used for generation. The input is the user's health data and information about ingredients in the refrigerator, and the output is a recipe appropriate for the user's health condition. The generated recipe is then sent to the user's smart device.

[1064] Step 4: Identify missing ingredients

[1065] The server creates a list of ingredients needed based on the generated recipe and compares it with the refrigerator's inventory information. It identifies missing ingredients and compiles that information. The input is the list of ingredients needed for the recipe and the refrigerator's inventory information, and the output is the list of missing ingredients.

[1066] Step 5: Automated ordering of ingredients

[1067] The server automatically orders the identified missing ingredients from the online store. The ordering is done using the API of the partner online store. The input is the list of missing ingredients, and the output is order confirmation information. The user is notified of the automatic order via their smart device.

[1068] Step 6: Order confirmation and receipt notification

[1069] Users can confirm and change their order details through their smart device. After placing an order, they receive a notification about where to pick up the ingredients in the store. The input is order confirmation information, and the output is a notification to the user and instructions on where to pick up the ingredients.

[1070] Through the above processing flow, the present invention can support efficient and healthy meal planning in physical stores and provide an automatic ordering function for missing ingredients.

[1071] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1072] This invention is a system that automatically acquires information about ingredients in a refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health and emotional data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[1073] A natural language description of the program's operation

[1074] Acquiring ingredient information

[1075] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[1076] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[1077] Health and emotional monitoring and profile updates

[1078] The user's health and emotional information is sent from the smart device to the server.

[1079] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. It then uses an emotion engine to analyze the user's voice and facial expressions to generate emotion data. This data is updated based on the user profile. The server also analyzes the user's health and emotion information and notifies the user of any significant changes.

[1080] Recipe Suggestions

[1081] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[1082] The server compares the user's health information and emotional state (e.g., stress level and mood) with the ingredients available in the refrigerator and generates recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences, health, and emotional state, and are notified to the user via their smart device.

[1083] Automatic ordering of missing ingredients

[1084] The server automatically orders missing ingredients online for the suggested recipe.

[1085] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[1086] Specific examples

[1087] Recipe suggestions based on emotional data

[1088] 1. Add ingredients and update information

[1089] When a new milk carton is added to the refrigerator, the server uses an RFID reader to read the information about the new milk carton (e.g., type, quantity, expiration date).

[1090] The server records the obtained information in a database.

[1091] The device then notifies the user that "New milk has been added."

[1092] 2. Collecting emotional data using an emotion engine

[1093] When a user uses a smart device, the emotion engine analyzes facial expressions and voice to identify the user's emotional state (e.g., stress level and mood).

[1094] The emotion data is sent to the server and updated in the user profile.

[1095] 3. Recipe suggestions based on emotional and health data

[1096] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective in reducing stress) based on the user's current health and emotional state.

[1097] The server checks the information about ingredients in the refrigerator and identifies the ingredients needed for the recipe.

[1098] Additionally, if necessary, any missing ingredients will be automatically ordered from the online store.

[1099] 4. Check recipe notifications and auto-orders

[1100] The generated recipe is notified to the user via their device, and a message is displayed saying, "A salad recipe that is effective in reducing stress has been suggested."

[1101] The user selects a salad recipe and the server automatically orders any missing ingredients.

[1102] The terminal will send the user an automated order confirmation and allow the user to review and modify the order.

[1103] This system allows users to efficiently manage ingredients in their refrigerator, and offers recipe suggestions and automatic ordering based on health and emotional information, thereby reducing the hassle of everyday shopping and meal planning and supporting a lifestyle that is healthy and meets emotional needs.

[1104] The processing flow will be explained below.

[1105] Step 1:

[1106] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[1107] Step 2:

[1108] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[1109] Step 3:

[1110] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[1111] Step 4:

[1112] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[1113] Step 5:

[1114] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[1115] Step 6:

[1116] The terminal (smart device) uses an emotion engine to analyze the user's facial expressions and voice to recognize emotions. The recognized emotion data is sent to the server and updated in the user profile.

[1117] Step 7:

[1118] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies, nutritional requirements, and emotional state. The generated recipes are prioritized based on the user's preferences, health status, and emotional state.

[1119] Step 8:

[1120] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[1121] Step 9:

[1122] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[1123] Step 10:

[1124] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[1125] Step 11:

[1126] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[1127] Specific examples

[1128] Recipe suggestions based on emotional data

[1129] Step 1:

[1130] The server uses the refrigerator's camera to take pictures of the inside of the refrigerator.

[1131] The captured image is analyzed to identify information about ingredients such as apples and milk.

[1132] The analysis results are recorded in a database.

[1133] Step 2:

[1134] The server scans the barcode of the milk using a barcode scanner in the refrigerator.

[1135] Detailed information about the milk (type, quantity, expiration date) is recorded in a database from the barcode information.

[1136] Step 3:

[1137] When a user adds a new yogurt to the refrigerator, the barcode is read using the barcode scanner function of the terminal (smart device).

[1138] The information read is sent to a server, and the yogurt information is recorded in a database.

[1139] Step 4:

[1140] The terminal (smart device) periodically measures the user's weight, blood pressure, heart rate, blood sugar level, etc. and sends the data to the server.

[1141] The server updates the user profile with the received data.

[1142] Step 5:

[1143] The server analyzes health data and detects high blood pressure.

[1144] Based on this information, recipes are generated that take into account appropriate nutritional balance.

[1145] Step 6:

[1146] The emotion engine in the device (smart device) analyzes the user's facial expressions and voice and recognizes that the user is currently feeling stressed.

[1147] The emotion data is sent to the server and updated in the user profile.

[1148] Step 7:

[1149] The server compares the user's health and emotional data and suggests recipes that are effective in reducing stress.

[1150] The generated recipe is notified to the user as an "herbal tea recipe effective for stress relief."

[1151] Step 8:

[1152] The user sees the notification and selects the suggested recipe.

[1153] The server double-checks the ingredients needed for the recipe and compares them with the stock in the refrigerator.

[1154] Identify missing herbal tea ingredients.

[1155] Step 9:

[1156] The server calls the API of a partner online store and automatically orders the missing herbs for herbal tea.

[1157] The order process is completed and the details are recorded in a database.

[1158] Step 10:

[1159] The terminal (smart device) sends the user a confirmation of the automatic order.

[1160] The user can review the notification and approve or change the order.

[1161] In this way, a system that combines an emotion engine can suggest individual recipes based on the user's health condition and emotions, providing more effective support for daily stress management and health management.

[1162] Example 2

[1163] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1164] In today's lifestyle, managing the ingredients in the refrigerator and creating meal plans based on the user's health and emotional information is extremely time-consuming. Furthermore, checking and purchasing ingredients that are lacking also increases the time required. A system that solves these problems and supports an efficient and healthy lifestyle is needed.

[1165] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a photographing means, an identification means, a means for recording information about ingredients in the refrigerator, a means for receiving the user's health data from the smart device and updating it in the user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating recipes, a means for ordering missing ingredients online, a means for collecting the user's emotion data and updating the profile, a means for notifying the user of significant changes in the health data and emotion data, and a means for sending a confirmation notice of the automatic order. This automates a series of processes, from ingredient management to health information monitoring, recipe suggestions, and automatic ordering of missing ingredients, enabling the user to maintain an efficient and healthy lifestyle.

[1166] The "photography means" is a tool that is installed inside the refrigerator and is used to take pictures of ingredients.

[1167] "Identification means" refers to a method for analyzing the captured image and read information to identify the type and quantity of ingredients.

[1168] The "recording means" is a method for storing the collected information on ingredients in a database.

[1169] A "smart device" is a terminal that collects a user's health information and emotional data and transmits it to a server.

[1170] A "user profile" is a database entry that records and updates a user's health and emotional state.

[1171] The "matching means" is a method for comparing user profile data with information about ingredients in the refrigerator and generating an appropriate recipe based on that information.

[1172] A "generator" is a method for creating a recipe based on the collated data.

[1173] An "online ordering method" is a method for automatically purchasing missing ingredients required for a suggested recipe.

[1174] "Emotional data" is data on the emotional state of a user obtained by analyzing their voice and facial expressions.

[1175] "Means for notifying significant changes" refers to a method for notifying a user when there is a significant change in the user's health data or emotional data.

[1176] "Confirmation notification means" refers to a method for informing users of the contents of an online order after it has been completed and prompting them to confirm or change the order.

[1177] This invention relates to a system that automatically acquires information about ingredients in a refrigerator based on the user's health and emotional state, suggests appropriate recipes, and orders missing ingredients online.

[1178] Specifically, this system is implemented using the following hardware and software.

[1179] Hardware Configuration

[1180] 1. Photographing means inside the refrigerator: A camera placed inside the refrigerator is a device that takes pictures of ingredients.

[1181] 2. Identification means: A device that uses a barcode scanner or RFID reader to identify the type and quantity of ingredients.

[1182] 3. Smart devices: These are devices such as smartwatches and fitness trackers used by users that collect health and emotional data and send it to a server.

[1183] Software Configuration

[1184] 1. Image recognition algorithm: Software that runs on the server and analyzes the captured image to identify the type and quantity of ingredients.

[1185] 2. Database management system: A system for storing and managing collected food ingredient information, health data, and emotional data.

[1186] 3. Recipe generation engine: Software that generates appropriate recipes based on the user's health and emotional state.

[1187] 4. Order management system: A system that identifies shortages of ingredients and automatically places orders by calling the API of affiliated online stores.

[1188] 5. Notification Engine: This software notifies users with recipe suggestions and automatic ordering confirmations.

[1189] Example of operation

[1190] Example 1: Collecting and recording information about ingredients in the refrigerator

[1191] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[1192] The server uses an image recognition algorithm to analyze the captured image and identify the type and quantity of ingredients.

[1193] For newly added ingredients, the server uses a barcode scanner or RFID reader in the refrigerator to read the identifying information.

[1194] The server records the obtained information in a database, saving the names, quantities, expiration dates, etc. of ingredients.

[1195] Example 2: Monitoring health and emotional information and updating profiles

[1196] Users wear smart devices and collect health data (e.g., weight, blood pressure, heart rate, blood glucose levels) on a daily basis.

[1197] Smart devices periodically send the collected data to a server.

[1198] The server uses an emotion engine to analyze the user's voice and facial expressions and generate emotion data.

[1199] The collected health and emotional data is updated by the server into a user profile.

[1200] For example, if a user's stress level increases, the server uses a notification engine to notify the user of important fluctuations.

[1201] Example 3: Recipe suggestions

[1202] The server matches the user profile data with the ingredients in the refrigerator and generates appropriate recipes based on the user's health and emotional state.

[1203] For example, if a user's stress level is high, the app will generate recipes using ingredients that are effective in reducing stress.

[1204] The server notifies the smart device of the generated recipe so that the user can check it.

[1205] Example 4: Automatic ordering and notification of missing ingredients

[1206] The server checks the ingredients needed for the generated recipe against the inventory information in the refrigerator.

[1207] When a missing ingredient is identified, the server calls the API of a partner online store to automatically place the order.

[1208] After the order is completed, the server uses a notification engine to send a confirmation to the user, who can then review the order details through their smart device and make any necessary changes.

[1209] This system allows users to efficiently carry out all processes, from managing ingredients in the refrigerator to monitoring health information, collecting emotional data, suggesting recipes, and automatically ordering ingredients when they are running low.

[1210] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1211] Step 1: Collect images of the inside of the refrigerator

[1212] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[1213] Input: An image of the inside of a refrigerator taken by a camera.

[1214] Output: Captured image data.

[1215] Specific operation: The server activates the camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added.

[1216] Step 2: Analyze with image recognition algorithm

[1217] The server uses image recognition algorithms to analyze the images sent to it.

[1218] Input: Captured image data.

[1219] Output: Data on the type and quantity of ingredients analyzed.

[1220] What it does: The server analyzes the received image and runs an image recognition algorithm to identify the type and quantity of ingredients.

[1221] Step 3: Barcode scanning or RFID reading

[1222] The server activates the barcode scanner and RFID reader inside the refrigerator.

[1223] Input: Barcode or RFID tag information of the newly added ingredient.

[1224] Output: Detailed information about the ingredients read (e.g. name, quantity, expiration date).

[1225] Specific operation: The server activates a barcode scanner or RFID reader to read the information about the new ingredients.

[1226] Step 4: Record in the database

[1227] The server records the analysis results and the barcode / RFID tag information read in a database.

[1228] Input: Parsed ingredient information, read ingredient information.

[1229] Output: Detailed information such as ingredient name, quantity, expiration date, etc. recorded in the database.

[1230] Specific operation: The server stores the obtained ingredient information in a database and manages the ingredients.

[1231] Step 5: Collecting health data

[1232] The smart device that serves as the terminal collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.).

[1233] Input: User's everyday health data.

[1234] Output: Collected health data.

[1235] How it works: Smart devices such as smartwatches and fitness trackers continuously collect biometric data from their users.

[1236] Step 6: Collect emotion data

[1237] The smart device analyzes the user's voice and facial expressions.

[1238] Input: User voice and facial expression data.

[1239] Output: Parsed emotion data.

[1240] What it does: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state.

[1241] Step 7: Update your user profile

[1242] The server updates the user profile based on the health and emotional data.

[1243] Input: Collected health and emotion data.

[1244] Output: Updated user profile.

[1245] Specific operation: The server updates the profile with newly acquired data, maintaining the latest health and emotional state.

[1246] Step 8: Notification of significant changes

[1247] The server analyzes the user's health and emotional information and notifies them if there are any significant changes.

[1248] Input: Continuously updated health and emotion data.

[1249] Output: Notification message of significant fluctuations.

[1250] Specific operation: The server will send an alert to the user if there is a fluctuation that exceeds a certain standard.

[1251] Step 9: Matching profile data with ingredient information

[1252] The server matches the user profile with the information about ingredients in the refrigerator.

[1253] Input: User profile data, refrigerator food information.

[1254] Output: Matching results.

[1255] Specific operation: The server compares the profile data with the ingredient information and obtains the data necessary to suggest appropriate recipes.

[1256] Step 10: Recipe Generation

[1257] The server generates an appropriate recipe based on the matching results.

[1258] Input: Matching result.

[1259] Output: The generated recipe.

[1260] How it works: The server uses a generative AI model to create a recipe tailored to the user.

[1261] Step 11: Prioritize and notify

[1262] The server prioritizes the generated recipes based on the user's preferences and health status and notifies them.

[1263] Input: The generated recipe.

[1264] Output: Prioritized recipe notifications.

[1265] Specific operation: The server evaluates the importance of the recipe and notifies the user via the smart device.

[1266] Step 12: Identify missing ingredients

[1267] The server checks the ingredients needed for the proposed recipe against the inventory information in the refrigerator.

[1268] Inputs: Suggested recipes, refrigerator inventory.

[1269] Output: A list of missing ingredients.

[1270] What happens: The server compares the ingredients required for the recipe with the current inventory and identifies any missing ingredients.

[1271] Step 13: Order online

[1272] The server will then process the missing ingredients for ordering online.

[1273] Input: Missing ingredient list.

[1274] Output: Order information from the online store.

[1275] Specific operation: The server calls the API of the affiliated online store and automatically orders the necessary ingredients.

[1276] Step 14: Order confirmation

[1277] The server will send a confirmation to the user after the order is completed.

[1278] Input: Order checkout completion information.

[1279] Output: Order confirmation notification.

[1280] Specific operation: Once the order is completed, the server notifies the user of the order details via the smart device, allowing them to confirm or change the order.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1283] Conventional refrigerator management systems only managed food inventory and suggested recipes, but were unable to provide suggestions that took into account the user's health or emotional state. They also lacked the ability to automatically order ingredients when they were running low or display recipes in real time, making them inconvenient for users. Furthermore, the recipe options were limited, leaving a need for a more flexible system that could adapt to changes in the user's health and emotions.

[1284] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means arranged in the refrigerator, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for receiving audio or video from the smart device and determining the user's emotional state, and a means for displaying the recipe on the smart glasses. This enables the system to propose appropriate recipes taking into account the user's health and emotional state, automatically order missing ingredients, and display the recipe on the smart glasses in real time.

[1285] The "image recognition means" is a means for analyzing an image captured using a camera and identifying an object contained in the image.

[1286] "Barcode scanning means" refers to means for reading a barcode and obtaining the information written thereon.

[1287] "RFID reading means" refers to a means for reading data from an RFID tag using wireless communication.

[1288] The "means for recording in a database" refers to a means for recording in a database for managing and storing the acquired data.

[1289] The "means for updating the user profile" refers to a means for updating the user profile with the latest information based on the health data and emotional data received from the user.

[1290] The "means for generating an appropriate recipe" is a means for creating an optimal recipe based on the user's profile data and information about ingredients in the refrigerator.

[1291] "Means for automatic ordering from online store" refers to a means for automatically ordering missing ingredients from an online store.

[1292] The "means for displaying a recipe on smart glasses" refers to a means for displaying the generated recipe on the display of smart glasses.

[1293] "Means for receiving audio or video and determining the emotional state of a user" refers to means for analyzing audio or video from a smart device and identifying and determining the emotional state of a user.

[1294] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system works in conjunction with smart glasses and smart devices to support efficient and healthy meal planning.

[1295] System configuration

[1296] Hardware Configuration

[1297] 1. Refrigerator

[1298] It is equipped with a built-in camera, barcode scanning means, and RFID reading means, which are used to collect information about ingredients in the refrigerator.

[1299] 2. Smart Glasses

[1300] Equipped with a display, camera, and microphone, it monitors the user's health and emotional state while displaying generated recipes.

[1301] 3. Smart Devices

[1302] Devices such as smartwatches that collect health data such as weight, blood pressure, heart rate, and blood sugar levels.

[1303] Software Configuration

[1304] 1. Image recognition algorithms (e.g., OpenCV)

[1305] The system analyzes images taken by a camera inside the refrigerator to determine the type, quantity, and expiration date of ingredients.

[1306] 2. Sentiment analysis engine (e.g., Emotion API)

[1307] It analyzes data from the smart glasses' camera or microphone to identify the user's emotional state.

[1308] 3. Data Management System

[1309] A database that manages ingredient information and user profile data.

[1310] 4. Recipe Generation Algorithm

[1311] An algorithm that generates appropriate recipes based on the user's health and emotional data.

[1312] 5. Online Store API (e.g., API for online retailers)

[1313] An API for automatically ordering missing ingredients.

[1314] System Operation

[1315] Acquiring ingredient information

[1316] The server uses the refrigerator's built-in camera, barcode scanner, and RFID reader to take pictures of the refrigerator's interior and collect information about ingredients. It uses image recognition algorithms to identify the type and quantity of ingredients and records this information in a database. The information is updated periodically or when new ingredients are added.

[1317] Health and emotional monitoring and profile updates

[1318] The server receives user health and emotional data from smart glasses and smart devices. Data such as weight, blood pressure, heart rate, and blood sugar levels are collected from smartwatches and other devices, and an emotional analysis engine is used to analyze voice and facial expressions to generate emotional data. This information is updated based on the user profile.

[1319] Recipe Suggestions

[1320] The server generates optimal recipes based on the user profile data and the ingredients in the refrigerator. The generated recipes are prioritized and notified to the user through the smart glasses display. The user can then view the recipes on the smart glasses.

[1321] Automatic ordering of missing ingredients

[1322] The server checks the ingredients required for the proposed recipe and automatically orders any missing ingredients from the online store. The order is processed through the online store API, and a confirmation of the automatic order is sent to the user via the smart glasses. The user can then confirm and change the order details.

[1323] Specific examples

[1324] Recipe suggestions based on emotional data

[1325] 1. Add ingredients and update information

[1326] When a user adds a new ingredient (e.g., a carton of milk) to the refrigerator, the built-in RFID reader reads the new ingredient information, and the server records it in the database. The terminal notifies the user that "a new ingredient has been added."

[1327] 2. Collecting emotional data using an emotion engine

[1328] When a user wears the smart glasses, the emotion engine analyzes their facial expressions and voice to identify their emotional state, and the emotion data is sent to the server and updated in the user profile.

[1329] 3. Recipe suggestions based on emotional and health data

[1330] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective for stress reduction) based on the user's health and emotional state. If necessary, missing ingredients are automatically ordered from the online store.

[1331] 4. Check recipe notifications and auto-orders

[1332] The generated recipe is sent to the user through the smart glasses. The user selects the recipe and the server automatically orders any missing ingredients. The order details are displayed on the smart glasses display, and the user can check and change them.

[1333] Prompt Sentence Examples

[1334] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1:

[1337] The server uses a camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added. This obtains image data of the inside of the refrigerator. The server then uses an image recognition algorithm (e.g., OpenCV) to identify the type, quantity, and expiration date of each ingredient in the image. This information is recorded in a database. The input is the image data of the inside of the refrigerator, and the output is ingredient information (type, quantity, expiration date).

[1338] Step 2:

[1339] The server receives the user's health data from the smart device (e.g., smart watch). This data includes weight, blood pressure, heart rate, blood glucose level, etc. The server updates this data to the user profile. It also collects emotion data from the user's voice and video using the camera and microphone of the smart glasses, and analyzes it using an emotion analysis engine (e.g., Emotion API). The input is the audio and video data from the smart device and smart glasses, and the output is the updated health data and emotion data.

[1340] Step 3:

[1341] The server compares the collected health and emotional data with the information about ingredients in the refrigerator. It then runs a recipe generation algorithm to generate optimal recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences and current health and emotional state. The inputs are the health data, emotional data, and ingredient information, and the output is the generated recipe.

[1342] Step 4:

[1343] The device displays the generated recipe on the smart glasses display, allowing the user to check and select the recipe on the smart glasses. The input is the generated recipe, and the output is the recipe display on the smart glasses.

[1344] Step 5:

[1345] The server checks whether the refrigerator contains all the necessary ingredients based on the selected recipe. If any ingredients are missing, it identifies them and calls the online store API to automatically place an order. The order details are displayed on the smart glasses and can be confirmed and changed by the user. The input is the information about ingredients in the refrigerator and the selected recipe, and the output is the ordered ingredients and their confirmation information.

[1346] Specific examples of processing

[1347] In step 1, when new milk is added to the refrigerator, an image of the inside of the refrigerator is taken with a camera, and the type, quantity, and expiration date of the milk are identified using OpenCV, and this is recorded in the database.

[1348] In step 2, the server receives weight and heart rate data from the smartwatch, and sends voice data from the smartglasses to an emotion analysis engine to determine the user's emotional state and update the profile.

[1349] In step 3, a recipe generation algorithm uses this data to generate and prioritize "stress-reducing salads."

[1350] In step 4, the recipe is displayed on the smart glasses display for the user to review.

[1351] In step 5, if it is determined that the refrigerator is low on necessary vegetables, an automatic order is placed using the online store API, and the order details are displayed on the smart glass.

[1352] Prompt Sentence Examples

[1353] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[1354] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1355] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1356] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1357] [Fourth embodiment]

[1358] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1359] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1360] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1361] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1362] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1363] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1364] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1365] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1366] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1369] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1370] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1371] This invention is a system that automatically acquires information about ingredients stored in a refrigerator, suggests recipes based on the user's health condition and nutritional requirements, and automatically orders missing ingredients online. The system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients stored in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[1372] A natural language description of the program's operation

[1373] Acquiring ingredient information

[1374] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[1375] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[1376] Health monitoring and profile updates

[1377] The user's health information is sent from the smart device, which serves as the terminal, to the server.

[1378] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. The received data is updated based on the user's profile. The server also analyzes the user's health information and notifies them of any significant changes.

[1379] Recipe Suggestions

[1380] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[1381] The server compares the user's health information (e.g., nutritional requirements and allergies) with the ingredients available in the refrigerator and generates recipes suited to the user's health condition. The generated recipes are prioritized based on the user's preferences and health condition and are notified to the user via their smart device.

[1382] Automatic ordering of missing ingredients

[1383] The server automatically orders missing ingredients online for the suggested recipe.

[1384] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[1385] Specific examples

[1386] Adding ingredients and updating information

[1387] 1. Suppose you add a new carton of milk to the refrigerator.

[1388] 2. The server uses an RFID reader to read the information on the new milk carton (e.g. type, quantity, expiration date).

[1389] 3. The server records the obtained information in a database.

[1390] 4. The device then notifies the user that "New milk has been added."

[1391] Recipe suggestions and auto-ordering

[1392] 1. The server analyzes the user's profile data and selects low-carb dinner recipes that take blood sugar levels into consideration.

[1393] 2. Check the ingredients in the refrigerator and confirm that you are missing tomatoes and lettuce, which are ingredients needed for the recipe.

[1394] 3. The server automatically orders the missing ingredients from the online store and notifies the user that "Some of the ingredients required for the specified recipe have been ordered."

[1395] 4. The user checks the notification on their smart device and approves or changes the order as needed.

[1396] This system allows users to efficiently manage ingredients in their refrigerators and automatically order and suggest recipes based on health information, reducing the hassle of everyday shopping and meal planning and supporting a healthy lifestyle.

[1397] The processing flow will be explained below.

[1398] Step 1:

[1399] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[1400] Step 2:

[1401] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[1402] Step 3:

[1403] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[1404] Step 4:

[1405] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[1406] Step 5:

[1407] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[1408] Step 6:

[1409] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies and nutritional requirements, and prioritizes the recipes based on the user's health status and preferences.

[1410] Step 7:

[1411] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[1412] Step 8:

[1413] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[1414] Step 9:

[1415] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[1416] Step 10:

[1417] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[1418] Example 1

[1419] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1420] In today's busy lifestyles, managing ingredients and planning meals based on health status takes a lot of time and effort. Furthermore, if necessary ingredients are in short supply, users are forced to go shopping every time, wasting time and effort. There is a need for a system that can efficiently solve these issues and simultaneously improve users' health and convenience.

[1421] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1422] In this invention, the server includes an image recognition means, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for updating the database and notifying the user when an ingredient is added, and a means for notifying the user's smart device of the generated recipe. This allows the user to efficiently manage ingredients, monitor health information, suggest recipes, and automatically order missing ingredients.

[1423] The "image recognition means" is a device that uses a camera placed inside the refrigerator to take images of ingredients and includes an algorithm that identifies the type and quantity of ingredients.

[1424] A "barcode scanning means" is a device that reads the barcode printed on the food package.

[1425] The "RFID reading means" is a device that acquires information from the RFID tag attached to the food material.

[1426] The "database" is a system for storing and managing collected food ingredient information and user health information.

[1427] A "smart device" is a terminal that has the function of collecting a user's health data and transmitting it to a server via a communication means.

[1428] A "user profile" is a set of information that includes personal data such as a user's health status, nutritional requirements, and allergy information.

[1429] The "recipe generation means" is an algorithm that generates appropriate cooking recipes based on the user's profile data and information about ingredients in the refrigerator.

[1430] An "online store" is an online sales site where you can purchase products via the Internet.

[1431] The "online ordering method" is a system that calls the API of affiliated online stores and automatically orders ingredients that are in short supply.

[1432] "Notification means" is a function for sending notifications to users, such as new information, recipe suggestions, and automatic order confirmations.

[1433] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and nutritional requirements, and automatically orders missing ingredients online. This system integrates various hardware and software to enable efficient ingredient management and healthy meal planning.

[1434] Hardware and software used

[1435] Image recognition means: A camera placed inside the refrigerator is used to take images of the interior and run image recognition algorithms to identify the type and quantity of ingredients.

[1436] Barcode scanning means: A device that reads the barcode printed on food packaging.

[1437] RFID reading means: A device that acquires information from the RFID tag attached to the food ingredient.

[1438] Database: A system that stores and manages collected food ingredient information and user health information.

[1439] Smart device: A device (e.g., smartwatch, fitness tracker) that has the ability to collect user health data and transmit it to a server via a communication means.

[1440] Server: A computer system that matches the user's profile data with the information about ingredients in the refrigerator and generates appropriate recipes.

[1441] Program processing

[1442] Acquiring ingredient information

[1443] The server periodically activates the refrigerator's camera to take pictures of the interior. These pictures are then analyzed using image recognition algorithms to identify the type and quantity of ingredients. Barcode scanners and RFID readers are also used to collect detailed information about the ingredients, which is then recorded in a database.

[1444] Health monitoring and profile updates

[1445] Smart devices send user health data (weight, blood pressure, heart rate, blood sugar level, etc.) to a server, which updates the user profile based on the received data, analyzes it as needed, and notifies users of any significant changes.

[1446] Recipe Suggestions

[1447] The server compares the user's profile data with the ingredients in the refrigerator and generates recipes suited to the user's health and nutritional requirements. The generated recipes are then sent to the user via their smart device.

[1448] Automatic ordering of missing ingredients

[1449] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If missing ingredients are identified, the server calls the API of a partner online store to automatically place an order. The user receives an order confirmation notification via their smart device.

[1450] Specific examples

[1451] 1. Add ingredients and update information

[1452] When a user adds a new carton of milk to the refrigerator, the server uses an RFID reader to read the information about the new carton (type, quantity, expiration date), then records the information in the database and notifies the user that "new milk has been added."

[1453] 2. Recipe suggestions and automatic ordering

[1454] The server analyzes the user's profile data and selects a low-carb dinner recipe for a user who wants to manage their blood sugar levels, for example. After checking the information about ingredients in the refrigerator, it is determined that the tomato and lettuce needed for the recipe are missing. The server automatically orders the missing ingredients from the online store and notifies the user that "some of the ingredients needed for the specified recipe have been ordered." The user checks the notification on their smart device and approves or changes the order as necessary.

[1455] This system allows users to efficiently manage ingredients, suggest recipes based on health information, and automatically order, reducing the hassle of everyday shopping and meal planning while supporting a healthy lifestyle.

[1456] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1457] Step 1: Photographing ingredients and analyzing the images

[1458] The server periodically takes pictures of the inside of the refrigerator using a camera inside the refrigerator. It takes the image of the inside of the refrigerator as input and sends the image data to an image recognition algorithm. The server then analyzes the image using the image recognition algorithm and identifies the type and quantity of ingredients as output. Specifically, it identifies three apples and one carton of eggs.

[1459] Step 2: Obtain barcode or RFID information

[1460] When the server detects that a newly added ingredient has been added, it uses a barcode scanner or RFID reader to read the ingredient's information. The barcode or RFID tag information is taken as input, and the server analyzes it to extract data such as type, quantity, and expiration date. The output is to record the specific ingredient information in the database. A specific example of its operation is when a user adds a new carton of milk, recording it as "1L milk, expiration date: October 30, 2023."

[1461] Step 3: Receive health information and update your profile

[1462] The terminal (smart device) collects the user's health data (weight, blood pressure, heart rate, blood glucose level, etc.) on a daily basis and sends the data to a server. The server receives the health data as input, analyzes the data, and updates it with the latest information to update the user profile. The latest user profile is updated as an output. Specifically, the smartwatch sends the nighttime sleep data and morning blood pressure measurement.

[1463] Step 4: Recipe generation and notification

[1464] The server compares the user's profile data with the information about ingredients in the refrigerator and generates an appropriate recipe. It takes the profile data and ingredient information as input, and uses a generative AI model to generate an appropriate recipe. The generated recipe is then sent to the smart device as output. Specifically, it generates a low-carb dinner recipe and sends a notification to the smartphone saying, "Would you like to make a chicken and vegetable salad?"

[1465] Step 5: Auto-order missing ingredients

[1466] The server checks the ingredients needed for the proposed recipe and identifies any missing ingredients by comparing them with the inventory information in the refrigerator. The server receives the list of ingredients needed for the recipe and inventory information as input, and automatically orders the missing ingredients via the online store API. The output is an order confirmation notification sent to the user. Specifically, it identifies that two tomatoes and one lettuce are missing, and completes the automatic ordering process at the online store.

[1467] Step 6: Order confirmation and user notification

[1468] The server sends an automated order confirmation to the user's smart device and provides an interface that allows the user to confirm or change the order details. The server takes the order details as input, generates a confirmation notice, and sends it to the user. The output is a notification that is received by the user. Specifically, the user's smartphone displays "You have ordered two tomatoes and one lettuce," allowing the user to confirm the order details.

[1469] (Application example 1)

[1470] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1471] In today's busy lifestyles, managing food and planning healthy meals has become increasingly difficult. Furthermore, even when shopping in brick-and-mortar stores, managing ingredient inventory and planning purchases can be a time-consuming process. However, due to the lack of appropriate systems to solve these problems, consumers find it difficult to plan healthy meals efficiently. In addition, there is a demand for personalized recipe suggestions based on health data and the ability to automatically order ingredients when they are in short supply. The present invention aims to solve these problems.

[1472] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1473] In this invention, the server includes an image recognition device installed in the refrigerator, a barcode scanning device or an RFID reader, a means for recording information about ingredients in the refrigerator in a database, a means for receiving user health data from a smart device and updating it in a user profile, a means for comparing the user profile data with the information about ingredients in the refrigerator and generating appropriate recipes, a means for automatically ordering missing ingredients from an online store, a means for collecting ingredient information using an image recognition device installed in smart shelves or smart refrigerators in physical stores, and a means for generating recipes suited to the customer's health condition based on the information and automating food ordering. This enables the creation of efficient and healthy meal plans and reduces the hassle of shopping through the automatic ordering function.

[1474] The "image recognition means" is a device that analyzes images taken with a camera and identifies the type and quantity of ingredients.

[1475] The "barcode scanning means" is a device that reads the barcode printed on the food package and obtains the ingredient information.

[1476] The "RFID reading means" is a device that reads the information attached to the RFID tag and acquires the ingredient information.

[1477] The "means for recording in a database" is a system for storing the acquired ingredient information in a database.

[1478] A "smart device" is a portable electronic device for managing health data and user profiles.

[1479] "Means for receiving user health data and updating the user profile" refers to a system in which a server receives the user's health information sent from a smart device and reflects it in the user profile as the latest information.

[1480] "Means for matching user profile data with information on ingredients in the refrigerator and generating appropriate recipes" refers to a system that suggests recipes using ingredients in the refrigerator based on the user's health condition and nutritional requirements.

[1481] The "means for automatically ordering missing ingredients from an online store" is a system that checks the stock of ingredients required for a proposed recipe and automatically orders the missing ingredients from an online store.

[1482] "Image recognition devices installed on smart shelves and smart refrigerators in physical stores" are devices equipped with cameras and analysis systems that automatically grasp the types of ingredients and inventory in physical stores.

[1483] The "means for automating food ordering" is a system that automatically orders missing ingredients, allowing users to obtain the ingredients they need without any hassle.

[1484] This invention is a system that uses smart shelves and smart refrigerators in brick-and-mortar stores to automatically obtain information about ingredients, suggest recipes based on the user's health status and nutritional requirements, and automatically order missing ingredients online. This system is realized using multiple hardware and software components.

[1485] System configuration

[1486] How to obtain food information

[1487] The server uses image recognition, barcode scanning, or RFID reading methods built into smart refrigerators and smart shelves in physical stores. This allows the server to identify the type, quantity, and expiration date of ingredients and record the obtained information in a database. TensorFlow is used for image recognition, and common scanners and readers are used for barcode scanning and RFID reading.

[1488] For example, when organic vegetables are added as new ingredients to a physical store, the server uses an RFID reader to obtain this information and stores it in a database, which also stores the type, quantity, and expiration date of the ingredients.

[1489] Health information monitoring and profile updating methods

[1490] The server receives health data from smart devices such as smartwatches and fitness trackers. Received data includes the user's weight, blood pressure, heart rate, blood sugar level, etc. This received data is automatically updated to the user profile on the server. Python's Pandas and NumPy are used to manage and analyze the data.

[1491] For example, a user's new weight measurement on their smartwatch is sent to the server, and the data is updated in the user profile.

[1492] Recipe suggestion method

[1493] The server generates appropriate recipes based on the user's health information and the ingredients in the refrigerator. This recipe is generated using the machine learning model scikit-learn. The generated recipe is then sent to the user's smart device.

[1494] For example, the server selects a low-carb dinner recipe, compares the ingredients in the refrigerator to generate the recipe, and notifies the user.

[1495] Automatic ordering of ingredients that are in short supply

[1496] The server checks the refrigerator stock of ingredients needed for the proposed recipe and automatically orders any missing ingredients from the online store. The order process calls the API of the affiliated online store. If the order details need to be confirmed or changed, a notification is sent to the user's smart device, allowing them to make changes via the interface.

[1497] If the missing ingredients are identified as tomatoes and lettuce, the server uses the online store's ordering API to order these ingredients and notifies the user that "some of the ingredients required for the specified recipe have been ordered."

[1498] Prompt Sentence Examples

[1499] "Automatically order missing ingredients online based on your desired recipe."

[1500] As described above, the present invention provides a system that allows for efficient and healthy meal planning and significantly reduces the effort required for shopping.

[1501] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1502] Step 1: Obtaining ingredient information

[1503] The server acquires ingredient information using cameras, barcode scanners, or RFID readers built into smart shelves and smart refrigerators in physical stores. Specifically, the camera takes an image of the ingredient, and an image recognition algorithm (e.g., TensorFlow) is used to identify the type and quantity of the ingredient. The acquired information is recorded in a database. The input is an image of the ingredient or barcode / RFID label data, and the output is ingredient information including the ingredient name, quantity, and expiration date.

[1504] Step 2: Monitor your health information and update your profile

[1505] The server receives health data sent from the user's smart device (smartwatch or fitness tracker). The received data includes weight, blood pressure, heart rate, blood glucose level, etc. The server analyzes the data and updates the user profile. The input is the health data sent from the smart device, and the output is the updated user profile.

[1506] Step 3: Recipe suggestions

[1507] The server generates recipes based on the user's health information and information about ingredients in the refrigerator. A machine learning model (scikit-learn) is used for generation. The input is the user's health data and information about ingredients in the refrigerator, and the output is a recipe appropriate for the user's health condition. The generated recipe is then sent to the user's smart device.

[1508] Step 4: Identify missing ingredients

[1509] The server creates a list of ingredients needed based on the generated recipe and compares it with the refrigerator's inventory information. It identifies missing ingredients and compiles that information. The input is the list of ingredients needed for the recipe and the refrigerator's inventory information, and the output is the list of missing ingredients.

[1510] Step 5: Automated ordering of ingredients

[1511] The server automatically orders the identified missing ingredients from the online store. The ordering is done using the API of the partner online store. The input is the list of missing ingredients, and the output is order confirmation information. The user is notified of the automatic order via their smart device.

[1512] Step 6: Order confirmation and receipt notification

[1513] Users can confirm and change their order details through their smart device. After placing an order, they receive a notification about where to pick up the ingredients in the store. The input is order confirmation information, and the output is a notification to the user and instructions on where to pick up the ingredients.

[1514] Through the above processing flow, the present invention can support efficient and healthy meal planning in physical stores and provide an automatic ordering function for missing ingredients.

[1515] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1516] This invention is a system that automatically acquires information about ingredients in a refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system collects ingredient information using image recognition, barcode scanning, or RFID reading means installed in the refrigerator and records it in a database. It also receives the user's health and emotional data from a smart device and updates it in the user profile. This allows the system to compare the user's profile data with the information about ingredients in the refrigerator and generate appropriate recipes. Additionally, the system supports efficient and healthy meal planning by automatically ordering missing ingredients from an online store as needed.

[1517] A natural language description of the program's operation

[1518] Acquiring ingredient information

[1519] The server collects information about ingredients in the refrigerator using the refrigerator's built-in camera, barcode scanner, and RFID reader.

[1520] The server uses a camera to take pictures of the inside of the refrigerator periodically, or whenever new ingredients are added, and analyzes the images using image recognition algorithms. This allows it to identify the type and quantity of ingredients. It also uses a barcode scanner and RFID reader to read barcodes and RFID tags on food packages. The obtained information is recorded in a database as detailed information such as the name, quantity, and expiration date of the ingredients.

[1521] Health and emotional monitoring and profile updates

[1522] The user's health and emotional information is sent from the smart device to the server.

[1523] The server receives data such as a user's weight, blood pressure, heart rate, and blood sugar level collected daily by smart devices such as smartwatches and fitness trackers. It then uses an emotion engine to analyze the user's voice and facial expressions to generate emotion data. This data is updated based on the user profile. The server also analyzes the user's health and emotion information and notifies the user of any significant changes.

[1524] Recipe Suggestions

[1525] The server suggests appropriate recipes based on the user's profile data and information about ingredients in the refrigerator.

[1526] The server compares the user's health information and emotional state (e.g., stress level and mood) with the ingredients available in the refrigerator and generates recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences, health, and emotional state, and are notified to the user via their smart device.

[1527] Automatic ordering of missing ingredients

[1528] The server automatically orders missing ingredients online for the suggested recipe.

[1529] The server checks the ingredients required for the proposed recipe and compares them with the refrigerator's inventory to identify any missing ingredients. If any ingredients are missing, the server calls the API of a partner online store and automatically places an order for the ingredients. The user receives a confirmation of the automatic order via their smart device and is provided with an interface where they can review or change the order details.

[1530] Specific examples

[1531] Recipe suggestions based on emotional data

[1532] 1. Add ingredients and update information

[1533] When a new milk carton is added to the refrigerator, the server uses an RFID reader to read the information about the new milk carton (e.g., type, quantity, expiration date).

[1534] The server records the obtained information in a database.

[1535] The device then notifies the user that "New milk has been added."

[1536] 2. Collecting emotional data using an emotion engine

[1537] When a user uses a smart device, the emotion engine analyzes facial expressions and voice to identify the user's emotional state (e.g., stress level and mood).

[1538] The emotion data is sent to the server and updated in the user profile.

[1539] 3. Recipe suggestions based on emotional and health data

[1540] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective in reducing stress) based on the user's current health and emotional state.

[1541] The server checks the information about ingredients in the refrigerator and identifies the ingredients needed for the recipe.

[1542] Additionally, if necessary, any missing ingredients will be automatically ordered from the online store.

[1543] 4. Check recipe notifications and auto-orders

[1544] The generated recipe is notified to the user via their device, and a message is displayed saying, "A salad recipe that is effective in reducing stress has been suggested."

[1545] The user selects a salad recipe and the server automatically orders any missing ingredients.

[1546] The terminal will send the user an automated order confirmation and allow the user to review and modify the order.

[1547] This system allows users to efficiently manage ingredients in their refrigerator, and offers recipe suggestions and automatic ordering based on health and emotional information, thereby reducing the hassle of everyday shopping and meal planning and supporting a lifestyle that is healthy and meets emotional needs.

[1548] The processing flow will be explained below.

[1549] Step 1:

[1550] The server periodically takes pictures using a camera inside the refrigerator. The images are analyzed by an image recognition algorithm inside the server to identify the type and quantity of ingredients. The analysis results are then recorded in a database.

[1551] Step 2:

[1552] The server uses a barcode scanner or RFID reader in the refrigerator to read the barcode or RFID tag, and based on the information read, details such as the ingredient name, quantity, and expiration date are recorded in a database.

[1553] Step 3:

[1554] The terminal (smart device) provides an interface for users to manually add new ingredients using a barcode scanner, and the barcode information is sent to the server and recorded in a database.

[1555] Step 4:

[1556] The terminal (smart device) periodically collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.) and sends it to the server. The sent data is updated based on the user profile.

[1557] Step 5:

[1558] The server analyzes the received user health data and notifies the user if any significant changes are detected, and the health data is automatically updated in the user's profile.

[1559] Step 6:

[1560] The terminal (smart device) uses an emotion engine to analyze the user's facial expressions and voice to recognize emotions. The recognized emotion data is sent to the server and updated in the user profile.

[1561] Step 7:

[1562] The server compares the user's profile data with the ingredients in their refrigerator and generates recipes that take into account allergies, nutritional requirements, and emotional state. The generated recipes are prioritized based on the user's preferences, health status, and emotional state.

[1563] Step 8:

[1564] The terminal (smart device) receives the list of recipes generated from the server and notifies the user, who can then check the notification and select from the suggested recipes.

[1565] Step 9:

[1566] The server checks the ingredients required for the proposed recipe and identifies any missing ingredients by checking the inventory in the refrigerator. A list of missing ingredients is generated.

[1567] Step 10:

[1568] The server calls the API of the partner online store and automatically orders the identified missing ingredients. After the ordering process for the missing ingredients is completed, the order details are recorded in the database.

[1569] Step 11:

[1570] The terminal (smart device) will send the user a confirmation notification of the automated order, allowing the user to review the order details and make changes or cancellations as necessary.

[1571] Specific examples

[1572] Recipe suggestions based on emotional data

[1573] Step 1:

[1574] The server uses the refrigerator's camera to take pictures of the inside of the refrigerator.

[1575] The captured image is analyzed to identify information about ingredients such as apples and milk.

[1576] The analysis results are recorded in a database.

[1577] Step 2:

[1578] The server scans the barcode of the milk using a barcode scanner in the refrigerator.

[1579] Detailed information about the milk (type, quantity, expiration date) is recorded in a database from the barcode information.

[1580] Step 3:

[1581] When a user adds a new yogurt to the refrigerator, the barcode is read using the barcode scanner function of the terminal (smart device).

[1582] The information read is sent to a server, and the yogurt information is recorded in a database.

[1583] Step 4:

[1584] The terminal (smart device) periodically measures the user's weight, blood pressure, heart rate, blood sugar level, etc. and sends the data to the server.

[1585] The server updates the user profile with the received data.

[1586] Step 5:

[1587] The server analyzes health data and detects high blood pressure.

[1588] Based on this information, recipes are generated that take into account appropriate nutritional balance.

[1589] Step 6:

[1590] The emotion engine in the device (smart device) analyzes the user's facial expressions and voice and recognizes that the user is currently feeling stressed.

[1591] The emotion data is sent to the server and updated in the user profile.

[1592] Step 7:

[1593] The server compares the user's health and emotional data and suggests recipes that are effective in reducing stress.

[1594] The generated recipe is notified to the user as an "herbal tea recipe effective for stress relief."

[1595] Step 8:

[1596] The user sees the notification and selects the suggested recipe.

[1597] The server double-checks the ingredients needed for the recipe and compares them with the stock in the refrigerator.

[1598] Identify missing herbal tea ingredients.

[1599] Step 9:

[1600] The server calls the API of a partner online store and automatically orders the missing herbs for herbal tea.

[1601] The order process is completed and the details are recorded in a database.

[1602] Step 10:

[1603] The terminal (smart device) sends the user a confirmation of the automatic order.

[1604] The user can review the notification and approve or change the order.

[1605] In this way, a system that combines an emotion engine can suggest individual recipes based on the user's health condition and emotions, providing more effective support for daily stress management and health management.

[1606] Example 2

[1607] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1608] In today's lifestyle, managing the ingredients in the refrigerator and creating meal plans based on the user's health and emotional information is extremely time-consuming. Furthermore, checking and purchasing ingredients that are lacking also increases the time required. A system that solves these problems and supports an efficient and healthy lifestyle is needed.

[1609] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a photographing means, an identification means, a means for recording information about ingredients in the refrigerator, a means for receiving the user's health data from the smart device and updating it in the user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating recipes, a means for ordering missing ingredients online, a means for collecting the user's emotion data and updating the profile, a means for notifying the user of significant changes in the health data and emotion data, and a means for sending a confirmation notice of the automatic order. This automates a series of processes, from ingredient management to health information monitoring, recipe suggestions, and automatic ordering of missing ingredients, enabling the user to maintain an efficient and healthy lifestyle.

[1610] The "photography means" is a tool that is installed inside the refrigerator and is used to take pictures of ingredients.

[1611] "Identification means" refers to a method for analyzing the captured image and read information to identify the type and quantity of ingredients.

[1612] The "recording means" is a method for storing the collected information on ingredients in a database.

[1613] A "smart device" is a terminal that collects a user's health information and emotional data and transmits it to a server.

[1614] A "user profile" is a database entry that records and updates a user's health and emotional state.

[1615] The "matching means" is a method for comparing user profile data with information about ingredients in the refrigerator and generating an appropriate recipe based on that information.

[1616] A "generator" is a method for creating a recipe based on the collated data.

[1617] An "online ordering method" is a method for automatically purchasing missing ingredients required for a suggested recipe.

[1618] "Emotional data" is data on the emotional state of a user obtained by analyzing their voice and facial expressions.

[1619] "Means for notifying significant changes" refers to a method for notifying a user when there is a significant change in the user's health data or emotional data.

[1620] "Confirmation notification means" refers to a method for informing users of the contents of an online order after it has been completed and prompting them to confirm or change the order.

[1621] This invention relates to a system that automatically acquires information about ingredients in a refrigerator based on the user's health and emotional state, suggests appropriate recipes, and orders missing ingredients online.

[1622] Specifically, this system is implemented using the following hardware and software.

[1623] Hardware Configuration

[1624] 1. Photographing means inside the refrigerator: A camera placed inside the refrigerator is a device that takes pictures of ingredients.

[1625] 2. Identification means: A device that uses a barcode scanner or RFID reader to identify the type and quantity of ingredients.

[1626] 3. Smart devices: These are devices such as smartwatches and fitness trackers used by users that collect health and emotional data and send it to a server.

[1627] Software Configuration

[1628] 1. Image recognition algorithm: Software that runs on the server and analyzes the captured image to identify the type and quantity of ingredients.

[1629] 2. Database management system: A system for storing and managing collected food ingredient information, health data, and emotional data.

[1630] 3. Recipe generation engine: Software that generates appropriate recipes based on the user's health and emotional state.

[1631] 4. Order management system: A system that identifies shortages of ingredients and automatically places orders by calling the API of affiliated online stores.

[1632] 5. Notification Engine: This software notifies users with recipe suggestions and automatic ordering confirmations.

[1633] Example of operation

[1634] Example 1: Collecting and recording information about ingredients in the refrigerator

[1635] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[1636] The server uses an image recognition algorithm to analyze the captured image and identify the type and quantity of ingredients.

[1637] For newly added ingredients, the server uses a barcode scanner or RFID reader in the refrigerator to read the identifying information.

[1638] The server records the obtained information in a database, saving the names, quantities, expiration dates, etc. of ingredients.

[1639] Example 2: Monitoring health and emotional information and updating profiles

[1640] Users wear smart devices and collect health data (e.g., weight, blood pressure, heart rate, blood glucose levels) on a daily basis.

[1641] Smart devices periodically send the collected data to a server.

[1642] The server uses an emotion engine to analyze the user's voice and facial expressions and generate emotion data.

[1643] The collected health and emotional data is updated by the server into a user profile.

[1644] For example, if a user's stress level increases, the server uses a notification engine to notify the user of important fluctuations.

[1645] Example 3: Recipe suggestions

[1646] The server matches the user profile data with the ingredients in the refrigerator and generates appropriate recipes based on the user's health and emotional state.

[1647] For example, if a user's stress level is high, the app will generate recipes using ingredients that are effective in reducing stress.

[1648] The server notifies the smart device of the generated recipe so that the user can check it.

[1649] Example 4: Automatic ordering and notification of missing ingredients

[1650] The server checks the ingredients needed for the generated recipe against the inventory information in the refrigerator.

[1651] When a missing ingredient is identified, the server calls the API of a partner online store to automatically place the order.

[1652] After the order is completed, the server uses a notification engine to send a confirmation to the user, who can then review the order details through their smart device and make any necessary changes.

[1653] This system allows users to efficiently carry out all processes, from managing ingredients in the refrigerator to monitoring health information, collecting emotional data, suggesting recipes, and automatically ordering ingredients when they are running low.

[1654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1655] Step 1: Collect images of the inside of the refrigerator

[1656] The server activates a camera inside the refrigerator to take pictures of the ingredients.

[1657] Input: An image of the inside of a refrigerator taken by a camera.

[1658] Output: Captured image data.

[1659] Specific operation: The server activates the camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added.

[1660] Step 2: Analyze with image recognition algorithm

[1661] The server uses image recognition algorithms to analyze the images sent to it.

[1662] Input: Captured image data.

[1663] Output: Data on the type and quantity of ingredients analyzed.

[1664] What it does: The server analyzes the received image and runs an image recognition algorithm to identify the type and quantity of ingredients.

[1665] Step 3: Barcode scanning or RFID reading

[1666] The server activates the barcode scanner and RFID reader inside the refrigerator.

[1667] Input: Barcode or RFID tag information of the newly added ingredient.

[1668] Output: Detailed information about the ingredients read (e.g. name, quantity, expiration date).

[1669] Specific operation: The server activates a barcode scanner or RFID reader to read the information about the new ingredients.

[1670] Step 4: Record in the database

[1671] The server records the analysis results and the barcode / RFID tag information read in a database.

[1672] Input: Parsed ingredient information, read ingredient information.

[1673] Output: Detailed information such as ingredient name, quantity, expiration date, etc. recorded in the database.

[1674] Specific operation: The server stores the obtained ingredient information in a database and manages the ingredients.

[1675] Step 5: Collecting health data

[1676] The smart device that serves as the terminal collects the user's health data (weight, blood pressure, heart rate, blood sugar level, etc.).

[1677] Input: User's everyday health data.

[1678] Output: Collected health data.

[1679] How it works: Smart devices such as smartwatches and fitness trackers continuously collect biometric data from their users.

[1680] Step 6: Collect emotion data

[1681] The smart device analyzes the user's voice and facial expressions.

[1682] Input: User voice and facial expression data.

[1683] Output: Parsed emotion data.

[1684] What it does: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state.

[1685] Step 7: Update your user profile

[1686] The server updates the user profile based on the health and emotional data.

[1687] Input: Collected health and emotion data.

[1688] Output: Updated user profile.

[1689] Specific operation: The server updates the profile with newly acquired data, maintaining the latest health and emotional state.

[1690] Step 8: Notification of significant changes

[1691] The server analyzes the user's health and emotional information and notifies them if there are any significant changes.

[1692] Input: Continuously updated health and emotion data.

[1693] Output: Notification message of significant fluctuations.

[1694] Specific operation: The server will send an alert to the user if there is a fluctuation that exceeds a certain standard.

[1695] Step 9: Matching profile data with ingredient information

[1696] The server matches the user profile with the information about ingredients in the refrigerator.

[1697] Input: User profile data, refrigerator food information.

[1698] Output: Matching results.

[1699] Specific operation: The server compares the profile data with the ingredient information and obtains the data necessary to suggest appropriate recipes.

[1700] Step 10: Recipe Generation

[1701] The server generates an appropriate recipe based on the matching results.

[1702] Input: Matching result.

[1703] Output: The generated recipe.

[1704] How it works: The server uses a generative AI model to create a recipe tailored to the user.

[1705] Step 11: Prioritize and notify

[1706] The server prioritizes the generated recipes based on the user's preferences and health status and notifies them.

[1707] Input: The generated recipe.

[1708] Output: Prioritized recipe notifications.

[1709] Specific operation: The server evaluates the importance of the recipe and notifies the user via the smart device.

[1710] Step 12: Identify missing ingredients

[1711] The server checks the ingredients needed for the proposed recipe against the inventory information in the refrigerator.

[1712] Inputs: Suggested recipes, refrigerator inventory.

[1713] Output: A list of missing ingredients.

[1714] What happens: The server compares the ingredients required for the recipe with the current inventory and identifies any missing ingredients.

[1715] Step 13: Order online

[1716] The server will then process the missing ingredients for ordering online.

[1717] Input: Missing ingredient list.

[1718] Output: Order information from the online store.

[1719] Specific operation: The server calls the API of the affiliated online store and automatically orders the necessary ingredients.

[1720] Step 14: Order confirmation

[1721] The server will send a confirmation to the user after the order is completed.

[1722] Input: Order checkout completion information.

[1723] Output: Order confirmation notification.

[1724] Specific operation: Once the order is completed, the server notifies the user of the order details via the smart device, allowing them to confirm or change the order.

[1725] (Application example 2)

[1726] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1727] Conventional refrigerator management systems only managed food inventory and suggested recipes, but were unable to provide suggestions that took into account the user's health or emotional state. They also lacked the ability to automatically order ingredients when they were running low or display recipes in real time, making them inconvenient for users. Furthermore, the recipe options were limited, leaving a need for a more flexible system that could adapt to changes in the user's health and emotions.

[1728] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means arranged in the refrigerator, a barcode scanning means or an RFID reading means, a means for recording information about ingredients in the refrigerator in a database, a means for receiving the user's health data from the smart device and updating it in a user profile, a means for comparing the user's profile data with the information about ingredients in the refrigerator and generating an appropriate recipe, a means for automatically ordering missing ingredients from an online store, a means for receiving audio or video from the smart device and determining the user's emotional state, and a means for displaying the recipe on the smart glasses. This enables the system to propose appropriate recipes taking into account the user's health and emotional state, automatically order missing ingredients, and display the recipe on the smart glasses in real time.

[1729] The "image recognition means" is a means for analyzing an image captured using a camera and identifying an object contained in the image.

[1730] "Barcode scanning means" refers to means for reading a barcode and obtaining the information written thereon.

[1731] "RFID reading means" refers to a means for reading data from an RFID tag using wireless communication.

[1732] The "means for recording in a database" refers to a means for recording in a database for managing and storing the acquired data.

[1733] The "means for updating the user profile" refers to a means for updating the user profile with the latest information based on the health data and emotional data received from the user.

[1734] The "means for generating an appropriate recipe" is a means for creating an optimal recipe based on the user's profile data and information about ingredients in the refrigerator.

[1735] "Means for automatic ordering from online store" refers to a means for automatically ordering missing ingredients from an online store.

[1736] The "means for displaying a recipe on smart glasses" refers to a means for displaying the generated recipe on the display of smart glasses.

[1737] "Means for receiving audio or video and determining the emotional state of a user" refers to means for analyzing audio or video from a smart device and identifying and determining the emotional state of a user.

[1738] This system automatically acquires information about ingredients in the refrigerator, suggests recipes based on the user's health and emotional state, and automatically orders missing ingredients online. This system works in conjunction with smart glasses and smart devices to support efficient and healthy meal planning.

[1739] System configuration

[1740] Hardware Configuration

[1741] 1. Refrigerator

[1742] It is equipped with a built-in camera, barcode scanning means, and RFID reading means, which are used to collect information about ingredients in the refrigerator.

[1743] 2. Smart Glasses

[1744] Equipped with a display, camera, and microphone, it monitors the user's health and emotional state while displaying generated recipes.

[1745] 3. Smart Devices

[1746] Devices such as smartwatches that collect health data such as weight, blood pressure, heart rate, and blood sugar levels.

[1747] Software Configuration

[1748] 1. Image recognition algorithms (e.g., OpenCV)

[1749] The system analyzes images taken by a camera inside the refrigerator to determine the type, quantity, and expiration date of ingredients.

[1750] 2. Sentiment analysis engine (e.g., Emotion API)

[1751] It analyzes data from the smart glasses' camera or microphone to identify the user's emotional state.

[1752] 3. Data Management System

[1753] A database that manages ingredient information and user profile data.

[1754] 4. Recipe Generation Algorithm

[1755] An algorithm that generates appropriate recipes based on the user's health and emotional data.

[1756] 5. Online Store API (e.g., API for online retailers)

[1757] An API for automatically ordering missing ingredients.

[1758] System Operation

[1759] Acquiring ingredient information

[1760] The server uses the refrigerator's built-in camera, barcode scanner, and RFID reader to take pictures of the refrigerator's interior and collect information about ingredients. It uses image recognition algorithms to identify the type and quantity of ingredients and records this information in a database. The information is updated periodically or when new ingredients are added.

[1761] Health and emotional monitoring and profile updates

[1762] The server receives user health and emotional data from smart glasses and smart devices. Data such as weight, blood pressure, heart rate, and blood sugar levels are collected from smartwatches and other devices, and an emotional analysis engine is used to analyze voice and facial expressions to generate emotional data. This information is updated based on the user profile.

[1763] Recipe Suggestions

[1764] The server generates optimal recipes based on the user profile data and the ingredients in the refrigerator. The generated recipes are prioritized and notified to the user through the smart glasses display. The user can then view the recipes on the smart glasses.

[1765] Automatic ordering of missing ingredients

[1766] The server checks the ingredients required for the proposed recipe and automatically orders any missing ingredients from the online store. The order is processed through the online store API, and a confirmation of the automatic order is sent to the user via the smart glasses. The user can then confirm and change the order details.

[1767] Specific examples

[1768] Recipe suggestions based on emotional data

[1769] 1. Add ingredients and update information

[1770] When a user adds a new ingredient (e.g., a carton of milk) to the refrigerator, the built-in RFID reader reads the new ingredient information, and the server records it in the database. The terminal notifies the user that "a new ingredient has been added."

[1771] 2. Collecting emotional data using an emotion engine

[1772] When a user wears the smart glasses, the emotion engine analyzes their facial expressions and voice to identify their emotional state, and the emotion data is sent to the server and updated in the user profile.

[1773] 3. Recipe suggestions based on emotional and health data

[1774] The server analyzes the user's profile data and generates appropriate recipes (e.g., recipes containing ingredients effective for stress reduction) based on the user's health and emotional state. If necessary, missing ingredients are automatically ordered from the online store.

[1775] 4. Check recipe notifications and auto-orders

[1776] The generated recipe is sent to the user through the smart glasses. The user selects the recipe and the server automatically orders any missing ingredients. The order details are displayed on the smart glasses display, and the user can check and change them.

[1777] Prompt Sentence Examples

[1778] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[1779] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1780] Step 1:

[1781] The server uses a camera to take pictures of the inside of the refrigerator periodically or when new ingredients are added. This obtains image data of the inside of the refrigerator. The server then uses an image recognition algorithm (e.g., OpenCV) to identify the type, quantity, and expiration date of each ingredient in the image. This information is recorded in a database. The input is the image data of the inside of the refrigerator, and the output is ingredient information (type, quantity, expiration date).

[1782] Step 2:

[1783] The server receives the user's health data from the smart device (e.g., smart watch). This data includes weight, blood pressure, heart rate, blood glucose level, etc. The server updates this data to the user profile. It also collects emotion data from the user's voice and video using the camera and microphone of the smart glasses, and analyzes it using an emotion analysis engine (e.g., Emotion API). The input is the audio and video data from the smart device and smart glasses, and the output is the updated health data and emotion data.

[1784] Step 3:

[1785] The server compares the collected health and emotional data with the information about ingredients in the refrigerator. It then runs a recipe generation algorithm to generate optimal recipes based on the user's health and emotional state. The generated recipes are prioritized based on the user's preferences and current health and emotional state. The inputs are the health data, emotional data, and ingredient information, and the output is the generated recipe.

[1786] Step 4:

[1787] The device displays the generated recipe on the smart glasses display, allowing the user to check and select the recipe on the smart glasses. The input is the generated recipe, and the output is the recipe display on the smart glasses.

[1788] Step 5:

[1789] The server checks whether the refrigerator contains all the necessary ingredients based on the selected recipe. If any ingredients are missing, it identifies them and calls the online store API to automatically place an order. The order details are displayed on the smart glasses and can be confirmed and changed by the user. The input is the information about ingredients in the refrigerator and the selected recipe, and the output is the ordered ingredients and their confirmation information.

[1790] Specific examples of processing

[1791] In step 1, when new milk is added to the refrigerator, an image of the inside of the refrigerator is taken with a camera, and the type, quantity, and expiration date of the milk are identified using OpenCV, and this is recorded in the database.

[1792] In step 2, the server receives weight and heart rate data from the smartwatch, and sends voice data from the smartglasses to an emotion analysis engine to determine the user's emotional state and update the profile.

[1793] In step 3, a recipe generation algorithm uses this data to generate and prioritize "stress-reducing salads."

[1794] In step 4, the recipe is displayed on the smart glasses display for the user to review.

[1795] In step 5, if it is determined that the refrigerator is low on necessary vegetables, an automatic order is placed using the online store API, and the order details are displayed on the smart glass.

[1796] Prompt Sentence Examples

[1797] "Based on the information about ingredients in the refrigerator, please generate recipes suited to the user's health and emotional state and display them on the smart glasses. Please also provide a function to automatically order ingredients online if they are missing."

[1798] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1799] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1800] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1801] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1802] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1803] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1804] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1805] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotio...

Claims

1. an image recognition means disposed in the refrigerator; a barcode scanning means or an RFID reading means; A means for recording information about ingredients in the refrigerator in a database; means for receiving the user's health data from the smart device and updating the user profile; A means for matching user profile data with information about ingredients in the refrigerator to generate appropriate recipes; A way to automatically order missing ingredients from the online store, A system including:

2. 10. The system of claim 1, further comprising means for capturing images of ingredients with a camera in the refrigerator and classifying the ingredients using an image recognition algorithm.

3. The system of claim 1 , further comprising means for automatically updating the user profile with the user's weight, blood pressure, heart rate, and blood glucose data periodically transmitted from the smart device.

Images