System

Abstract

A system is provided.SOLUTION: A system including a means for installing a web camera in a refrigerator, a pantry, and a cupboard and capturing images of the inside of the refrigerator, the pantry, and the cupboard, an artificial intelligence means for analyzing the captured images and generating a food ingredient list, a cooking menu based on the food ingredient list, a means for monitoring opening and closing states of doors of the refrigerator, the pantry, and the cupboard, and a means for detecting an abnormality based on the opening and closing states and notifying a user terminal of the abnormality.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, manually managing ingredients in the refrigerator or pantry is extremely time-consuming and laborious. Furthermore, users who want to cook at home face the challenge of coming up with recipes that effectively utilize the ingredients in their refrigerator. Furthermore, remote monitoring of elderly relatives requires relying on expensive professional services, which increases costs. A system that can simultaneously solve these problems is needed. [Means for solving the problem]

[0005] The present invention provides a system that includes a webcam installed in a refrigerator, pantry, or cupboard to capture images of the interior of these items; an artificial intelligence (AI) system that analyzes the captured images to generate an ingredient list; an AI system that suggests cooking menus based on the ingredient list; a system that monitors whether the refrigerator, pantry, or cupboard doors are open or closed; and a system that detects abnormalities based on the open / closed status and notifies a user terminal. The system also includes a system that stores the generated ingredient list in a database and suggests additional ingredients to purchase based on the stored ingredient list. The system also includes a system that receives images transmitted from the webcam and stores them in a buffer; an AI system that analyzes the stored images; and a system that transmits the analysis results to the user terminal. This system allows users to efficiently manage ingredients in their refrigerators, receive appropriate cooking menu suggestions, and easily monitor elderly people remotely.

[0006] A "webcam" is a camera device for capturing and transmitting video over a network.

[0007] A "refrigerator" is a home or commercial device for keeping food and beverages cool.

[0008] A "pantry" is a storage space or room for storing food and kitchen utensils.

[0009] A "cupboard" is a piece of furniture used to store dishes and cooking utensils.

[0010] An "image" is digital data representing visual information, especially a photograph or video frame taken by a camera.

[0011] "Artificial intelligence means" are software or algorithms that analyze data, learn, and perform specific tasks.

[0012] The "ingredient list" is a list of ingredients present in the refrigerator or pantry, derived from the analyzed image.

[0013] A "cooking menu" is a specific recipe and steps for cooking that are generated based on an ingredient list.

[0014] "Door opening / closing status" is data indicating the opening and closing of the doors of the refrigerator, pantry, and cupboard.

[0015] "User terminal" means a device used by a user to receive, display, and operate information, such as a computer or mobile device.

[0016] An "anomaly" is a condition or occurrence that deviates from the normal pattern or norm.

[0017] A "database" is a system or software for efficiently storing, managing, and retrieving data.

[0018] A "buffer" is a memory area for temporarily storing data.

[0019] "Trend analysis" is a method for analyzing past and present data patterns to predict future trends.

[0020] "Push notification" is a communication method for sending messages and information to devices in real time. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, and generates and manages ingredient lists. It also has cooking menu suggestions and monitoring functions. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[0043] System Configuration

[0044] 1. Webcam

[0045] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[0046] 2. Image capture and transmission

[0047] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[0048] 3. Image Analysis

[0049] Server: Uses artificial intelligence (such as GPT-4 (registered trademark)) to analyze the received image. The purpose of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[0050] 4. Creating and managing ingredient lists

[0051] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[0052] Cooking menu suggestions

[0053] 5. Cooking menu generation

[0054] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[0055] 6. Suggesting inadequate ingredients

[0056] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[0057] Monitoring function

[0058] 7. Monitoring door opening and closing status

[0059] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0060] 8. Trend Analysis and Anomaly Detection

[0061] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[0062] Specific examples

[0063] Ingredients management

[0064] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[0065] Device (camera): Take a picture at 8:00 and send it to the server.

[0066] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[0067] Cooking menu suggestions

[0068] User: Checks grocery list on device before lunch.

[0069] Server: Based on the ingredients list, suggests menu items such as "Tomato soup" and "Lettuce salad." The user selects "Tomato soup."

[0070] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[0071] Monitoring function implementation

[0072] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0073] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[0074] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[0075] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to keep an eye on elderly people.

[0076] The processing flow will be explained below.

[0077] Program processing flow

[0078] Imaging and analysis

[0079] Step 1:

[0080] The device (camera) takes pictures of the inside of the refrigerator and pantry at regular intervals (e.g., every hour).

[0081] Step 2:

[0082] The captured image is sent from the device (camera) to the server.

[0083] Step 3:

[0084] The server receives the image and temporarily stores it in a buffer.

[0085] Image analysis and ingredient list generation

[0086] Step 4:

[0087] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[0088] Step 5:

[0089] The server identifies ingredients in the image and generates an ingredient list.

[0090] Step 6:

[0091] The generated ingredient list is stored in a database on the server.

[0092] Step 7:

[0093] The server transmits the stored ingredient list to the user's terminal.

[0094] Cooking menu suggestions

[0095] Step 8:

[0096] The user checks the ingredient list on the device.

[0097] Step 9:

[0098] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[0099] Step 10:

[0100] The server transmits the generated cooking menu to the user's terminal.

[0101] Step 11:

[0102] The user checks the presented cooking menu on the terminal and selects one.

[0103] Suggestions for ingredients that are in short supply

[0104] Step 12:

[0105] The server will list any missing ingredients based on the menu selected.

[0106] Step 13:

[0107] The server sends a list of ingredients that are in short supply to the user's terminal.

[0108] Step 14:

[0109] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[0110] Monitoring function

[0111] Step 15:

[0112] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[0113] Step 16:

[0114] The terminal (camera) periodically sends door opening / closing information to the server.

[0115] Step 17:

[0116] The server accumulates the opening and closing data received and performs trend analysis.

[0117] Step 18:

[0118] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[0119] Step 19:

[0120] If the server detects an abnormality, it generates warning information.

[0121] Step 20:

[0122] The server generates a push notification of the warning information to the user's device.

[0123] Step 21:

[0124] The user receives a push notification to check the status.

[0125] Through the above processing steps, the present invention makes it possible to efficiently manage ingredients and monitor elderly people all at once.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] The need for food management and home cooking assistance for modern families and the elderly is increasing, but there are few systems that can accurately grasp food inventory status and efficiently suggest cooking menus.Furthermore, there are almost no systems that have a function to monitor the elderly, so there is a need for a low-cost, easy-to-use system that combines both functions.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes means for installing an image capture device in a storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device, and means for detecting abnormalities based on the opening and closing status and notifying a user terminal. This allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people.

[0131] An "image capture device" is a device used to capture images of the inside of a refrigerator, pantry, or cupboard, and generally refers to a webcam or digital camera.

[0132] "Storage equipment" refers to equipment for storing food and ingredients, and specifically includes refrigerators, pantries, and cupboards.

[0133] "Artificial intelligence means" refers to technologies for image analysis and data processing, particularly machine learning algorithms and generative AI models.

[0134] A "database" is a digital system for organizing and storing information such as ingredient lists, and includes SQL databases and NoSQL databases.

[0135] A "user terminal" is a device operated by system users, and includes smartphones, PCs, and tablets.

[0136] "Image analysis" refers to the process of analyzing images captured by an image capture device using software or artificial intelligence means.

[0137] The "ingredient list" is a list of ingredients in the refrigerator or pantry, generated based on the analyzed image information.

[0138] "Cooking menu" refers to recipes and cooking methods suggested based on an ingredient list.

[0139] "Opening and closing status" refers to the state and frequency of opening and closing the doors of the refrigerator, pantry, and cupboard.

[0140] "Anomaly detection" is the process of identifying unusual patterns or behaviors and recognizing problems.

[0141] "Push notification" is a function that detects anomalies and sends important information to user devices in real time.

[0142] The present invention is a system that uses an image capture device to manage ingredients stored in a storage device and suggests appropriate cooking menus, and also has a monitoring function for elderly people. Specific embodiments of this system will be described below.

[0143] System Configuration and Operation

[0144] 1. Installation of imaging device

[0145] A user places an image capture device (such as a webcam) in a storage device such as a refrigerator, pantry, or cupboard, and adjusts the angle and position of the device to capture the entire food item in the storage device.

[0146] 2. Take and send images

[0147] The terminal (image capture device) takes pictures of the storage device at a set time. For example, it can be set to take pictures of the inside of the refrigerator every morning at 8:00. The images are connected to the Internet via a Wi-Fi module and sent to a server.

[0148] 3. Image Analysis

[0149] The server receives the image and performs image analysis. It uses artificial intelligence (e.g., GPT-4, TENSORFLOW (registered trademark), etc.) to identify and recognize the ingredients in the image. For example, if "milk, tomato, lettuce" is recognized, it is added to the ingredients list.

[0150] 4. Creating and managing ingredient lists

[0151] The server generates an ingredient list based on the recognized ingredient information and stores it in a database. Duplicate ingredients are automatically omitted. This allows the user device to always receive the latest ingredient list.

[0152] 5. Cooking menu suggestions

[0153] The server generates a cooking menu based on the saved ingredient list. It inputs prompts to a generative AI model (e.g., GPT-4) to suggest appropriate recipes. An example of a prompt is "Please suggest a recommended cooking menu based on the current ingredient list."

[0154] 6. Suggestions for missing ingredients

[0155] If the server detects that a suggested menu item is missing, it lists the ingredients that need to be purchased and notifies the user. For example, if a recipe for "tomato soup" requires "chicken soup base" but is in short supply, the server notifies the user terminal, "You are running low on chicken soup base. Please buy more."

[0156] 7. Monitoring door opening and closing status

[0157] The terminal (image capture device) is attached with a sensor on the door of the storage device to monitor the door opening and closing status. The opening and closing data is periodically recorded and sent to the server.

[0158] 8. Trend Analysis and Anomaly Detection

[0159] The server performs trend analysis based on the received door opening / closing data. It learns normal opening / closing frequencies and patterns, and detects abnormal patterns or cases where the door has not been opened / closed for a certain period of time. For example, if the door is normally opened / closed four times a day, but there has been no opening / closing for more than 24 hours, this is detected as an abnormality. If this abnormality is detected, a push notification is sent to the user's device saying, "The refrigerator has not been opened / closed for more than 24 hours. Please check."

[0160] This allows users to efficiently manage ingredients and receive appropriate cooking menus. It also makes it possible to simultaneously monitor elderly people, improving safety.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1: Installing the imaging device

[0163] The user places an image capture device, such as a webcam, on a refrigerator, pantry, or cupboard at an appropriate angle and position so that the entire contents of the storage device are captured. For example, the camera can be attached to the top of the refrigerator and point downward to capture the interior.

[0164] Step 2: Take and send an image

[0165] The terminal (image capture device) automatically captures images in the storage device at a set time. A timer setting (e.g., every morning at 8:00) is given as input. The captured images are connected to the Internet via Wi-Fi and sent to a server. As output, a JPEG image file is generated.

[0166] Step 3: Image analysis

[0167] The server receives the transmitted image and performs image analysis. As input, a JPEG image file is provided. Specifically, artificial intelligence tools (e.g., GPT-4 or TensorFlow) are used to identify and recognize the ingredients in the image. An image processing algorithm is applied to process the data, and as output, a list of ingredient information (e.g., "Milk, Tomato, Lettuce") is generated.

[0168] Step 4: Generate and manage your ingredient list

[0169] The server generates an ingredient list based on the analyzed ingredient information and stores it in a database. The generated ingredient information list is given as input. Data calculations include removing duplicate items and reorganizing the list. An updated ingredient list is generated as output and synchronized with the user's device.

[0170] Step 5: Cooking menu suggestions

[0171] The server generates a cooking menu based on the saved ingredient list. The latest ingredient list is given as input. A prompt is input to the generative AI model (e.g., GPT-4) to suggest an appropriate recipe. For example, the prompt is "Please suggest a recommended cooking menu based on the current ingredient list." As output, multiple cooking menus (e.g., "Tomato soup," "Lettuce salad") are generated and sent to the user device.

[0172] Step 6: Suggest missing ingredients

[0173] The server detects if any ingredients required for the proposed menu are missing. As input, the proposed menu and a list of ingredients are given. As data processing, the ingredients required for the menu are compared with the currently available ingredients. As output, a list of missing ingredients (e.g., "chicken soup base") is generated and notified to the user's terminal.

[0174] Step 7: Monitor the door opening and closing status

[0175] The terminal (image capture device) uses a sensor installed on the door of the storage device to monitor the door's opening and closing status. As input, a timestamp of the door opening and closing is recorded. As data processing, a log of the opening and closing status is generated and sent to a server via the Internet. As output, opening and closing log data is generated.

[0176] Step 8: Trend analysis and anomaly detection

[0177] The server performs trend analysis based on the received opening / closing log data. The door opening / closing log data is given as input. Data calculations learn the normal opening / closing frequency and patterns, and detect abnormal patterns. For example, if the door is normally opened and closed four times a day, but there is no opening / closing for more than 24 hours, this is considered an abnormality. The output is the result of the abnormality detection (e.g., "The refrigerator has not been opened or closed for more than 24 hours. Please check"), and a push notification is sent to the user's device.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] Conventional systems for managing the contents of refrigerators, pantries, and cupboards have functions such as managing ingredients, suggesting cooking menus, and monitoring opening and closing status, but there are few ways to provide these functions in a visually easy-to-understand manner for elderly people. It is also difficult to immediately visually check information about shortages of ingredients or monitoring functions. Therefore, efficiently managing ingredients and safety at home for elderly people has been a challenge.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the open / close status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the open / close status and notifying a user terminal, means for displaying the ingredient list and the open / close status of the doors on a head-mounted display, and means for visually notifying the user via the information notified and suggested on the head-mounted display. This allows even elderly people to visually manage ingredients, monitor the open / close status, and check safety.

[0183] A "webcam" is a camera device that can capture and transmit video over the Internet.

[0184] "Artificial intelligence means" refers to AI algorithms or software that run on a computer and learn, infer, and analyze data input to it.

[0185] A "grocery list" refers to a list of the types and quantities of ingredients in your refrigerator, pantry, or cupboard.

[0186] "Cooking menu" refers to a list of recipes for dishes that can be made using specific ingredients.

[0187] The "open / close status monitoring means" is a device or system that monitors the opening and closing of refrigerator, pantry, and cupboard doors and acquires the data.

[0188] An "anomaly detection means" is a system or algorithm that determines if an unusual pattern or problem has occurred based on acquired data.

[0189] A "user terminal" is an electronic device used by a user to receive, check, and operate information, and includes smartphones, tablets, and personal computers.

[0190] A "head-mounted display" is a display device that a user wears on their head to display visual information.

[0191] "Notification means" refers to a function or device for conveying specific information to a user, and includes push notifications, audio alarms, etc.

[0192] A "buffer" refers to a memory area that temporarily stores data, enabling high-speed reading and writing of data.

[0193] A "database" is a system or software for systematically storing data and for efficient searching and retrieval.

[0194] A "visual notification means" is a device or system for displaying information so that it can be perceived visually by a user.

[0195] This invention is a system that uses a webcam to capture images of the inside of refrigerators, pantries, and cupboards, analyzes the images, and generates and manages an ingredient list, providing visual information to users, including the elderly. In particular, by combining image analysis using artificial intelligence with a visual notification function using a head-mounted display, it provides a mechanism that allows users to quickly check the status of ingredients and any abnormalities.

[0196] System Configuration

[0197] 1. Webcam

[0198] User: Place a webcam on your refrigerator, pantry, or cupboard, and angle and position it to capture the entire area.

[0199] 2. Image capture and transmission

[0200] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to the server, allowing you to keep up to date with the latest food information.

[0201] 3. Image Analysis

[0202] Server: Uses a generative AI model (such as GPT-4) to analyze the submitted image, recognize ingredients in the image, and generate an ingredient list.

[0203] 4. Creating and managing ingredient lists

[0204] Server: Generates an ingredient list based on the analysis results and stores it in a database. The generated list can be viewed on the user's head-mounted display or other devices.

[0205] 5. Cooking menu suggestions

[0206] Server: Based on the stored ingredient list, the server uses a generative AI model to suggest cooking menus. The suggested menus are sent to the user's head-mounted display, where they can be selected and used.

[0207] 6. Suggestions for missing ingredients

[0208] Server: If any ingredients required for the proposed menu are missing, the server generates a list of those ingredients and notifies the head-mounted display and user device, suggesting that the missing ingredients be purchased.

[0209] Monitoring function

[0210] 7. Monitoring door opening and closing status

[0211] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0212] 8. Trend Analysis and Anomaly Detection

[0213] Server: Performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, it notifies the head-mounted display and other user devices via push notification.

[0214] Specific examples

[0215] Ingredients management

[0216] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[0217] Device (camera): Take a picture at 8:00 and send it to the server.

[0218] Server: Analyzes the image and generates a list of ingredients (e.g., "milk, tomato, lettuce"). The list is stored in a database and provided to the head-mounted display and other user devices.

[0219] Cooking menu suggestions

[0220] User: Checks the grocery list on the head-mounted display before lunch.

[0221] Server: Based on the ingredients list, suggests a cooking menu (e.g., "tomato soup" or "lettuce salad"). The user selects "tomato soup."

[0222] Server: Detects that the "chicken soup mix" required for the "tomato soup" is missing, and notifies the head-mounted display and other user devices of this fact.

[0223] Monitoring function implementation

[0224] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0225] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[0226] Server: Detects this anomaly and sends a push notification to the head-mounted display and other user devices. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[0227] Prompt Sentence Examples

[0228] "Analyze an image of the inside of a refrigerator and generate a list of ingredients including milk, tomatoes, and lettuce. Also, tell me what ingredients are missing to make tomato soup."

[0229] This system allows users, including the elderly, to efficiently manage and cook ingredients while ensuring safety.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] Webcam installation and image capture settings

[0233] Users can install web cameras on refrigerators, pantries, and cupboards, adjust the angle and position to capture the entire area, and set the timing of the camera captures.

[0234] Input: Physical location of the camera and timing settings.

[0235] Output: After the setup is complete, the camera will take an image at the specified time.

[0236] Step 2:

[0237] Taking and sending images

[0238] The device (camera) takes pictures of the inside of the refrigerator or pantry at a set time and sends them to a server via the Internet.

[0239] Input: The image captured by the camera.

[0240] Data processing: Generation of image files and transmission over the Internet.

[0241] Output: The image file sent to the server.

[0242] Step 3:

[0243] Image analysis

[0244] The server uses a generative AI model (e.g., GPT-4) to analyze the received image, recognize ingredients in the image, and generate an ingredient list.

[0245] Input: The submitted image file.

[0246] Data Computing: Image recognition and ingredient list generation using generative AI models.

[0247] Output: A list of ingredients (e.g. "Milk, Tomato, Lettuce").

[0248] Step 4:

[0249] Save your grocery list

[0250] The server stores the generated ingredient list in a database.

[0251] Input: Ingredient list.

[0252] Data processing: Writing list data to a database.

[0253] Output: Ingredient list stored in database.

[0254] Step 5:

[0255] Cooking menu suggestions

[0256] The server uses a generative AI model to suggest cooking menus based on the stored ingredient list and transmits the results to the head-mounted display and other user devices.

[0257] Input: Saved ingredient list.

[0258] Data Computing: Menu suggestions using generative AI models and results transmission.

[0259] Output: A cooking menu displayed on the user's device (e.g., "Tomato soup" or "Lettuce salad").

[0260] Step 6:

[0261] Suggestions for ingredients that are in short supply

[0262] If any ingredients required for the proposed menu are missing, the server generates a list of the missing ingredients and notifies the head-mounted display and other user terminals of the list.

[0263] Input: Proposed cooking menu and current ingredients list.

[0264] Data calculation: Determine shortages of ingredients and generate notification lists.

[0265] Output: Notification of missing ingredients (e.g. "You are running low on chicken soup base").

[0266] Step 7:

[0267] Door opening / closing status monitoring

[0268] The terminal (camera) monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0269] Input: Signal from door open / close sensor.

[0270] Data processing: Collection and transmission of door opening and closing data.

[0271] Output: Door open / close data sent to the server.

[0272] Step 8:

[0273] Trend analysis and anomaly detection

[0274] The server performs trend analysis based on the received door opening and closing data to detect abnormal patterns or cases where the door has not been opened or closed within a certain period of time. If an abnormality is detected, a push notification is sent to the user's head-mounted display and other devices.

[0275] Input: Door opening and closing data.

[0276] Data arithmetic: Applying trend analysis and anomaly detection algorithms.

[0277] Output: Push notification if an abnormality is detected (e.g. "The refrigerator has not been opened or closed for more than 24 hours. Please check it.").

[0278] Step 9:

[0279] Visual Notifications

[0280] The server sends various information (ingredient list, cooking menu, missing ingredients, abnormality notification) to the head-mounted display and notifies the user visually.

[0281] Input: Generated information (ingredient list, cooking menu, missing ingredients, abnormality notification).

[0282] Data processing: information format conversion and transmission procedures.

[0283] Output: Information visually displayed on a head-mounted display.

[0284] Through the above processing steps, the user can efficiently manage, cook, and check the safety of ingredients.

[0285] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0286] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, creates and manages ingredient lists, and also has cooking menu suggestions and monitoring functions, and even recognizes the user's emotions to suggest appropriate menus. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[0287] System Configuration

[0288] 1. Webcam

[0289] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[0290] 2. Image capture and transmission

[0291] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[0292] 3. Image Analysis

[0293] Server: Uses artificial intelligence to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[0294] 4. Creating and managing ingredient lists

[0295] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[0296] Cooking menu suggestions

[0297] 5. Cooking menu generation

[0298] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[0299] 6. Suggesting inadequate ingredients

[0300] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[0301] Monitoring function

[0302] 7. Monitoring door opening and closing status

[0303] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0304] 8. Trend Analysis and Anomaly Detection

[0305] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[0306] Emotion recognition function

[0307] 9. User Emotion Recognition

[0308] Device: Uses a camera and microphone to detect the user's facial expressions and voice, and sends the data to the emotion engine.

[0309] Emotion engine: Analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[0310] 10. Emotion-Based Suggestions

[0311] Server: Based on data from the emotion engine, it generates a cooking menu according to the user's emotional state.

[0312] Server: The server sends the generated menu to the user's device and makes suggestions that match their emotional state. For example, if the user is feeling stressed, a menu with a relaxing effect will be selected.

[0313] 11. Recording and analyzing emotional data

[0314] Server: Periodically records user emotion data and performs trend analysis.

[0315] Server: Accumulates data to provide users with menu preferences and appropriate suggestions based on their past emotional patterns.

[0316] Specific examples

[0317] Ingredient management and emotion recognition

[0318] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[0319] Device (camera): Take a picture at 8:00 and send it to the server.

[0320] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[0321] Device: The camera and microphone also detect the user's facial expressions and voice data, and send them to the emotion engine.

[0322] Cooking menu suggestions

[0323] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[0324] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[0325] Emotion-based suggestions

[0326] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[0327] Monitoring function implementation

[0328] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0329] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[0330] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[0331] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[0332] The processing flow will be explained below.

[0333] Program processing flow

[0334] Imaging and analysis

[0335] Step 1:

[0336] The device (camera) takes pictures of the refrigerator and pantry at regular intervals each day (e.g., 8:00 a.m.).

[0337] Step 2:

[0338] The captured image is sent from the device (camera) to the server.

[0339] Step 3:

[0340] The server receives the image and temporarily stores it in a buffer.

[0341] Step 4:

[0342] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[0343] Step 5:

[0344] The server identifies ingredients in the image and generates an ingredient list.

[0345] Step 6:

[0346] The generated ingredient list is stored in a database on the server.

[0347] Step 7:

[0348] The server transmits the stored ingredient list to the user's terminal.

[0349] Cooking menu suggestions

[0350] Step 8:

[0351] The user checks the ingredient list on the device.

[0352] Step 9:

[0353] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[0354] Step 10:

[0355] The server transmits the generated cooking menu to the user's terminal.

[0356] Step 11:

[0357] The user checks the presented cooking menu on the terminal and selects one.

[0358] Suggestions for ingredients that are in short supply

[0359] Step 12:

[0360] The server will list any missing ingredients based on the menu selected.

[0361] Step 13:

[0362] The server sends a list of ingredients that are in short supply to the user's terminal.

[0363] Step 14:

[0364] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[0365] Monitoring function

[0366] Step 15:

[0367] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[0368] Step 16:

[0369] The terminal (camera) periodically sends door opening / closing information to the server.

[0370] Step 17:

[0371] The server accumulates the opening and closing data received and performs trend analysis.

[0372] Step 18:

[0373] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[0374] Step 19:

[0375] If the server detects an abnormality, it generates warning information.

[0376] Step 20:

[0377] The server generates a push notification of the warning information to the user's device.

[0378] Step 21:

[0379] The user receives a push notification to check the status.

[0380] Emotion recognition function

[0381] Step 22:

[0382] The device uses a camera and microphone to detect the user's facial expressions and voice and sends the data to the emotion engine.

[0383] Step 23:

[0384] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, happiness).

[0385] Step 24:

[0386] The server generates a cooking menu according to the user's emotional state based on the data from the emotion engine.

[0387] Step 25:

[0388] The server sends the generated menu to the user's device and makes suggestions that match the user's emotional state.

[0389] Step 26:

[0390] The server periodically records the user's emotional data and performs trend analysis.

[0391] Step 27:

[0392] The server accumulates data based on the user's past emotional patterns to provide appropriate suggestions and menus that the user prefers.

[0393] Step 28:

[0394] The device then performs appropriate actions based on the user's emotional state (e.g., suggesting favorite recipes or sending encouraging messages).

[0395] Through the above processing steps, the present invention can efficiently manage ingredients and monitor elderly people all at once, and can also suggest menus that match the user's mood through emotion recognition.

[0396] Example 2

[0397] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Currently, there are systems on the market that manage ingredients in refrigerators and pantries and suggest cooking menus, but these do not detect anomalies based on the user's emotions or the opening and closing status of storage device doors. Therefore, there is a need for systems that can efficiently and low-costly support home cooking and monitor the elderly. In particular, there is a lack of systems that can suggest personalized menus based on the status of ingredients and the user's emotional state.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0400] In this invention, the server includes means for installing a webcam in the storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images and generating an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device door, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, means for detecting the user's facial expression and voice and recognizing emotions, and means for proposing cooking menus based on the emotion data. This enables flexible and efficient ingredient management, cooking menu suggestions, and abnormality detection based on the user's emotions and storage status.

[0401] "Storage equipment" means a device for storing food or goods, examples of which include refrigerators, pantries, and cupboards.

[0402] A "webcam" is a camera device that captures and transmits images and videos over the Internet.

[0403] The "food list" is a list that lists the types and quantities of food extracted from the analyzed image data.

[0404] "Artificial intelligence means" refers to algorithms and machine learning models used for image analysis, data processing, menu suggestions, etc.

[0405] "Door open / close status" is data indicating the state of the door of the storage device, whether it is open or closed.

[0406] "Anomaly detection" is the process of recognizing behavior that deviates from normal opening and closing patterns and reporting the problem.

[0407] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0408] "Means for detecting facial expressions and voice" refers to hardware and software for recognizing the user's facial movements and voice and collecting them as data.

[0409] "Means for recognizing emotions" refers to an algorithm or system that analyzes detected facial and voice data to determine the emotional state of the user.

[0410] "Trend analysis" is the process of analyzing collected data to identify trends and patterns over a period of time.

[0411] This invention is a system that manages ingredients stored in storage devices, suggests cooking menus, monitors elderly people, and suggests menus based on their emotions. The system combines webcams, image analysis AI, and emotion recognition technology to provide an efficient and personalized experience.

[0412] Hardware and Software Configuration

[0413] 1. Webcam

[0414] Users place a webcam, such as the Logitech C920, in their refrigerator, pantry, or cupboard, capturing a full view of the contents of the storage device and recording daily changes in the ingredients.

[0415] 2. Image capture and transmission

[0416] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. The software used here is custom firmware, which provides the scheduled shooting and automatic sending functions.

[0417] 3. Image Analysis

[0418] The server uses image recognition artificial intelligence tools such as TensorFlow to analyze the received image, recognize ingredients in the image (e.g., milk, tomato, lettuce), and generate an ingredient list based on that information.

[0419] 4. Creating and managing ingredient lists

[0420] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL (registered trademark). The generated ingredient list is then provided to the user's smartphone or tablet.

[0421] 5. Recipe generation and suggestions

[0422] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, recipes for "tomato soup" and "lettuce salad" using "milk," "tomato," and "lettuce" are generated and sent to the user's device.

[0423] 6. Suggestion of necessary ingredients

[0424] If the server is running low on ingredients for a proposed menu, it generates a list of those ingredients and notifies the user. For example, it recognizes that the "chicken soup base" needed for "tomato soup" is missing, and notifies the user of that information.

[0425] 7. Monitoring door opening and closing status

[0426] The device (camera) monitors whether the storage device door is open or closed, and periodically sends the data to the server. The timing of the door opening and closing is recorded and compiled by the server.

[0427] 8. Trend Analysis and Anomaly Detection

[0428] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, a push notification is sent to the user's device.

[0429] 9. User Emotion Recognition

[0430] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[0431] 10. Emotion-Based Suggestions

[0432] The server generates a cooking menu based on the data from the emotion engine according to the user's emotional state. For example, if the user is feeling depressed, it will suggest a "herbal tea recipe" that has a relaxing effect. The generated menu is sent to the user's device.

[0433] 11. Recording and analyzing emotional data

[0434] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to suggest menus that the user prefers and make appropriate suggestions.

[0435] Examples and prompts

[0436] As a specific example, the following procedure is carried out.

[0437] Ingredient management and emotion recognition

[0438] User: Places a Logitech C920 webcam inside the refrigerator and sets it to automatically record at 8:00 AM every day.

[0439] Example prompt: "Set up your webcam and take a picture of the inside of your refrigerator every day at 8 AM."

[0440] Image capture and transmission

[0441] Device (camera): Take a picture at 8:00 and send it to the server.

[0442] Image analysis

[0443] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[0444] Emotion Recognition and Suggestions

[0445] Device (camera and microphone): Detects the user's facial expressions and voice data and sends them to the emotion engine.

[0446] Cooking menu suggestions

[0447] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[0448] Sample prompt: "We analyzed the image of your refrigerator and detected milk, tomatoes, and lettuce. We've added them to your current ingredient list."

[0449] Suggestion of necessary ingredients

[0450] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[0451] Emotion-based suggestions

[0452] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[0453] Example prompt: "The user's facial expression appears depressed. Suggest a relaxing herbal tea recipe."

[0454] Monitoring function implementation

[0455] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0456] Trend analysis and anomaly detection

[0457] Server: Conducts trend analysis and detects that the door normally opens and closes several times a day, but has not opened or closed for more than 24 hours. This anomaly is detected and a push notification is sent to the user's device.

[0458] Sample prompt: "The refrigerator hasn't been opened or closed for over 24 hours. Please check."

[0459] This system allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[0460] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0461] Processing Steps

[0462] Step 1: Setting up your webcam

[0463] Users place a webcam, such as the Logitech C920, on a refrigerator, pantry, or cupboard and adjust the angle and position to capture the entire interior. For example, place the webcam on top of the refrigerator and adjust it to a position that allows a full view of the interior.

[0464] Input: Webcam installation instructions, refrigerator location

[0465] Output: A webcam placed in a suitable position, ready to record.

[0466] Step 2: Take and send an image

[0467] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. Custom firmware is used to implement the scheduled shooting and automatic sending functions.

[0468] Input: Set shooting schedule, webcam location information

[0469] Output: Captured image data, image data transmission to server

[0470] Step 3: Image analysis

[0471] The server analyzes the received image using image recognition AI such as TensorFlow, which identifies ingredients in the image (e.g., milk, tomato, lettuce), and generates an ingredient list based on that information.

[0472] Input: Captured image data

[0473] Output: Ingredient list data as the analysis result

[0474] Step 4: Generate and manage your ingredient list

[0475] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[0476] Input: Ingredient list data as a result of image analysis

[0477] Output: Ingredient list stored in the database, provided to the user's device

[0478] Step 5: Recipe generation and suggestions

[0479] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, a recipe for "tomato soup" or "lettuce salad" using "milk," "tomato," and "lettuce" is generated and sent to the user's device.

[0480] Input: List of ingredients in the database

[0481] Output: Generated recipes, suggestions to user terminal

[0482] Step 6: Suggesting ingredients

[0483] If the server is missing any ingredients needed for the proposed menu, it generates a list of those ingredients and notifies the user terminal. For example, it recognizes that "chicken soup base" needed for "tomato soup" is missing and notifies the user of that information.

[0484] Input: Cooking menu, current ingredients list

[0485] Output: List of missing ingredients, notification to user device

[0486] Step 7: Monitor the door opening and closing status

[0487] The device (camera) monitors the opening and closing status of the storage device door and periodically transmits the data to the server. The timing of the door opening and closing is recorded and transmitted to the server.

[0488] Input: Door open / close status, timestamp

[0489] Output: Open / close data sent to the server

[0490] Step 8: Trend analysis and anomaly detection

[0491] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device.

[0492] Input: Door opening / closing data

[0493] Output: Anomaly detection results, notification to user device

[0494] Step 9: Recognizing User Emotions

[0495] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[0496] Input: facial expression data, voice data

[0497] Output: Emotion recognition result

[0498] Step 10: Emotion-Based Suggestions

[0499] The server generates a cooking menu based on the user's emotional state based on the data from the emotion engine. For example, if the user is feeling depressed, it will suggest a relaxing "herbal tea recipe." The generated menu is sent to the user's device.

[0500] Input: Emotion recognition results, list of ingredients

[0501] Output: Emotion-based cooking menu, sent to user device

[0502] Step 11: Record and analyze emotional data

[0503] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to provide the user with menus they prefer and make appropriate suggestions.

[0504] Input: Periodically collected emotion data

[0505] Output: Sentiment trend analysis results, personalized menu suggestion data

[0506] (Application example 2)

[0507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0508] In the past, managing ingredients in the refrigerator or pantry and suggesting cooking menus was often done manually, making it difficult for users to efficiently manage ingredients and select cooking menus. Furthermore, there were issues with systems that were unable to provide services that monitored elderly people or responded to the user's emotions. Furthermore, even in brick-and-mortar stores, there was a demand for automated services based on customers' emotions and food inventory management, but these systems were difficult to implement.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0510] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, emotion recognition means for recognizing the user's facial expressions and voice and analyzing the data, and means for proposing cooking menus based on the emotion data obtained from the emotion recognition means. This enables automation of ingredient management in brick-and-mortar stores and the proposing of personalized cooking menus based on the user's emotions.

[0511] A "webcam" is a camera device that captures digital images or video and transmits them in real time over a computer or network.

[0512] The "ingredient list" is a list showing ingredients and items present in the refrigerator, pantry, and cupboard.

[0513] An "artificial intelligence means" is a system or software that uses computer algorithms to analyze data and mimic human intellectual activity.

[0514] "Door opening / closing status" is information indicating the status of the opening and closing of the refrigerator, pantry, and cupboard doors.

[0515] An "anomaly" is a condition or action that deviates from a normal pattern or expected behavior.

[0516] "User terminal" means a computer device operated by a user to receive and display information.

[0517] "Emotion recognition means" refers to a system or software that analyzes a user's facial expressions and voice to identify their emotions.

[0518] "Emotion data" is data that represents the emotional state of the user analyzed from facial expressions and voice.

[0519] A "personalized cooking menu" is a cooking menu suggested to suit the user's individual preferences and condition.

[0520] This system uses a webcam to capture images of food storage areas such as kitchens and stockrooms in brick-and-mortar stores, analyzes the images to generate and manage ingredient lists, and suggests cooking menus based on customer sentiment. This system automates food management in brick-and-mortar stores and improves the customer experience.

[0521] System Configuration

[0522] 1. Webcam installation

[0523] User (store clerk): Install a webcam in the kitchen or stockroom of the physical store and adjust the angle and position to capture the entire scene.

[0524] 2. Image capture and transmission

[0525] Terminal (camera): Takes images at regular intervals (e.g., every two hours) and sends them to a server via the Internet.

[0526] 3. Image Analysis

[0527] Server: Analyzes the received image using AI (e.g., AWS (registered trademark) Rekognition or Google (registered trademark) Cloud Vision), recognizes the ingredients in the image, and generates an ingredient list.

[0528] 4. Creating and managing ingredient lists

[0529] Server: Stores the generated ingredient list in a database and updates and manages it as needed.

[0530] Cooking menu suggestions

[0531] 5. Cooking menu generation

[0532] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to and displayed on the smartphones or smart glasses of customers and store staff.

[0533] 6. Suggestions for missing ingredients

[0534] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[0535] Monitoring function

[0536] 7. Monitoring door opening and closing status

[0537] Terminal (camera): Monitors whether refrigerators and pantry doors are open or closed in physical stores, and periodically sends this data to the server.

[0538] 8. Trend Analysis and Anomaly Detection

[0539] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[0540] Emotion recognition function

[0541] 9. User Emotion Recognition

[0542] Device: Uses a camera and microphone to detect the customer's facial expressions and voice, and sends the data to an emotion engine (e.g., Microsoft® Azure® Cognitive Services Face API).

[0543] Server: Analyzes the data received from the emotion engine and recognizes the emotional state of the customer (e.g., joy, anger, sadness, or happiness).

[0544] 10. Emotion-Based Suggestions

[0545] Server: Generates cooking menus based on emotion data and sends them to the customers' and staff's devices. For example, if a customer is feeling stressed, the server suggests menus that will have a relaxing effect.

[0546] 11. Recording and analyzing emotional data

[0547] Server: Periodically records user emotion data and performs trend analysis.

[0548] Server: Accumulates data to determine customers' preferred menu items and make appropriate suggestions based on past emotional patterns.

[0549] Specific examples

[0550] Ingredient management and emotion recognition

[0551] User (Store Staff): Set up a webcam in the kitchen to capture images of the food storage area at 8:00 AM and 2:00 PM every day.

[0552] Terminal (camera): Takes a picture at the set time and sends it to the server.

[0553] Server: Analyzes the image, generates an ingredient list, and stores it in a database.

[0554] Terminal: The camera and microphone also detect the customer's facial expressions and voice data, which are then sent to the emotion engine.

[0555] Cooking menu suggestions

[0556] Server: Based on the ingredients list and the analysis results of the emotion engine, the server proposes multiple menu items and sends them to the user's device. The customer selects a menu item.

[0557] Suggestions for ingredients that are in short supply

[0558] Server: If any ingredients required for the selected menu are missing, it generates a list of them and notifies the user terminal of that fact.

[0559] Emotion-based suggestions

[0560] Server: If a customer's expression is deemed depressed, the server will suggest, for example, a relaxing "herbal tea recipe."

[0561] Monitoring function implementation

[0562] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0563] Server: Conducts trend analysis and detects when the refrigerator has not been opened or closed for 24 hours or more. If an abnormality is detected, a push notification is sent to the user's device. The push notification displays the message, "The refrigerator has not been opened or closed for 24 hours or more. Please check."

[0564] Prompt Sentence Examples

[0565] By inputting the following prompts into the generative AI model, it is possible to suggest cooking menus that correspond to the customer's emotions.

[0566] Example prompt sentence:

[0567] Based on the user's emotional data, please create and suggest a menu that will have a relaxing effect. If the emotional state is "stressed" or "depressed," please explain what menu should be suggested and why.

[0568] With this configuration, the system can automate food ingredient management in physical stores and propose personalized cooking menus based on customers' emotions, providing an efficient and satisfying customer experience.

[0569] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0570] Step 1:

[0571] Camera installation and image capture

[0572] The user (store clerk) installs a webcam in the kitchen or stockroom of a physical store and adjusts the angle and position to capture the entire scene. The webcam takes images at regular intervals (e.g., every two hours) and generates image data. The input is the camera's installation position and shooting time settings, and the output is the captured image data.

[0573] Step 2:

[0574] Image transmission

[0575] The terminal (webcam) sends the captured image to a server via the Internet. The input is the captured image data, and the output is the image data sent to the server.

[0576] Step 3:

[0577] Image analysis

[0578] The server analyzes the received image data using an AI model (e.g., AWS Rekognition or Google Cloud Vision) to recognize the ingredients present in the image. It generates an ingredient list based on the analysis results. The input is the transmitted image data, and the output is the generated ingredient list.

[0579] Step 4:

[0580] Save Ingredient List

[0581] The server stores the generated ingredient list in a database. The input is the generated ingredient list, and the output is the ingredient list stored in the database.

[0582] Step 5:

[0583] Cooking menu suggestions

[0584] The server uses AI to generate cooking menus based on the ingredient list stored in the database and sends them to the user's device. The input is the stored ingredient list, and the output is the proposed cooking menu.

[0585] Step 6:

[0586] Notification of insufficient ingredients

[0587] If any ingredients required for the proposed cooking menu are missing, the server generates a list of those ingredients and notifies the user of the missing ingredients. The input is the proposed cooking menu and the list of ingredients, and the output is the list of missing ingredients and a notification of that list.

[0588] Step 7:

[0589] Door opening / closing status monitoring

[0590] The terminal (web camera) monitors the opening and closing status of the refrigerator and pantry doors and periodically transmits the data to the server. The input is the door opening and closing behavior, and the output is the opening and closing status data.

[0591] Step 8:

[0592] Anomaly detection

[0593] The server performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device. The input is the opening / closing status data, and the output is the abnormality detection and notification.

[0594] Step 9:

[0595] emotion recognition

[0596] The device uses a camera and microphone to detect the facial expressions and voices of customers and sends the data to an emotion engine (e.g., Microsoft Azure Cognitive Services Face API). The input is facial expression and voice data, and the output is emotion data.

[0597] Step 10:

[0598] Emotional Data Analysis

[0599] The server analyzes the data received from the emotion engine and recognizes the emotional state of the customer. The input is the emotion data, and the output is the analysis result.

[0600] Step 11:

[0601] Emotion-based menu suggestions

[0602] The server then proposes cooking menus to the customers' and store staff's devices based on the results of the emotion data analysis. For example, if a customer is feeling stressed, it will propose a menu with a relaxing effect. The input is the emotion data analysis results, and the output is a cooking menu proposal based on the customer's emotion.

[0603] Step 12:

[0604] Emotional data recording and analysis

[0605] The server periodically records users' emotional data and performs trend analysis. Based on past emotional patterns, the server accumulates data to help customers choose menu items they like and make appropriate suggestions. The input is emotional data, and the output is the recorded emotional data and the analysis results.

[0606] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0607] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0608] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0609] [Second embodiment]

[0610] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0611] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0612] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0613] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0614] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0615] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0616] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0617] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0618] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0619] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0620] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0621] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0622] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, and generates and manages ingredient lists. It also has cooking menu suggestions and monitoring functions. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[0623] System Configuration

[0624] 1. Webcam

[0625] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[0626] 2. Image capture and transmission

[0627] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[0628] 3. Image Analysis

[0629] Server: Uses artificial intelligence (such as GPT-4) to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[0630] 4. Creating and managing ingredient lists

[0631] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[0632] Cooking menu suggestions

[0633] 5. Cooking menu generation

[0634] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[0635] 6. Suggesting inadequate ingredients

[0636] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[0637] Monitoring function

[0638] 7. Monitoring door opening and closing status

[0639] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0640] 8. Trend Analysis and Anomaly Detection

[0641] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[0642] Specific examples

[0643] Ingredients management

[0644] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[0645] Device (camera): Take a picture at 8:00 and send it to the server.

[0646] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[0647] Cooking menu suggestions

[0648] User: Checks grocery list on device before lunch.

[0649] Server: Based on the ingredients list, suggests menu items such as "Tomato soup" and "Lettuce salad." The user selects "Tomato soup."

[0650] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[0651] Monitoring function implementation

[0652] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0653] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[0654] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[0655] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to keep an eye on elderly people.

[0656] The processing flow will be explained below.

[0657] Program processing flow

[0658] Imaging and analysis

[0659] Step 1:

[0660] The device (camera) takes pictures of the inside of the refrigerator and pantry at regular intervals (e.g., every hour).

[0661] Step 2:

[0662] The captured image is sent from the device (camera) to the server.

[0663] Step 3:

[0664] The server receives the image and temporarily stores it in a buffer.

[0665] Image analysis and ingredient list generation

[0666] Step 4:

[0667] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[0668] Step 5:

[0669] The server identifies ingredients in the image and generates an ingredient list.

[0670] Step 6:

[0671] The generated ingredient list is stored in a database on the server.

[0672] Step 7:

[0673] The server transmits the stored ingredient list to the user's terminal.

[0674] Cooking menu suggestions

[0675] Step 8:

[0676] The user checks the ingredient list on the device.

[0677] Step 9:

[0678] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[0679] Step 10:

[0680] The server transmits the generated cooking menu to the user's terminal.

[0681] Step 11:

[0682] The user checks the presented cooking menu on the terminal and selects one.

[0683] Suggestions for ingredients that are in short supply

[0684] Step 12:

[0685] The server will list any missing ingredients based on the menu selected.

[0686] Step 13:

[0687] The server sends a list of ingredients that are in short supply to the user's terminal.

[0688] Step 14:

[0689] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[0690] Monitoring function

[0691] Step 15:

[0692] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[0693] Step 16:

[0694] The terminal (camera) periodically sends door opening / closing information to the server.

[0695] Step 17:

[0696] The server accumulates the opening and closing data received and performs trend analysis.

[0697] Step 18:

[0698] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[0699] Step 19:

[0700] If the server detects an abnormality, it generates warning information.

[0701] Step 20:

[0702] The server generates a push notification of the warning information to the user's device.

[0703] Step 21:

[0704] The user receives a push notification to check the status.

[0705] Through the above processing steps, the present invention makes it possible to efficiently manage ingredients and monitor elderly people all at once.

[0706] Example 1

[0707] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] The need for food management and home cooking assistance for modern families and the elderly is increasing, but there are few systems that can accurately grasp food inventory status and efficiently suggest cooking menus.Furthermore, there are almost no systems that have a function to monitor the elderly, so there is a need for a low-cost, easy-to-use system that combines both functions.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0710] In this invention, the server includes means for installing an image capture device in a storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device, and means for detecting abnormalities based on the opening and closing status and notifying a user terminal. This allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people.

[0711] An "image capture device" is a device used to capture images of the inside of a refrigerator, pantry, or cupboard, and generally refers to a webcam or digital camera.

[0712] "Storage equipment" refers to equipment for storing food and ingredients, and specifically includes refrigerators, pantries, and cupboards.

[0713] "Artificial intelligence means" refers to technologies for image analysis and data processing, particularly machine learning algorithms and generative AI models.

[0714] A "database" is a digital system for organizing and storing information such as ingredient lists, and includes SQL databases and NoSQL databases.

[0715] A "user terminal" is a device operated by system users, and includes smartphones, PCs, and tablets.

[0716] "Image analysis" refers to the process of analyzing images captured by an image capture device using software or artificial intelligence means.

[0717] The "ingredient list" is a list of ingredients in the refrigerator or pantry, generated based on the analyzed image information.

[0718] "Cooking menu" refers to recipes and cooking methods suggested based on an ingredient list.

[0719] "Opening and closing status" refers to the state and frequency of opening and closing the doors of the refrigerator, pantry, and cupboard.

[0720] "Anomaly detection" is the process of identifying unusual patterns or behaviors and recognizing problems.

[0721] "Push notification" is a function that detects anomalies and sends important information to user devices in real time.

[0722] The present invention is a system that uses an image capture device to manage ingredients stored in a storage device and suggests appropriate cooking menus, and also has a monitoring function for elderly people. Specific embodiments of this system will be described below.

[0723] System Configuration and Operation

[0724] 1. Installation of imaging device

[0725] A user places an image capture device (such as a webcam) in a storage device such as a refrigerator, pantry, or cupboard, and adjusts the angle and position of the device to capture the entire food item in the storage device.

[0726] 2. Take and send images

[0727] The terminal (image capture device) takes pictures of the storage device at a set time. For example, it can be set to take pictures of the inside of the refrigerator every morning at 8:00. The images are connected to the Internet via a Wi-Fi module and sent to a server.

[0728] 3. Image Analysis

[0729] The server receives the image and performs image analysis. It uses artificial intelligence (e.g., GPT-4, TensorFlow, etc.) to identify and recognize ingredients in the image. For example, if "milk, tomato, lettuce" is recognized, it is added to the ingredients list.

[0730] 4. Creating and managing ingredient lists

[0731] The server generates an ingredient list based on the recognized ingredient information and stores it in a database. Duplicate ingredients are automatically omitted. This allows the user device to always receive the latest ingredient list.

[0732] 5. Cooking menu suggestions

[0733] The server generates a cooking menu based on the saved ingredient list. It inputs prompts to a generative AI model (e.g., GPT-4) to suggest appropriate recipes. An example of a prompt is "Please suggest a recommended cooking menu based on the current ingredient list."

[0734] 6. Suggestions for missing ingredients

[0735] If the server detects that a suggested menu item is missing, it lists the ingredients that need to be purchased and notifies the user. For example, if a recipe for "tomato soup" requires "chicken soup base" but is in short supply, the server notifies the user terminal, "You are running low on chicken soup base. Please buy more."

[0736] 7. Monitoring door opening and closing status

[0737] The terminal (image capture device) is attached with a sensor on the door of the storage device to monitor the door opening and closing status. The opening and closing data is periodically recorded and sent to the server.

[0738] 8. Trend Analysis and Anomaly Detection

[0739] The server performs trend analysis based on the received door opening / closing data. It learns normal opening / closing frequencies and patterns, and detects abnormal patterns or cases where the door has not been opened / closed for a certain period of time. For example, if the door is normally opened / closed four times a day, but there has been no opening / closing for more than 24 hours, this is detected as an abnormality. If this abnormality is detected, a push notification is sent to the user's device saying, "The refrigerator has not been opened / closed for more than 24 hours. Please check."

[0740] This allows users to efficiently manage ingredients and receive appropriate cooking menus. It also makes it possible to simultaneously monitor elderly people, improving safety.

[0741] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0742] Step 1: Installing the imaging device

[0743] The user places an image capture device, such as a webcam, on a refrigerator, pantry, or cupboard at an appropriate angle and position so that the entire contents of the storage device are captured. For example, the camera can be attached to the top of the refrigerator and point downward to capture the interior.

[0744] Step 2: Take and send an image

[0745] The terminal (image capture device) automatically captures images in the storage device at a set time. A timer setting (e.g., every morning at 8:00) is given as input. The captured images are connected to the Internet via Wi-Fi and sent to a server. As output, a JPEG image file is generated.

[0746] Step 3: Image analysis

[0747] The server receives the transmitted image and performs image analysis. As input, a JPEG image file is provided. Specifically, artificial intelligence tools (e.g., GPT-4 or TensorFlow) are used to identify and recognize the ingredients in the image. An image processing algorithm is applied to process the data, and as output, a list of ingredient information (e.g., "Milk, Tomato, Lettuce") is generated.

[0748] Step 4: Generate and manage your ingredient list

[0749] The server generates an ingredient list based on the analyzed ingredient information and stores it in a database. The generated ingredient information list is given as input. Data calculations include removing duplicate items and reorganizing the list. An updated ingredient list is generated as output and synchronized with the user's device.

[0750] Step 5: Cooking menu suggestions

[0751] The server generates a cooking menu based on the saved ingredient list. The latest ingredient list is given as input. A prompt is input to the generative AI model (e.g., GPT-4) to suggest an appropriate recipe. For example, the prompt is "Please suggest a recommended cooking menu based on the current ingredient list." As output, multiple cooking menus (e.g., "Tomato soup," "Lettuce salad") are generated and sent to the user device.

[0752] Step 6: Suggest missing ingredients

[0753] The server detects if any ingredients required for the proposed menu are missing. As input, the proposed menu and a list of ingredients are given. As data processing, the ingredients required for the menu are compared with the currently available ingredients. As output, a list of missing ingredients (e.g., "chicken soup base") is generated and notified to the user's terminal.

[0754] Step 7: Monitor the door opening and closing status

[0755] The terminal (image capture device) uses a sensor installed on the door of the storage device to monitor the door's opening and closing status. As input, a timestamp of the door opening and closing is recorded. As data processing, a log of the opening and closing status is generated and sent to a server via the Internet. As output, opening and closing log data is generated.

[0756] Step 8: Trend analysis and anomaly detection

[0757] The server performs trend analysis based on the received opening / closing log data. The door opening / closing log data is given as input. Data calculations learn the normal opening / closing frequency and patterns, and detect abnormal patterns. For example, if the door is normally opened and closed four times a day, but there is no opening / closing for more than 24 hours, this is considered an abnormality. The output is the result of the abnormality detection (e.g., "The refrigerator has not been opened or closed for more than 24 hours. Please check"), and a push notification is sent to the user's device.

[0758] (Application example 1)

[0759] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0760] Conventional systems for managing the contents of refrigerators, pantries, and cupboards have functions such as managing ingredients, suggesting cooking menus, and monitoring opening and closing status, but there are few ways to provide these functions in a visually easy-to-understand manner for elderly people. It is also difficult to immediately visually check information about shortages of ingredients or monitoring functions. Therefore, efficiently managing ingredients and safety at home for elderly people has been a challenge.

[0761] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0762] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the open / close status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the open / close status and notifying a user terminal, means for displaying the ingredient list and the open / close status of the doors on a head-mounted display, and means for visually notifying the user via the information notified and suggested on the head-mounted display. This allows even elderly people to visually manage ingredients, monitor the open / close status, and check safety.

[0763] A "webcam" is a camera device that can capture and transmit video over the Internet.

[0764] "Artificial intelligence means" refers to AI algorithms or software that run on a computer and learn, infer, and analyze data input to it.

[0765] A "grocery list" refers to a list of the types and quantities of ingredients in your refrigerator, pantry, or cupboard.

[0766] "Cooking menu" refers to a list of recipes for dishes that can be made using specific ingredients.

[0767] The "open / close status monitoring means" is a device or system that monitors the opening and closing of refrigerator, pantry, and cupboard doors and acquires the data.

[0768] An "anomaly detection means" is a system or algorithm that determines if an unusual pattern or problem has occurred based on acquired data.

[0769] A "user terminal" is an electronic device used by a user to receive, check, and operate information, and includes smartphones, tablets, and personal computers.

[0770] A "head-mounted display" is a display device that a user wears on their head to display visual information.

[0771] "Notification means" refers to a function or device for conveying specific information to a user, and includes push notifications, audio alarms, etc.

[0772] A "buffer" refers to a memory area that temporarily stores data, enabling high-speed reading and writing of data.

[0773] A "database" is a system or software for systematically storing data and for efficient searching and retrieval.

[0774] A "visual notification means" is a device or system for displaying information so that it can be perceived visually by a user.

[0775] This invention is a system that uses a webcam to capture images of the inside of refrigerators, pantries, and cupboards, analyzes the images, and generates and manages an ingredient list, providing visual information to users, including the elderly. In particular, by combining image analysis using artificial intelligence with a visual notification function using a head-mounted display, it provides a mechanism that allows users to quickly check the status of ingredients and any abnormalities.

[0776] System Configuration

[0777] 1. Webcam

[0778] User: Place a webcam on your refrigerator, pantry, or cupboard, and angle and position it to capture the entire area.

[0779] 2. Image capture and transmission

[0780] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to the server, allowing you to keep up to date with the latest food information.

[0781] 3. Image Analysis

[0782] Server: Uses a generative AI model (such as GPT-4) to analyze the submitted image, recognize ingredients in the image, and generate an ingredient list.

[0783] 4. Creating and managing ingredient lists

[0784] Server: Generates an ingredient list based on the analysis results and stores it in a database. The generated list can be viewed on the user's head-mounted display or other devices.

[0785] 5. Cooking menu suggestions

[0786] Server: Based on the stored ingredient list, the server uses a generative AI model to suggest cooking menus. The suggested menus are sent to the user's head-mounted display, where they can be selected and used.

[0787] 6. Suggestions for missing ingredients

[0788] Server: If any ingredients required for the proposed menu are missing, the server generates a list of those ingredients and notifies the head-mounted display and user device, suggesting that the missing ingredients be purchased.

[0789] Monitoring function

[0790] 7. Monitoring door opening and closing status

[0791] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0792] 8. Trend Analysis and Anomaly Detection

[0793] Server: Performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, it notifies the head-mounted display and other user devices via push notification.

[0794] Specific examples

[0795] Ingredients management

[0796] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[0797] Device (camera): Take a picture at 8:00 and send it to the server.

[0798] Server: Analyzes the image and generates a list of ingredients (e.g., "milk, tomato, lettuce"). The list is stored in a database and provided to the head-mounted display and other user devices.

[0799] Cooking menu suggestions

[0800] User: Checks the grocery list on the head-mounted display before lunch.

[0801] Server: Based on the ingredients list, suggests a cooking menu (e.g., "tomato soup" or "lettuce salad"). The user selects "tomato soup."

[0802] Server: Detects that the "chicken soup mix" required for the "tomato soup" is missing, and notifies the head-mounted display and other user devices of this fact.

[0803] Monitoring function implementation

[0804] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0805] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[0806] Server: Detects this anomaly and sends a push notification to the head-mounted display and other user devices. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[0807] Prompt Sentence Examples

[0808] "Analyze an image of the inside of a refrigerator and generate a list of ingredients including milk, tomatoes, and lettuce. Also, tell me what ingredients are missing to make tomato soup."

[0809] This system allows users, including the elderly, to efficiently manage and cook ingredients while ensuring safety.

[0810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0811] Step 1:

[0812] Webcam installation and image capture settings

[0813] Users can install web cameras on refrigerators, pantries, and cupboards, adjust the angle and position to capture the entire area, and set the timing of the camera captures.

[0814] Input: Physical location of the camera and timing settings.

[0815] Output: After the setup is complete, the camera will take an image at the specified time.

[0816] Step 2:

[0817] Taking and sending images

[0818] The device (camera) takes pictures of the inside of the refrigerator or pantry at a set time and sends them to a server via the Internet.

[0819] Input: The image captured by the camera.

[0820] Data processing: Generation of image files and transmission over the Internet.

[0821] Output: The image file sent to the server.

[0822] Step 3:

[0823] Image analysis

[0824] The server uses a generative AI model (e.g., GPT-4) to analyze the received image, recognize ingredients in the image, and generate an ingredient list.

[0825] Input: The submitted image file.

[0826] Data Computing: Image recognition and ingredient list generation using generative AI models.

[0827] Output: A list of ingredients (e.g. "Milk, Tomato, Lettuce").

[0828] Step 4:

[0829] Save your grocery list

[0830] The server stores the generated ingredient list in a database.

[0831] Input: Ingredient list.

[0832] Data processing: Writing list data to a database.

[0833] Output: Ingredient list stored in database.

[0834] Step 5:

[0835] Cooking menu suggestions

[0836] The server uses a generative AI model to suggest cooking menus based on the stored ingredient list and transmits the results to the head-mounted display and other user devices.

[0837] Input: Saved ingredient list.

[0838] Data Computing: Menu suggestions using generative AI models and results transmission.

[0839] Output: A cooking menu displayed on the user's device (e.g., "Tomato soup" or "Lettuce salad").

[0840] Step 6:

[0841] Suggestions for ingredients that are in short supply

[0842] If any ingredients required for the proposed menu are missing, the server generates a list of the missing ingredients and notifies the head-mounted display and other user terminals of the list.

[0843] Input: Proposed cooking menu and current ingredients list.

[0844] Data calculation: Determine shortages of ingredients and generate notification lists.

[0845] Output: Notification of missing ingredients (e.g. "You are running low on chicken soup base").

[0846] Step 7:

[0847] Door opening / closing status monitoring

[0848] The terminal (camera) monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0849] Input: Signal from door open / close sensor.

[0850] Data processing: Collection and transmission of door opening and closing data.

[0851] Output: Door open / close data sent to the server.

[0852] Step 8:

[0853] Trend analysis and anomaly detection

[0854] The server performs trend analysis based on the received door opening and closing data to detect abnormal patterns or cases where the door has not been opened or closed within a certain period of time. If an abnormality is detected, a push notification is sent to the user's head-mounted display and other devices.

[0855] Input: Door opening and closing data.

[0856] Data arithmetic: Applying trend analysis and anomaly detection algorithms.

[0857] Output: Push notification if an abnormality is detected (e.g. "The refrigerator has not been opened or closed for more than 24 hours. Please check it.").

[0858] Step 9:

[0859] Visual Notifications

[0860] The server sends various information (ingredient list, cooking menu, missing ingredients, abnormality notification) to the head-mounted display and notifies the user visually.

[0861] Input: Generated information (ingredient list, cooking menu, missing ingredients, abnormality notification).

[0862] Data processing: information format conversion and transmission procedures.

[0863] Output: Information visually displayed on a head-mounted display.

[0864] Through the above processing steps, the user can efficiently manage, cook, and check the safety of ingredients.

[0865] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0866] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, creates and manages ingredient lists, and also has cooking menu suggestions and monitoring functions, and even recognizes the user's emotions to suggest appropriate menus. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[0867] System Configuration

[0868] 1. Webcam

[0869] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[0870] 2. Image capture and transmission

[0871] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[0872] 3. Image Analysis

[0873] Server: Uses artificial intelligence to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[0874] 4. Creating and managing ingredient lists

[0875] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[0876] Cooking menu suggestions

[0877] 5. Cooking menu generation

[0878] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[0879] 6. Suggesting inadequate ingredients

[0880] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[0881] Monitoring function

[0882] 7. Monitoring door opening and closing status

[0883] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[0884] 8. Trend Analysis and Anomaly Detection

[0885] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[0886] Emotion recognition function

[0887] 9. User Emotion Recognition

[0888] Device: Uses a camera and microphone to detect the user's facial expressions and voice, and sends the data to the emotion engine.

[0889] Emotion engine: Analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[0890] 10. Emotion-Based Suggestions

[0891] Server: Based on data from the emotion engine, it generates a cooking menu according to the user's emotional state.

[0892] Server: The server sends the generated menu to the user's device and makes suggestions that match their emotional state. For example, if the user is feeling stressed, a menu with a relaxing effect will be selected.

[0893] 11. Recording and analyzing emotional data

[0894] Server: Periodically records user emotion data and performs trend analysis.

[0895] Server: Accumulates data to provide users with menu preferences and appropriate suggestions based on their past emotional patterns.

[0896] Specific examples

[0897] Ingredient management and emotion recognition

[0898] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[0899] Device (camera): Take a picture at 8:00 and send it to the server.

[0900] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[0901] Device: The camera and microphone also detect the user's facial expressions and voice data, and send them to the emotion engine.

[0902] Cooking menu suggestions

[0903] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[0904] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[0905] Emotion-based suggestions

[0906] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[0907] Monitoring function implementation

[0908] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[0909] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[0910] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[0911] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[0912] The processing flow will be explained below.

[0913] Program processing flow

[0914] Imaging and analysis

[0915] Step 1:

[0916] The device (camera) takes pictures of the refrigerator and pantry at regular intervals each day (e.g., 8:00 a.m.).

[0917] Step 2:

[0918] The captured image is sent from the device (camera) to the server.

[0919] Step 3:

[0920] The server receives the image and temporarily stores it in a buffer.

[0921] Step 4:

[0922] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[0923] Step 5:

[0924] The server identifies ingredients in the image and generates an ingredient list.

[0925] Step 6:

[0926] The generated ingredient list is stored in a database on the server.

[0927] Step 7:

[0928] The server transmits the stored ingredient list to the user's terminal.

[0929] Cooking menu suggestions

[0930] Step 8:

[0931] The user checks the ingredient list on the device.

[0932] Step 9:

[0933] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[0934] Step 10:

[0935] The server transmits the generated cooking menu to the user's terminal.

[0936] Step 11:

[0937] The user checks the presented cooking menu on the terminal and selects one.

[0938] Suggestions for ingredients that are in short supply

[0939] Step 12:

[0940] The server will list any missing ingredients based on the menu selected.

[0941] Step 13:

[0942] The server sends a list of ingredients that are in short supply to the user's terminal.

[0943] Step 14:

[0944] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[0945] Monitoring function

[0946] Step 15:

[0947] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[0948] Step 16:

[0949] The terminal (camera) periodically sends door opening / closing information to the server.

[0950] Step 17:

[0951] The server accumulates the opening and closing data received and performs trend analysis.

[0952] Step 18:

[0953] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[0954] Step 19:

[0955] If the server detects an abnormality, it generates warning information.

[0956] Step 20:

[0957] The server generates a push notification of the warning information to the user's device.

[0958] Step 21:

[0959] The user receives a push notification to check the status.

[0960] Emotion recognition function

[0961] Step 22:

[0962] The device uses a camera and microphone to detect the user's facial expressions and voice and sends the data to the emotion engine.

[0963] Step 23:

[0964] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, happiness).

[0965] Step 24:

[0966] The server generates a cooking menu according to the user's emotional state based on the data from the emotion engine.

[0967] Step 25:

[0968] The server sends the generated menu to the user's device and makes suggestions that match the user's emotional state.

[0969] Step 26:

[0970] The server periodically records the user's emotional data and performs trend analysis.

[0971] Step 27:

[0972] The server accumulates data based on the user's past emotional patterns to provide appropriate suggestions and menus that the user prefers.

[0973] Step 28:

[0974] The device then performs appropriate actions based on the user's emotional state (e.g., suggesting favorite recipes or sending encouraging messages).

[0975] Through the above processing steps, the present invention can efficiently manage ingredients and monitor elderly people all at once, and can also suggest menus that match the user's mood through emotion recognition.

[0976] Example 2

[0977] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0978] Currently, there are systems on the market that manage ingredients in refrigerators and pantries and suggest cooking menus, but these do not detect anomalies based on the user's emotions or the opening and closing status of storage device doors. Therefore, there is a need for systems that can efficiently and low-costly support home cooking and monitor the elderly. In particular, there is a lack of systems that can suggest personalized menus based on the status of ingredients and the user's emotional state.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0980] In this invention, the server includes means for installing a webcam in the storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images and generating an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device door, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, means for detecting the user's facial expression and voice and recognizing emotions, and means for proposing cooking menus based on the emotion data. This enables flexible and efficient ingredient management, cooking menu suggestions, and abnormality detection based on the user's emotions and storage status.

[0981] "Storage equipment" means a device for storing food or goods, examples of which include refrigerators, pantries, and cupboards.

[0982] A "webcam" is a camera device that captures and transmits images and videos over the Internet.

[0983] The "food list" is a list that lists the types and quantities of food extracted from the analyzed image data.

[0984] "Artificial intelligence means" refers to algorithms and machine learning models used for image analysis, data processing, menu suggestions, etc.

[0985] "Door open / close status" is data indicating the state of the door of the storage device, whether it is open or closed.

[0986] "Anomaly detection" is the process of recognizing behavior that deviates from normal opening and closing patterns and reporting the problem.

[0987] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0988] "Means for detecting facial expressions and voice" refers to hardware and software for recognizing the user's facial movements and voice and collecting them as data.

[0989] "Means for recognizing emotions" refers to an algorithm or system that analyzes detected facial and voice data to determine the emotional state of the user.

[0990] "Trend analysis" is the process of analyzing collected data to identify trends and patterns over a period of time.

[0991] This invention is a system that manages ingredients stored in storage devices, suggests cooking menus, monitors elderly people, and suggests menus based on their emotions. The system combines webcams, image analysis AI, and emotion recognition technology to provide an efficient and personalized experience.

[0992] Hardware and Software Configuration

[0993] 1. Webcam

[0994] Users place a webcam, such as the Logitech C920, in their refrigerator, pantry, or cupboard, capturing a full view of the contents of the storage device and recording daily changes in the ingredients.

[0995] 2. Image capture and transmission

[0996] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. The software used here is custom firmware, which provides the scheduled shooting and automatic sending functions.

[0997] 3. Image Analysis

[0998] The server uses image recognition artificial intelligence tools such as TensorFlow to analyze the received image, recognize ingredients in the image (e.g., milk, tomato, lettuce), and generate an ingredient list based on that information.

[0999] 4. Creating and managing ingredient lists

[1000] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[1001] 5. Recipe generation and suggestions

[1002] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, recipes for "tomato soup" and "lettuce salad" using "milk," "tomato," and "lettuce" are generated and sent to the user's device.

[1003] 6. Suggestion of necessary ingredients

[1004] If the server is running low on ingredients for a proposed menu, it generates a list of those ingredients and notifies the user. For example, it recognizes that the "chicken soup base" needed for "tomato soup" is missing, and notifies the user of that information.

[1005] 7. Monitoring door opening and closing status

[1006] The device (camera) monitors whether the storage device door is open or closed, and periodically sends the data to the server. The timing of the door opening and closing is recorded and compiled by the server.

[1007] 8. Trend Analysis and Anomaly Detection

[1008] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, a push notification is sent to the user's device.

[1009] 9. User Emotion Recognition

[1010] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[1011] 10. Emotion-Based Suggestions

[1012] The server generates a cooking menu based on the data from the emotion engine according to the user's emotional state. For example, if the user is feeling depressed, it will suggest a "herbal tea recipe" that has a relaxing effect. The generated menu is sent to the user's device.

[1013] 11. Recording and analyzing emotional data

[1014] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to suggest menus that the user prefers and make appropriate suggestions.

[1015] Examples and prompts

[1016] As a specific example, the following procedure is carried out.

[1017] Ingredient management and emotion recognition

[1018] User: Places a Logitech C920 webcam inside the refrigerator and sets it to automatically record at 8:00 AM every day.

[1019] Example prompt: "Set up your webcam and take a picture of the inside of your refrigerator every day at 8 AM."

[1020] Image capture and transmission

[1021] Device (camera): Take a picture at 8:00 and send it to the server.

[1022] Image analysis

[1023] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[1024] Emotion Recognition and Suggestions

[1025] Device (camera and microphone): Detects the user's facial expressions and voice data and sends them to the emotion engine.

[1026] Cooking menu suggestions

[1027] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[1028] Sample prompt: "We analyzed the image of your refrigerator and detected milk, tomatoes, and lettuce. We've added them to your current ingredient list."

[1029] Suggestion of necessary ingredients

[1030] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[1031] Emotion-based suggestions

[1032] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[1033] Example prompt: "The user's facial expression appears depressed. Suggest a relaxing herbal tea recipe."

[1034] Monitoring function implementation

[1035] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1036] Trend analysis and anomaly detection

[1037] Server: Conducts trend analysis and detects that the door normally opens and closes several times a day, but has not opened or closed for more than 24 hours. This anomaly is detected and a push notification is sent to the user's device.

[1038] Sample prompt: "The refrigerator hasn't been opened or closed for over 24 hours. Please check."

[1039] This system allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[1040] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1041] Processing Steps

[1042] Step 1: Setting up your webcam

[1043] Users place a webcam, such as the Logitech C920, on a refrigerator, pantry, or cupboard and adjust the angle and position to capture the entire interior. For example, place the webcam on top of the refrigerator and adjust it to a position that allows a full view of the interior.

[1044] Input: Webcam installation instructions, refrigerator location

[1045] Output: A webcam placed in a suitable position, ready to record.

[1046] Step 2: Take and send an image

[1047] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. Custom firmware is used to implement the scheduled shooting and automatic sending functions.

[1048] Input: Set shooting schedule, webcam location information

[1049] Output: Captured image data, image data transmission to server

[1050] Step 3: Image analysis

[1051] The server analyzes the received image using image recognition AI such as TensorFlow, which identifies ingredients in the image (e.g., milk, tomato, lettuce), and generates an ingredient list based on that information.

[1052] Input: Captured image data

[1053] Output: Ingredient list data as the analysis result

[1054] Step 4: Generate and manage your ingredient list

[1055] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[1056] Input: Ingredient list data as a result of image analysis

[1057] Output: Ingredient list stored in the database, provided to the user's device

[1058] Step 5: Recipe generation and suggestions

[1059] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, a recipe for "tomato soup" or "lettuce salad" using "milk," "tomato," and "lettuce" is generated and sent to the user's device.

[1060] Input: List of ingredients in the database

[1061] Output: Generated recipes, suggestions to user terminal

[1062] Step 6: Suggesting ingredients

[1063] If the server is missing any ingredients needed for the proposed menu, it generates a list of those ingredients and notifies the user terminal. For example, it recognizes that "chicken soup base" needed for "tomato soup" is missing and notifies the user of that information.

[1064] Input: Cooking menu, current ingredients list

[1065] Output: List of missing ingredients, notification to user device

[1066] Step 7: Monitor the door opening and closing status

[1067] The device (camera) monitors the opening and closing status of the storage device door and periodically transmits the data to the server. The timing of the door opening and closing is recorded and transmitted to the server.

[1068] Input: Door open / close status, timestamp

[1069] Output: Open / close data sent to the server

[1070] Step 8: Trend analysis and anomaly detection

[1071] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device.

[1072] Input: Door opening / closing data

[1073] Output: Anomaly detection results, notification to user device

[1074] Step 9: Recognizing User Emotions

[1075] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[1076] Input: facial expression data, voice data

[1077] Output: Emotion recognition result

[1078] Step 10: Emotion-Based Suggestions

[1079] The server generates a cooking menu based on the user's emotional state based on the data from the emotion engine. For example, if the user is feeling depressed, it will suggest a relaxing "herbal tea recipe." The generated menu is sent to the user's device.

[1080] Input: Emotion recognition results, list of ingredients

[1081] Output: Emotion-based cooking menu, sent to user device

[1082] Step 11: Record and analyze emotional data

[1083] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to provide the user with menus they prefer and make appropriate suggestions.

[1084] Input: Periodically collected emotion data

[1085] Output: Sentiment trend analysis results, personalized menu suggestion data

[1086] (Application example 2)

[1087] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1088] In the past, managing ingredients in the refrigerator or pantry and suggesting cooking menus was often done manually, making it difficult for users to efficiently manage ingredients and select cooking menus. Furthermore, there were issues with systems that were unable to provide services that monitored elderly people or responded to the user's emotions. Furthermore, even in brick-and-mortar stores, there was a demand for automated services based on customers' emotions and food inventory management, but these systems were difficult to implement.

[1089] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1090] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, emotion recognition means for recognizing the user's facial expressions and voice and analyzing the data, and means for proposing cooking menus based on the emotion data obtained from the emotion recognition means. This enables automation of ingredient management in brick-and-mortar stores and the proposing of personalized cooking menus based on the user's emotions.

[1091] A "webcam" is a camera device that captures digital images or video and transmits them in real time over a computer or network.

[1092] The "ingredient list" is a list showing ingredients and items present in the refrigerator, pantry, and cupboard.

[1093] An "artificial intelligence means" is a system or software that uses computer algorithms to analyze data and mimic human intellectual activity.

[1094] "Door opening / closing status" is information indicating the status of the opening and closing of the refrigerator, pantry, and cupboard doors.

[1095] An "anomaly" is a condition or action that deviates from a normal pattern or expected behavior.

[1096] "User terminal" means a computer device operated by a user to receive and display information.

[1097] "Emotion recognition means" refers to a system or software that analyzes a user's facial expressions and voice to identify their emotions.

[1098] "Emotion data" is data that represents the emotional state of the user analyzed from facial expressions and voice.

[1099] A "personalized cooking menu" is a cooking menu suggested to suit the user's individual preferences and condition.

[1100] This system uses a webcam to capture images of food storage areas such as kitchens and stockrooms in brick-and-mortar stores, analyzes the images to generate and manage ingredient lists, and suggests cooking menus based on customer sentiment. This system automates food management in brick-and-mortar stores and improves the customer experience.

[1101] System Configuration

[1102] 1. Webcam installation

[1103] User (store clerk): Install a webcam in the kitchen or stockroom of the physical store and adjust the angle and position to capture the entire scene.

[1104] 2. Image capture and transmission

[1105] Terminal (camera): Takes images at regular intervals (e.g., every two hours) and sends them to a server via the Internet.

[1106] 3. Image Analysis

[1107] Server: Analyzes the received image using AI (e.g., AWS Rekognition or Google Cloud Vision), recognizes the ingredients in the image, and generates an ingredient list.

[1108] 4. Creating and managing ingredient lists

[1109] Server: Stores the generated ingredient list in a database and updates and manages it as needed.

[1110] Cooking menu suggestions

[1111] 5. Cooking menu generation

[1112] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to and displayed on the smartphones or smart glasses of customers and store staff.

[1113] 6. Suggestions for missing ingredients

[1114] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[1115] Monitoring function

[1116] 7. Monitoring door opening and closing status

[1117] Terminal (camera): Monitors whether refrigerators and pantry doors are open or closed in physical stores, and periodically sends this data to the server.

[1118] 8. Trend Analysis and Anomaly Detection

[1119] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[1120] Emotion recognition function

[1121] 9. User Emotion Recognition

[1122] Device: Uses a camera and microphone to detect the customer's facial expressions and voice, and sends the data to an emotion engine (e.g., Microsoft Azure Cognitive Services' Face API).

[1123] Server: Analyzes the data received from the emotion engine and recognizes the emotional state of the customer (e.g., joy, anger, sadness, or happiness).

[1124] 10. Emotion-Based Suggestions

[1125] Server: Generates cooking menus based on emotion data and sends them to the customers' and staff's devices. For example, if a customer is feeling stressed, the server suggests menus that will have a relaxing effect.

[1126] 11. Recording and analyzing emotional data

[1127] Server: Periodically records user emotion data and performs trend analysis.

[1128] Server: Accumulates data to determine customers' preferred menu items and make appropriate suggestions based on past emotional patterns.

[1129] Specific examples

[1130] Ingredient management and emotion recognition

[1131] User (Store Staff): Set up a webcam in the kitchen to capture images of the food storage area at 8:00 AM and 2:00 PM every day.

[1132] Terminal (camera): Takes a picture at the set time and sends it to the server.

[1133] Server: Analyzes the image, generates an ingredient list, and stores it in a database.

[1134] Terminal: The camera and microphone also detect the customer's facial expressions and voice data, which are then sent to the emotion engine.

[1135] Cooking menu suggestions

[1136] Server: Based on the ingredients list and the analysis results of the emotion engine, the server proposes multiple menu items and sends them to the user's device. The customer selects a menu item.

[1137] Suggestions for ingredients that are in short supply

[1138] Server: If any ingredients required for the selected menu are missing, it generates a list of them and notifies the user terminal of that fact.

[1139] Emotion-based suggestions

[1140] Server: If a customer's expression is deemed depressed, the server will suggest, for example, a relaxing "herbal tea recipe."

[1141] Monitoring function implementation

[1142] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1143] Server: Conducts trend analysis and detects when the refrigerator has not been opened or closed for 24 hours or more. If an abnormality is detected, a push notification is sent to the user's device. The push notification displays the message, "The refrigerator has not been opened or closed for 24 hours or more. Please check."

[1144] Prompt Sentence Examples

[1145] By inputting the following prompts into the generative AI model, it is possible to suggest cooking menus that correspond to the customer's emotions.

[1146] Example prompt sentence:

[1147] Based on the user's emotional data, please create and suggest a menu that will have a relaxing effect. If the emotional state is "stressed" or "depressed," please explain what menu should be suggested and why.

[1148] With this configuration, the system can automate food ingredient management in physical stores and propose personalized cooking menus based on customers' emotions, providing an efficient and satisfying customer experience.

[1149] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1150] Step 1:

[1151] Camera installation and image capture

[1152] The user (store clerk) installs a webcam in the kitchen or stockroom of a physical store and adjusts the angle and position to capture the entire scene. The webcam takes images at regular intervals (e.g., every two hours) and generates image data. The input is the camera's installation position and shooting time settings, and the output is the captured image data.

[1153] Step 2:

[1154] Image transmission

[1155] The terminal (webcam) sends the captured image to a server via the Internet. The input is the captured image data, and the output is the image data sent to the server.

[1156] Step 3:

[1157] Image analysis

[1158] The server analyzes the received image data using an AI model (e.g., AWS Rekognition or Google Cloud Vision) to recognize the ingredients present in the image. It generates an ingredient list based on the analysis results. The input is the transmitted image data, and the output is the generated ingredient list.

[1159] Step 4:

[1160] Save Ingredient List

[1161] The server stores the generated ingredient list in a database. The input is the generated ingredient list, and the output is the ingredient list stored in the database.

[1162] Step 5:

[1163] Cooking menu suggestions

[1164] The server uses AI to generate cooking menus based on the ingredient list stored in the database and sends them to the user's device. The input is the stored ingredient list, and the output is the proposed cooking menu.

[1165] Step 6:

[1166] Notification of insufficient ingredients

[1167] If any ingredients required for the proposed cooking menu are missing, the server generates a list of those ingredients and notifies the user of the missing ingredients. The input is the proposed cooking menu and the list of ingredients, and the output is the list of missing ingredients and a notification of that list.

[1168] Step 7:

[1169] Door opening / closing status monitoring

[1170] The terminal (web camera) monitors the opening and closing status of the refrigerator and pantry doors and periodically transmits the data to the server. The input is the door opening and closing behavior, and the output is the opening and closing status data.

[1171] Step 8:

[1172] Anomaly detection

[1173] The server performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device. The input is the opening / closing status data, and the output is the abnormality detection and notification.

[1174] Step 9:

[1175] emotion recognition

[1176] The device uses a camera and microphone to detect the facial expressions and voices of customers and sends the data to an emotion engine (e.g., Microsoft Azure Cognitive Services Face API). The input is facial expression and voice data, and the output is emotion data.

[1177] Step 10:

[1178] Emotional Data Analysis

[1179] The server analyzes the data received from the emotion engine and recognizes the emotional state of the customer. The input is the emotion data, and the output is the analysis result.

[1180] Step 11:

[1181] Emotion-based menu suggestions

[1182] The server then proposes cooking menus to the customers' and store staff's devices based on the results of the emotion data analysis. For example, if a customer is feeling stressed, it will propose a menu with a relaxing effect. The input is the emotion data analysis results, and the output is a cooking menu proposal based on the customer's emotion.

[1183] Step 12:

[1184] Emotional data recording and analysis

[1185] The server periodically records users' emotional data and performs trend analysis. Based on past emotional patterns, the server accumulates data to help customers choose menu items they like and make appropriate suggestions. The input is emotional data, and the output is the recorded emotional data and the analysis results.

[1186] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1187] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1188] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1189] [Third embodiment]

[1190] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1191] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1192] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1193] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1194] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1195] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1196] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1197] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1198] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1199] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1200] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1201] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1202] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, and generates and manages ingredient lists. It also has cooking menu suggestions and monitoring functions. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[1203] System Configuration

[1204] 1. Webcam

[1205] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[1206] 2. Image capture and transmission

[1207] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[1208] 3. Image Analysis

[1209] Server: Uses artificial intelligence (such as GPT-4) to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[1210] 4. Creating and managing ingredient lists

[1211] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[1212] Cooking menu suggestions

[1213] 5. Cooking menu generation

[1214] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[1215] 6. Suggesting inadequate ingredients

[1216] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[1217] Monitoring function

[1218] 7. Monitoring door opening and closing status

[1219] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[1220] 8. Trend Analysis and Anomaly Detection

[1221] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[1222] Specific examples

[1223] Ingredients management

[1224] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[1225] Device (camera): Take a picture at 8:00 and send it to the server.

[1226] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[1227] Cooking menu suggestions

[1228] User: Checks grocery list on device before lunch.

[1229] Server: Based on the ingredients list, suggests menu items such as "Tomato soup" and "Lettuce salad." The user selects "Tomato soup."

[1230] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[1231] Monitoring function implementation

[1232] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1233] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[1234] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[1235] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to keep an eye on elderly people.

[1236] The processing flow will be explained below.

[1237] Program processing flow

[1238] Imaging and analysis

[1239] Step 1:

[1240] The device (camera) takes pictures of the inside of the refrigerator and pantry at regular intervals (e.g., every hour).

[1241] Step 2:

[1242] The captured image is sent from the device (camera) to the server.

[1243] Step 3:

[1244] The server receives the image and temporarily stores it in a buffer.

[1245] Image analysis and ingredient list generation

[1246] Step 4:

[1247] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[1248] Step 5:

[1249] The server identifies ingredients in the image and generates an ingredient list.

[1250] Step 6:

[1251] The generated ingredient list is stored in a database on the server.

[1252] Step 7:

[1253] The server transmits the stored ingredient list to the user's terminal.

[1254] Cooking menu suggestions

[1255] Step 8:

[1256] The user checks the ingredient list on the device.

[1257] Step 9:

[1258] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[1259] Step 10:

[1260] The server transmits the generated cooking menu to the user's terminal.

[1261] Step 11:

[1262] The user checks the presented cooking menu on the terminal and selects one.

[1263] Suggestions for ingredients that are in short supply

[1264] Step 12:

[1265] The server will list any missing ingredients based on the menu selected.

[1266] Step 13:

[1267] The server sends a list of ingredients that are in short supply to the user's terminal.

[1268] Step 14:

[1269] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[1270] Monitoring function

[1271] Step 15:

[1272] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[1273] Step 16:

[1274] The terminal (camera) periodically sends door opening / closing information to the server.

[1275] Step 17:

[1276] The server accumulates the opening and closing data received and performs trend analysis.

[1277] Step 18:

[1278] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[1279] Step 19:

[1280] If the server detects an abnormality, it generates warning information.

[1281] Step 20:

[1282] The server generates a push notification of the warning information to the user's device.

[1283] Step 21:

[1284] The user receives a push notification to check the status.

[1285] Through the above processing steps, the present invention makes it possible to efficiently manage ingredients and monitor elderly people all at once.

[1286] Example 1

[1287] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1288] The need for food management and home cooking assistance for modern families and the elderly is increasing, but there are few systems that can accurately grasp food inventory status and efficiently suggest cooking menus.Furthermore, there are almost no systems that have a function to monitor the elderly, so there is a need for a low-cost, easy-to-use system that combines both functions.

[1289] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1290] In this invention, the server includes means for installing an image capture device in a storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device, and means for detecting abnormalities based on the opening and closing status and notifying a user terminal. This allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people.

[1291] An "image capture device" is a device used to capture images of the inside of a refrigerator, pantry, or cupboard, and generally refers to a webcam or digital camera.

[1292] "Storage equipment" refers to equipment for storing food and ingredients, and specifically includes refrigerators, pantries, and cupboards.

[1293] "Artificial intelligence means" refers to technologies for image analysis and data processing, particularly machine learning algorithms and generative AI models.

[1294] A "database" is a digital system for organizing and storing information such as ingredient lists, and includes SQL databases and NoSQL databases.

[1295] A "user terminal" is a device operated by system users, and includes smartphones, PCs, and tablets.

[1296] "Image analysis" refers to the process of analyzing images captured by an image capture device using software or artificial intelligence means.

[1297] The "ingredient list" is a list of ingredients in the refrigerator or pantry, generated based on the analyzed image information.

[1298] "Cooking menu" refers to recipes and cooking methods suggested based on an ingredient list.

[1299] "Opening and closing status" refers to the state and frequency of opening and closing the doors of the refrigerator, pantry, and cupboard.

[1300] "Anomaly detection" is the process of identifying unusual patterns or behaviors and recognizing problems.

[1301] "Push notification" is a function that detects anomalies and sends important information to user devices in real time.

[1302] The present invention is a system that uses an image capture device to manage ingredients stored in a storage device and suggests appropriate cooking menus, and also has a monitoring function for elderly people. Specific embodiments of this system will be described below.

[1303] System Configuration and Operation

[1304] 1. Installation of imaging device

[1305] A user places an image capture device (such as a webcam) in a storage device such as a refrigerator, pantry, or cupboard, and adjusts the angle and position of the device to capture the entire food item in the storage device.

[1306] 2. Take and send images

[1307] The terminal (image capture device) takes pictures of the storage device at a set time. For example, it can be set to take pictures of the inside of the refrigerator every morning at 8:00. The images are connected to the Internet via a Wi-Fi module and sent to a server.

[1308] 3. Image Analysis

[1309] The server receives the image and performs image analysis. It uses artificial intelligence (e.g., GPT-4, TensorFlow, etc.) to identify and recognize ingredients in the image. For example, if "milk, tomato, lettuce" is recognized, it is added to the ingredients list.

[1310] 4. Creating and managing ingredient lists

[1311] The server generates an ingredient list based on the recognized ingredient information and stores it in a database. Duplicate ingredients are automatically omitted. This allows the user device to always receive the latest ingredient list.

[1312] 5. Cooking menu suggestions

[1313] The server generates a cooking menu based on the saved ingredient list. It inputs prompts to a generative AI model (e.g., GPT-4) to suggest appropriate recipes. An example of a prompt is "Please suggest a recommended cooking menu based on the current ingredient list."

[1314] 6. Suggestions for missing ingredients

[1315] If the server detects that a suggested menu item is missing, it lists the ingredients that need to be purchased and notifies the user. For example, if a recipe for "tomato soup" requires "chicken soup base" but is in short supply, the server notifies the user terminal, "You are running low on chicken soup base. Please buy more."

[1316] 7. Monitoring door opening and closing status

[1317] The terminal (image capture device) is attached with a sensor on the door of the storage device to monitor the door opening and closing status. The opening and closing data is periodically recorded and sent to the server.

[1318] 8. Trend Analysis and Anomaly Detection

[1319] The server performs trend analysis based on the received door opening / closing data. It learns normal opening / closing frequencies and patterns, and detects abnormal patterns or cases where the door has not been opened / closed for a certain period of time. For example, if the door is normally opened / closed four times a day, but there has been no opening / closing for more than 24 hours, this is detected as an abnormality. If this abnormality is detected, a push notification is sent to the user's device saying, "The refrigerator has not been opened / closed for more than 24 hours. Please check."

[1320] This allows users to efficiently manage ingredients and receive appropriate cooking menus. It also makes it possible to simultaneously monitor elderly people, improving safety.

[1321] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1322] Step 1: Installing the imaging device

[1323] The user places an image capture device, such as a webcam, on a refrigerator, pantry, or cupboard at an appropriate angle and position so that the entire contents of the storage device are captured. For example, the camera can be attached to the top of the refrigerator and point downward to capture the interior.

[1324] Step 2: Take and send an image

[1325] The terminal (image capture device) automatically captures images in the storage device at a set time. A timer setting (e.g., every morning at 8:00) is given as input. The captured images are connected to the Internet via Wi-Fi and sent to a server. As output, a JPEG image file is generated.

[1326] Step 3: Image analysis

[1327] The server receives the transmitted image and performs image analysis. As input, a JPEG image file is provided. Specifically, artificial intelligence tools (e.g., GPT-4 or TensorFlow) are used to identify and recognize the ingredients in the image. An image processing algorithm is applied to process the data, and as output, a list of ingredient information (e.g., "Milk, Tomato, Lettuce") is generated.

[1328] Step 4: Generate and manage your ingredient list

[1329] The server generates an ingredient list based on the analyzed ingredient information and stores it in a database. The generated ingredient information list is given as input. Data calculations include removing duplicate items and reorganizing the list. An updated ingredient list is generated as output and synchronized with the user's device.

[1330] Step 5: Cooking menu suggestions

[1331] The server generates a cooking menu based on the saved ingredient list. The latest ingredient list is given as input. A prompt is input to the generative AI model (e.g., GPT-4) to suggest an appropriate recipe. For example, the prompt is "Please suggest a recommended cooking menu based on the current ingredient list." As output, multiple cooking menus (e.g., "Tomato soup," "Lettuce salad") are generated and sent to the user device.

[1332] Step 6: Suggest missing ingredients

[1333] The server detects if any ingredients required for the proposed menu are missing. As input, the proposed menu and a list of ingredients are given. As data processing, the ingredients required for the menu are compared with the currently available ingredients. As output, a list of missing ingredients (e.g., "chicken soup base") is generated and notified to the user's terminal.

[1334] Step 7: Monitor the door opening and closing status

[1335] The terminal (image capture device) uses a sensor installed on the door of the storage device to monitor the door's opening and closing status. As input, a timestamp of the door opening and closing is recorded. As data processing, a log of the opening and closing status is generated and sent to a server via the Internet. As output, opening and closing log data is generated.

[1336] Step 8: Trend analysis and anomaly detection

[1337] The server performs trend analysis based on the received opening / closing log data. The door opening / closing log data is given as input. Data calculations learn the normal opening / closing frequency and patterns, and detect abnormal patterns. For example, if the door is normally opened and closed four times a day, but there is no opening / closing for more than 24 hours, this is considered an abnormality. The output is the result of the abnormality detection (e.g., "The refrigerator has not been opened or closed for more than 24 hours. Please check"), and a push notification is sent to the user's device.

[1338] (Application example 1)

[1339] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] Conventional systems for managing the contents of refrigerators, pantries, and cupboards have functions such as managing ingredients, suggesting cooking menus, and monitoring opening and closing status, but there are few ways to provide these functions in a visually easy-to-understand manner for elderly people. It is also difficult to immediately visually check information about shortages of ingredients or monitoring functions. Therefore, efficiently managing ingredients and safety at home for elderly people has been a challenge.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1342] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the open / close status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the open / close status and notifying a user terminal, means for displaying the ingredient list and the open / close status of the doors on a head-mounted display, and means for visually notifying the user via the information notified and suggested on the head-mounted display. This allows even elderly people to visually manage ingredients, monitor the open / close status, and check safety.

[1343] A "webcam" is a camera device that can capture and transmit video over the Internet.

[1344] "Artificial intelligence means" refers to AI algorithms or software that run on a computer and learn, infer, and analyze data input to it.

[1345] A "grocery list" refers to a list of the types and quantities of ingredients in your refrigerator, pantry, or cupboard.

[1346] "Cooking menu" refers to a list of recipes for dishes that can be made using specific ingredients.

[1347] The "open / close status monitoring means" is a device or system that monitors the opening and closing of refrigerator, pantry, and cupboard doors and acquires the data.

[1348] An "anomaly detection means" is a system or algorithm that determines if an unusual pattern or problem has occurred based on acquired data.

[1349] A "user terminal" is an electronic device used by a user to receive, check, and operate information, and includes smartphones, tablets, and personal computers.

[1350] A "head-mounted display" is a display device that a user wears on their head to display visual information.

[1351] "Notification means" refers to a function or device for conveying specific information to a user, and includes push notifications, audio alarms, etc.

[1352] A "buffer" refers to a memory area that temporarily stores data, enabling high-speed reading and writing of data.

[1353] A "database" is a system or software for systematically storing data and for efficient searching and retrieval.

[1354] A "visual notification means" is a device or system for displaying information so that it can be perceived visually by a user.

[1355] This invention is a system that uses a webcam to capture images of the inside of refrigerators, pantries, and cupboards, analyzes the images, and generates and manages an ingredient list, providing visual information to users, including the elderly. In particular, by combining image analysis using artificial intelligence with a visual notification function using a head-mounted display, it provides a mechanism that allows users to quickly check the status of ingredients and any abnormalities.

[1356] System Configuration

[1357] 1. Webcam

[1358] User: Place a webcam on your refrigerator, pantry, or cupboard, and angle and position it to capture the entire area.

[1359] 2. Image capture and transmission

[1360] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to the server, allowing you to keep up to date with the latest food information.

[1361] 3. Image Analysis

[1362] Server: Uses a generative AI model (such as GPT-4) to analyze the submitted image, recognize ingredients in the image, and generate an ingredient list.

[1363] 4. Creating and managing ingredient lists

[1364] Server: Generates an ingredient list based on the analysis results and stores it in a database. The generated list can be viewed on the user's head-mounted display or other devices.

[1365] 5. Cooking menu suggestions

[1366] Server: Based on the stored ingredient list, the server uses a generative AI model to suggest cooking menus. The suggested menus are sent to the user's head-mounted display, where they can be selected and used.

[1367] 6. Suggestions for missing ingredients

[1368] Server: If any ingredients required for the proposed menu are missing, the server generates a list of those ingredients and notifies the head-mounted display and user device, suggesting that the missing ingredients be purchased.

[1369] Monitoring function

[1370] 7. Monitoring door opening and closing status

[1371] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[1372] 8. Trend Analysis and Anomaly Detection

[1373] Server: Performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, it notifies the head-mounted display and other user devices via push notification.

[1374] Specific examples

[1375] Ingredients management

[1376] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[1377] Device (camera): Take a picture at 8:00 and send it to the server.

[1378] Server: Analyzes the image and generates a list of ingredients (e.g., "milk, tomato, lettuce"). The list is stored in a database and provided to the head-mounted display and other user devices.

[1379] Cooking menu suggestions

[1380] User: Checks the grocery list on the head-mounted display before lunch.

[1381] Server: Based on the ingredients list, suggests a cooking menu (e.g., "tomato soup" or "lettuce salad"). The user selects "tomato soup."

[1382] Server: Detects that the "chicken soup mix" required for the "tomato soup" is missing, and notifies the head-mounted display and other user devices of this fact.

[1383] Monitoring function implementation

[1384] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1385] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[1386] Server: Detects this anomaly and sends a push notification to the head-mounted display and other user devices. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[1387] Prompt Sentence Examples

[1388] "Analyze an image of the inside of a refrigerator and generate a list of ingredients including milk, tomatoes, and lettuce. Also, tell me what ingredients are missing to make tomato soup."

[1389] This system allows users, including the elderly, to efficiently manage and cook ingredients while ensuring safety.

[1390] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1391] Step 1:

[1392] Webcam installation and image capture settings

[1393] Users can install web cameras on refrigerators, pantries, and cupboards, adjust the angle and position to capture the entire area, and set the timing of the camera captures.

[1394] Input: Physical location of the camera and timing settings.

[1395] Output: After the setup is complete, the camera will take an image at the specified time.

[1396] Step 2:

[1397] Taking and sending images

[1398] The device (camera) takes pictures of the inside of the refrigerator or pantry at a set time and sends them to a server via the Internet.

[1399] Input: The image captured by the camera.

[1400] Data processing: Generation of image files and transmission over the Internet.

[1401] Output: The image file sent to the server.

[1402] Step 3:

[1403] Image analysis

[1404] The server uses a generative AI model (e.g., GPT-4) to analyze the received image, recognize ingredients in the image, and generate an ingredient list.

[1405] Input: The submitted image file.

[1406] Data Computing: Image recognition and ingredient list generation using generative AI models.

[1407] Output: A list of ingredients (e.g. "Milk, Tomato, Lettuce").

[1408] Step 4:

[1409] Save your grocery list

[1410] The server stores the generated ingredient list in a database.

[1411] Input: Ingredient list.

[1412] Data processing: Writing list data to a database.

[1413] Output: Ingredient list stored in database.

[1414] Step 5:

[1415] Cooking menu suggestions

[1416] The server uses a generative AI model to suggest cooking menus based on the stored ingredient list and transmits the results to the head-mounted display and other user devices.

[1417] Input: Saved ingredient list.

[1418] Data Computing: Menu suggestions using generative AI models and results transmission.

[1419] Output: A cooking menu displayed on the user's device (e.g., "Tomato soup" or "Lettuce salad").

[1420] Step 6:

[1421] Suggestions for ingredients that are in short supply

[1422] If any ingredients required for the proposed menu are missing, the server generates a list of the missing ingredients and notifies the head-mounted display and other user terminals of the list.

[1423] Input: Proposed cooking menu and current ingredients list.

[1424] Data calculation: Determine shortages of ingredients and generate notification lists.

[1425] Output: Notification of missing ingredients (e.g. "You are running low on chicken soup base").

[1426] Step 7:

[1427] Door opening / closing status monitoring

[1428] The terminal (camera) monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[1429] Input: Signal from door open / close sensor.

[1430] Data processing: Collection and transmission of door opening and closing data.

[1431] Output: Door open / close data sent to the server.

[1432] Step 8:

[1433] Trend analysis and anomaly detection

[1434] The server performs trend analysis based on the received door opening and closing data to detect abnormal patterns or cases where the door has not been opened or closed within a certain period of time. If an abnormality is detected, a push notification is sent to the user's head-mounted display and other devices.

[1435] Input: Door opening and closing data.

[1436] Data arithmetic: Applying trend analysis and anomaly detection algorithms.

[1437] Output: Push notification if an abnormality is detected (e.g. "The refrigerator has not been opened or closed for more than 24 hours. Please check it.").

[1438] Step 9:

[1439] Visual Notifications

[1440] The server sends various information (ingredient list, cooking menu, missing ingredients, abnormality notification) to the head-mounted display and notifies the user visually.

[1441] Input: Generated information (ingredient list, cooking menu, missing ingredients, abnormality notification).

[1442] Data processing: information format conversion and transmission procedures.

[1443] Output: Information visually displayed on a head-mounted display.

[1444] Through the above processing steps, the user can efficiently manage, cook, and check the safety of ingredients.

[1445] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1446] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, creates and manages ingredient lists, and also has cooking menu suggestions and monitoring functions, and even recognizes the user's emotions to suggest appropriate menus. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[1447] System Configuration

[1448] 1. Webcam

[1449] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[1450] 2. Image capture and transmission

[1451] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[1452] 3. Image Analysis

[1453] Server: Uses artificial intelligence to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[1454] 4. Creating and managing ingredient lists

[1455] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[1456] Cooking menu suggestions

[1457] 5. Cooking menu generation

[1458] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[1459] 6. Suggesting inadequate ingredients

[1460] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[1461] Monitoring function

[1462] 7. Monitoring door opening and closing status

[1463] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[1464] 8. Trend Analysis and Anomaly Detection

[1465] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[1466] Emotion recognition function

[1467] 9. User Emotion Recognition

[1468] Device: Uses a camera and microphone to detect the user's facial expressions and voice, and sends the data to the emotion engine.

[1469] Emotion engine: Analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[1470] 10. Emotion-Based Suggestions

[1471] Server: Based on data from the emotion engine, it generates a cooking menu according to the user's emotional state.

[1472] Server: The server sends the generated menu to the user's device and makes suggestions that match their emotional state. For example, if the user is feeling stressed, a menu with a relaxing effect will be selected.

[1473] 11. Recording and analyzing emotional data

[1474] Server: Periodically records user emotion data and performs trend analysis.

[1475] Server: Accumulates data to provide users with menu preferences and appropriate suggestions based on their past emotional patterns.

[1476] Specific examples

[1477] Ingredient management and emotion recognition

[1478] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[1479] Device (camera): Take a picture at 8:00 and send it to the server.

[1480] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[1481] Device: The camera and microphone also detect the user's facial expressions and voice data, and send them to the emotion engine.

[1482] Cooking menu suggestions

[1483] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[1484] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[1485] Emotion-based suggestions

[1486] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[1487] Monitoring function implementation

[1488] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1489] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[1490] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[1491] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[1492] The processing flow will be explained below.

[1493] Program processing flow

[1494] Imaging and analysis

[1495] Step 1:

[1496] The device (camera) takes pictures of the refrigerator and pantry at regular intervals each day (e.g., 8:00 a.m.).

[1497] Step 2:

[1498] The captured image is sent from the device (camera) to the server.

[1499] Step 3:

[1500] The server receives the image and temporarily stores it in a buffer.

[1501] Step 4:

[1502] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[1503] Step 5:

[1504] The server identifies ingredients in the image and generates an ingredient list.

[1505] Step 6:

[1506] The generated ingredient list is stored in a database on the server.

[1507] Step 7:

[1508] The server transmits the stored ingredient list to the user's terminal.

[1509] Cooking menu suggestions

[1510] Step 8:

[1511] The user checks the ingredient list on the device.

[1512] Step 9:

[1513] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[1514] Step 10:

[1515] The server transmits the generated cooking menu to the user's terminal.

[1516] Step 11:

[1517] The user checks the presented cooking menu on the terminal and selects one.

[1518] Suggestions for ingredients that are in short supply

[1519] Step 12:

[1520] The server will list any missing ingredients based on the menu selected.

[1521] Step 13:

[1522] The server sends a list of ingredients that are in short supply to the user's terminal.

[1523] Step 14:

[1524] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[1525] Monitoring function

[1526] Step 15:

[1527] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[1528] Step 16:

[1529] The terminal (camera) periodically sends door opening / closing information to the server.

[1530] Step 17:

[1531] The server accumulates the opening and closing data received and performs trend analysis.

[1532] Step 18:

[1533] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[1534] Step 19:

[1535] If the server detects an abnormality, it generates warning information.

[1536] Step 20:

[1537] The server generates a push notification of the warning information to the user's device.

[1538] Step 21:

[1539] The user receives a push notification to check the status.

[1540] Emotion recognition function

[1541] Step 22:

[1542] The device uses a camera and microphone to detect the user's facial expressions and voice and sends the data to the emotion engine.

[1543] Step 23:

[1544] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, happiness).

[1545] Step 24:

[1546] The server generates a cooking menu according to the user's emotional state based on the data from the emotion engine.

[1547] Step 25:

[1548] The server sends the generated menu to the user's device and makes suggestions that match the user's emotional state.

[1549] Step 26:

[1550] The server periodically records the user's emotional data and performs trend analysis.

[1551] Step 27:

[1552] The server accumulates data based on the user's past emotional patterns to provide appropriate suggestions and menus that the user prefers.

[1553] Step 28:

[1554] The device then performs appropriate actions based on the user's emotional state (e.g., suggesting favorite recipes or sending encouraging messages).

[1555] Through the above processing steps, the present invention can efficiently manage ingredients and monitor elderly people all at once, and can also suggest menus that match the user's mood through emotion recognition.

[1556] Example 2

[1557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1558] Currently, there are systems on the market that manage ingredients in refrigerators and pantries and suggest cooking menus, but these do not detect anomalies based on the user's emotions or the opening and closing status of storage device doors. Therefore, there is a need for systems that can efficiently and low-costly support home cooking and monitor the elderly. In particular, there is a lack of systems that can suggest personalized menus based on the status of ingredients and the user's emotional state.

[1559] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1560] In this invention, the server includes means for installing a webcam in the storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images and generating an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device door, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, means for detecting the user's facial expression and voice and recognizing emotions, and means for proposing cooking menus based on the emotion data. This enables flexible and efficient ingredient management, cooking menu suggestions, and abnormality detection based on the user's emotions and storage status.

[1561] "Storage equipment" means a device for storing food or goods, examples of which include refrigerators, pantries, and cupboards.

[1562] A "webcam" is a camera device that captures and transmits images and videos over the Internet.

[1563] The "food list" is a list that lists the types and quantities of food extracted from the analyzed image data.

[1564] "Artificial intelligence means" refers to algorithms and machine learning models used for image analysis, data processing, menu suggestions, etc.

[1565] "Door open / close status" is data indicating the state of the door of the storage device, whether it is open or closed.

[1566] "Anomaly detection" is the process of recognizing behavior that deviates from normal opening and closing patterns and reporting the problem.

[1567] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1568] "Means for detecting facial expressions and voice" refers to hardware and software for recognizing the user's facial movements and voice and collecting them as data.

[1569] "Means for recognizing emotions" refers to an algorithm or system that analyzes detected facial and voice data to determine the emotional state of the user.

[1570] "Trend analysis" is the process of analyzing collected data to identify trends and patterns over a period of time.

[1571] This invention is a system that manages ingredients stored in storage devices, suggests cooking menus, monitors elderly people, and suggests menus based on their emotions. The system combines webcams, image analysis AI, and emotion recognition technology to provide an efficient and personalized experience.

[1572] Hardware and Software Configuration

[1573] 1. Webcam

[1574] Users place a webcam, such as the Logitech C920, in their refrigerator, pantry, or cupboard, capturing a full view of the contents of the storage device and recording daily changes in the ingredients.

[1575] 2. Image capture and transmission

[1576] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. The software used here is custom firmware, which provides the scheduled shooting and automatic sending functions.

[1577] 3. Image Analysis

[1578] The server uses image recognition artificial intelligence tools such as TensorFlow to analyze the received image, recognize ingredients in the image (e.g., milk, tomato, lettuce), and generate an ingredient list based on that information.

[1579] 4. Creating and managing ingredient lists

[1580] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[1581] 5. Recipe generation and suggestions

[1582] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, recipes for "tomato soup" and "lettuce salad" using "milk," "tomato," and "lettuce" are generated and sent to the user's device.

[1583] 6. Suggestion of necessary ingredients

[1584] If the server is running low on ingredients for a proposed menu, it generates a list of those ingredients and notifies the user. For example, it recognizes that the "chicken soup base" needed for "tomato soup" is missing, and notifies the user of that information.

[1585] 7. Monitoring door opening and closing status

[1586] The device (camera) monitors whether the storage device door is open or closed, and periodically sends the data to the server. The timing of the door opening and closing is recorded and compiled by the server.

[1587] 8. Trend Analysis and Anomaly Detection

[1588] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, a push notification is sent to the user's device.

[1589] 9. User Emotion Recognition

[1590] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[1591] 10. Emotion-Based Suggestions

[1592] The server generates a cooking menu based on the data from the emotion engine according to the user's emotional state. For example, if the user is feeling depressed, it will suggest a "herbal tea recipe" that has a relaxing effect. The generated menu is sent to the user's device.

[1593] 11. Recording and analyzing emotional data

[1594] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to suggest menus that the user prefers and make appropriate suggestions.

[1595] Examples and prompts

[1596] As a specific example, the following procedure is carried out.

[1597] Ingredient management and emotion recognition

[1598] User: Places a Logitech C920 webcam inside the refrigerator and sets it to automatically record at 8:00 AM every day.

[1599] Example prompt: "Set up your webcam and take a picture of the inside of your refrigerator every day at 8 AM."

[1600] Image capture and transmission

[1601] Device (camera): Take a picture at 8:00 and send it to the server.

[1602] Image analysis

[1603] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[1604] Emotion Recognition and Suggestions

[1605] Device (camera and microphone): Detects the user's facial expressions and voice data and sends them to the emotion engine.

[1606] Cooking menu suggestions

[1607] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[1608] Sample prompt: "We analyzed the image of your refrigerator and detected milk, tomatoes, and lettuce. We've added them to your current ingredient list."

[1609] Suggestion of necessary ingredients

[1610] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[1611] Emotion-based suggestions

[1612] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[1613] Example prompt: "The user's facial expression appears depressed. Suggest a relaxing herbal tea recipe."

[1614] Monitoring function implementation

[1615] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1616] Trend analysis and anomaly detection

[1617] Server: Conducts trend analysis and detects that the door normally opens and closes several times a day, but has not opened or closed for more than 24 hours. This anomaly is detected and a push notification is sent to the user's device.

[1618] Sample prompt: "The refrigerator hasn't been opened or closed for over 24 hours. Please check."

[1619] This system allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[1620] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1621] Processing Steps

[1622] Step 1: Setting up your webcam

[1623] Users place a webcam, such as the Logitech C920, on a refrigerator, pantry, or cupboard and adjust the angle and position to capture the entire interior. For example, place the webcam on top of the refrigerator and adjust it to a position that allows a full view of the interior.

[1624] Input: Webcam installation instructions, refrigerator location

[1625] Output: A webcam placed in a suitable position, ready to record.

[1626] Step 2: Take and send an image

[1627] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. Custom firmware is used to implement the scheduled shooting and automatic sending functions.

[1628] Input: Set shooting schedule, webcam location information

[1629] Output: Captured image data, image data transmission to server

[1630] Step 3: Image analysis

[1631] The server analyzes the received image using image recognition AI such as TensorFlow, which identifies ingredients in the image (e.g., milk, tomato, lettuce), and generates an ingredient list based on that information.

[1632] Input: Captured image data

[1633] Output: Ingredient list data as the analysis result

[1634] Step 4: Generate and manage your ingredient list

[1635] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[1636] Input: Ingredient list data as a result of image analysis

[1637] Output: Ingredient list stored in the database, provided to the user's device

[1638] Step 5: Recipe generation and suggestions

[1639] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, a recipe for "tomato soup" or "lettuce salad" using "milk," "tomato," and "lettuce" is generated and sent to the user's device.

[1640] Input: List of ingredients in the database

[1641] Output: Generated recipes, suggestions to user terminal

[1642] Step 6: Suggesting ingredients

[1643] If the server is missing any ingredients needed for the proposed menu, it generates a list of those ingredients and notifies the user terminal. For example, it recognizes that "chicken soup base" needed for "tomato soup" is missing and notifies the user of that information.

[1644] Input: Cooking menu, current ingredients list

[1645] Output: List of missing ingredients, notification to user device

[1646] Step 7: Monitor the door opening and closing status

[1647] The device (camera) monitors the opening and closing status of the storage device door and periodically transmits the data to the server. The timing of the door opening and closing is recorded and transmitted to the server.

[1648] Input: Door open / close status, timestamp

[1649] Output: Open / close data sent to the server

[1650] Step 8: Trend analysis and anomaly detection

[1651] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device.

[1652] Input: Door opening / closing data

[1653] Output: Anomaly detection results, notification to user device

[1654] Step 9: Recognizing User Emotions

[1655] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[1656] Input: facial expression data, voice data

[1657] Output: Emotion recognition result

[1658] Step 10: Emotion-Based Suggestions

[1659] The server generates a cooking menu based on the user's emotional state based on the data from the emotion engine. For example, if the user is feeling depressed, it will suggest a relaxing "herbal tea recipe." The generated menu is sent to the user's device.

[1660] Input: Emotion recognition results, list of ingredients

[1661] Output: Emotion-based cooking menu, sent to user device

[1662] Step 11: Record and analyze emotional data

[1663] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to provide the user with menus they prefer and make appropriate suggestions.

[1664] Input: Periodically collected emotion data

[1665] Output: Sentiment trend analysis results, personalized menu suggestion data

[1666] (Application example 2)

[1667] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1668] In the past, managing ingredients in the refrigerator or pantry and suggesting cooking menus was often done manually, making it difficult for users to efficiently manage ingredients and select cooking menus. Furthermore, there were issues with systems that were unable to provide services that monitored elderly people or responded to the user's emotions. Furthermore, even in brick-and-mortar stores, there was a demand for automated services based on customers' emotions and food inventory management, but these systems were difficult to implement.

[1669] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1670] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, emotion recognition means for recognizing the user's facial expressions and voice and analyzing the data, and means for proposing cooking menus based on the emotion data obtained from the emotion recognition means. This enables automation of ingredient management in brick-and-mortar stores and the proposing of personalized cooking menus based on the user's emotions.

[1671] A "webcam" is a camera device that captures digital images or video and transmits them in real time over a computer or network.

[1672] The "ingredient list" is a list showing ingredients and items present in the refrigerator, pantry, and cupboard.

[1673] An "artificial intelligence means" is a system or software that uses computer algorithms to analyze data and mimic human intellectual activity.

[1674] "Door opening / closing status" is information indicating the status of the opening and closing of the refrigerator, pantry, and cupboard doors.

[1675] An "anomaly" is a condition or action that deviates from a normal pattern or expected behavior.

[1676] "User terminal" means a computer device operated by a user to receive and display information.

[1677] "Emotion recognition means" refers to a system or software that analyzes a user's facial expressions and voice to identify their emotions.

[1678] "Emotion data" is data that represents the emotional state of the user analyzed from facial expressions and voice.

[1679] A "personalized cooking menu" is a cooking menu suggested to suit the user's individual preferences and condition.

[1680] This system uses a webcam to capture images of food storage areas such as kitchens and stockrooms in brick-and-mortar stores, analyzes the images to generate and manage ingredient lists, and suggests cooking menus based on customer sentiment. This system automates food management in brick-and-mortar stores and improves the customer experience.

[1681] System Configuration

[1682] 1. Webcam installation

[1683] User (store clerk): Install a webcam in the kitchen or stockroom of the physical store and adjust the angle and position to capture the entire scene.

[1684] 2. Image capture and transmission

[1685] Terminal (camera): Takes images at regular intervals (e.g., every two hours) and sends them to a server via the Internet.

[1686] 3. Image Analysis

[1687] Server: Analyzes the received image using AI (e.g., AWS Rekognition or Google Cloud Vision), recognizes the ingredients in the image, and generates an ingredient list.

[1688] 4. Creating and managing ingredient lists

[1689] Server: Stores the generated ingredient list in a database and updates and manages it as needed.

[1690] Cooking menu suggestions

[1691] 5. Cooking menu generation

[1692] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to and displayed on the smartphones or smart glasses of customers and store staff.

[1693] 6. Suggestions for missing ingredients

[1694] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[1695] Monitoring function

[1696] 7. Monitoring door opening and closing status

[1697] Terminal (camera): Monitors whether refrigerators and pantry doors are open or closed in physical stores, and periodically sends this data to the server.

[1698] 8. Trend Analysis and Anomaly Detection

[1699] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[1700] Emotion recognition function

[1701] 9. User Emotion Recognition

[1702] Device: Uses a camera and microphone to detect the customer's facial expressions and voice, and sends the data to an emotion engine (e.g., Microsoft Azure Cognitive Services' Face API).

[1703] Server: Analyzes the data received from the emotion engine and recognizes the emotional state of the customer (e.g., joy, anger, sadness, or happiness).

[1704] 10. Emotion-Based Suggestions

[1705] Server: Generates cooking menus based on emotion data and sends them to the customers' and staff's devices. For example, if a customer is feeling stressed, the server suggests menus that will have a relaxing effect.

[1706] 11. Recording and analyzing emotional data

[1707] Server: Periodically records user emotion data and performs trend analysis.

[1708] Server: Accumulates data to determine customers' preferred menu items and make appropriate suggestions based on past emotional patterns.

[1709] Specific examples

[1710] Ingredient management and emotion recognition

[1711] User (Store Staff): Set up a webcam in the kitchen to capture images of the food storage area at 8:00 AM and 2:00 PM every day.

[1712] Terminal (camera): Takes a picture at the set time and sends it to the server.

[1713] Server: Analyzes the image, generates an ingredient list, and stores it in a database.

[1714] Terminal: The camera and microphone also detect the customer's facial expressions and voice data, which are then sent to the emotion engine.

[1715] Cooking menu suggestions

[1716] Server: Based on the ingredients list and the analysis results of the emotion engine, the server proposes multiple menu items and sends them to the user's device. The customer selects a menu item.

[1717] Suggestions for ingredients that are in short supply

[1718] Server: If any ingredients required for the selected menu are missing, it generates a list of them and notifies the user terminal of that fact.

[1719] Emotion-based suggestions

[1720] Server: If a customer's expression is deemed depressed, the server will suggest, for example, a relaxing "herbal tea recipe."

[1721] Monitoring function implementation

[1722] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1723] Server: Conducts trend analysis and detects when the refrigerator has not been opened or closed for 24 hours or more. If an abnormality is detected, a push notification is sent to the user's device. The push notification displays the message, "The refrigerator has not been opened or closed for 24 hours or more. Please check."

[1724] Prompt Sentence Examples

[1725] By inputting the following prompts into the generative AI model, it is possible to suggest cooking menus that correspond to the customer's emotions.

[1726] Example prompt sentence:

[1727] Based on the user's emotional data, please create and suggest a menu that will have a relaxing effect. If the emotional state is "stressed" or "depressed," please explain what menu should be suggested and why.

[1728] With this configuration, the system can automate food ingredient management in physical stores and propose personalized cooking menus based on customers' emotions, providing an efficient and satisfying customer experience.

[1729] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1730] Step 1:

[1731] Camera installation and image capture

[1732] The user (store clerk) installs a webcam in the kitchen or stockroom of a physical store and adjusts the angle and position to capture the entire scene. The webcam takes images at regular intervals (e.g., every two hours) and generates image data. The input is the camera's installation position and shooting time settings, and the output is the captured image data.

[1733] Step 2:

[1734] Image transmission

[1735] The terminal (webcam) sends the captured image to a server via the Internet. The input is the captured image data, and the output is the image data sent to the server.

[1736] Step 3:

[1737] Image analysis

[1738] The server analyzes the received image data using an AI model (e.g., AWS Rekognition or Google Cloud Vision) to recognize the ingredients present in the image. It generates an ingredient list based on the analysis results. The input is the transmitted image data, and the output is the generated ingredient list.

[1739] Step 4:

[1740] Save Ingredient List

[1741] The server stores the generated ingredient list in a database. The input is the generated ingredient list, and the output is the ingredient list stored in the database.

[1742] Step 5:

[1743] Cooking menu suggestions

[1744] The server uses AI to generate cooking menus based on the ingredient list stored in the database and sends them to the user's device. The input is the stored ingredient list, and the output is the proposed cooking menu.

[1745] Step 6:

[1746] Notification of insufficient ingredients

[1747] If any ingredients required for the proposed cooking menu are missing, the server generates a list of those ingredients and notifies the user of the missing ingredients. The input is the proposed cooking menu and the list of ingredients, and the output is the list of missing ingredients and a notification of that list.

[1748] Step 7:

[1749] Door opening / closing status monitoring

[1750] The terminal (web camera) monitors the opening and closing status of the refrigerator and pantry doors and periodically transmits the data to the server. The input is the door opening and closing behavior, and the output is the opening and closing status data.

[1751] Step 8:

[1752] Anomaly detection

[1753] The server performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device. The input is the opening / closing status data, and the output is the abnormality detection and notification.

[1754] Step 9:

[1755] emotion recognition

[1756] The device uses a camera and microphone to detect the facial expressions and voices of customers and sends the data to an emotion engine (e.g., Microsoft Azure Cognitive Services Face API). The input is facial expression and voice data, and the output is emotion data.

[1757] Step 10:

[1758] Emotional Data Analysis

[1759] The server analyzes the data received from the emotion engine and recognizes the emotional state of the customer. The input is the emotion data, and the output is the analysis result.

[1760] Step 11:

[1761] Emotion-based menu suggestions

[1762] The server then proposes cooking menus to the customers' and store staff's devices based on the results of the emotion data analysis. For example, if a customer is feeling stressed, it will propose a menu with a relaxing effect. The input is the emotion data analysis results, and the output is a cooking menu proposal based on the customer's emotion.

[1763] Step 12:

[1764] Emotional data recording and analysis

[1765] The server periodically records users' emotional data and performs trend analysis. Based on past emotional patterns, the server accumulates data to help customers choose menu items they like and make appropriate suggestions. The input is emotional data, and the output is the recorded emotional data and the analysis results.

[1766] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1767] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1768] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1769] [Fourth embodiment]

[1770] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1771] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1772] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1773] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1774] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1775] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1776] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1777] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1778] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1779] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1780] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1781] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1782] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1783] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, and generates and manages ingredient lists. It also has cooking menu suggestions and monitoring functions. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[1784] System Configuration

[1785] 1. Webcam

[1786] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[1787] 2. Image capture and transmission

[1788] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[1789] 3. Image Analysis

[1790] Server: Uses artificial intelligence (such as GPT-4) to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[1791] 4. Creating and managing ingredient lists

[1792] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[1793] Cooking menu suggestions

[1794] 5. Cooking menu generation

[1795] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[1796] 6. Suggesting inadequate ingredients

[1797] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[1798] Monitoring function

[1799] 7. Monitoring door opening and closing status

[1800] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[1801] 8. Trend Analysis and Anomaly Detection

[1802] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[1803] Specific examples

[1804] Ingredients management

[1805] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[1806] Device (camera): Take a picture at 8:00 and send it to the server.

[1807] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[1808] Cooking menu suggestions

[1809] User: Checks grocery list on device before lunch.

[1810] Server: Based on the ingredients list, suggests menu items such as "Tomato soup" and "Lettuce salad." The user selects "Tomato soup."

[1811] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[1812] Monitoring function implementation

[1813] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1814] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[1815] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[1816] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to keep an eye on elderly people.

[1817] The processing flow will be explained below.

[1818] Program processing flow

[1819] Imaging and analysis

[1820] Step 1:

[1821] The device (camera) takes pictures of the inside of the refrigerator and pantry at regular intervals (e.g., every hour).

[1822] Step 2:

[1823] The captured image is sent from the device (camera) to the server.

[1824] Step 3:

[1825] The server receives the image and temporarily stores it in a buffer.

[1826] Image analysis and ingredient list generation

[1827] Step 4:

[1828] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[1829] Step 5:

[1830] The server identifies ingredients in the image and generates an ingredient list.

[1831] Step 6:

[1832] The generated ingredient list is stored in a database on the server.

[1833] Step 7:

[1834] The server transmits the stored ingredient list to the user's terminal.

[1835] Cooking menu suggestions

[1836] Step 8:

[1837] The user checks the ingredient list on the device.

[1838] Step 9:

[1839] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[1840] Step 10:

[1841] The server transmits the generated cooking menu to the user's terminal.

[1842] Step 11:

[1843] The user checks the presented cooking menu on the terminal and selects one.

[1844] Suggestions for ingredients that are in short supply

[1845] Step 12:

[1846] The server will list any missing ingredients based on the menu selected.

[1847] Step 13:

[1848] The server sends a list of ingredients that are in short supply to the user's terminal.

[1849] Step 14:

[1850] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[1851] Monitoring function

[1852] Step 15:

[1853] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[1854] Step 16:

[1855] The terminal (camera) periodically sends door opening / closing information to the server.

[1856] Step 17:

[1857] The server accumulates the opening and closing data received and performs trend analysis.

[1858] Step 18:

[1859] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[1860] Step 19:

[1861] If the server detects an abnormality, it generates warning information.

[1862] Step 20:

[1863] The server generates a push notification of the warning information to the user's device.

[1864] Step 21:

[1865] The user receives a push notification to check the status.

[1866] Through the above processing steps, the present invention makes it possible to efficiently manage ingredients and monitor elderly people all at once.

[1867] Example 1

[1868] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] The need for food management and home cooking assistance for modern families and the elderly is increasing, but there are few systems that can accurately grasp food inventory status and efficiently suggest cooking menus.Furthermore, there are almost no systems that have a function to monitor the elderly, so there is a need for a low-cost, easy-to-use system that combines both functions.

[1870] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1871] In this invention, the server includes means for installing an image capture device in a storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device, and means for detecting abnormalities based on the opening and closing status and notifying a user terminal. This allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people.

[1872] An "image capture device" is a device used to capture images of the inside of a refrigerator, pantry, or cupboard, and generally refers to a webcam or digital camera.

[1873] "Storage equipment" refers to equipment for storing food and ingredients, and specifically includes refrigerators, pantries, and cupboards.

[1874] "Artificial intelligence means" refers to technologies for image analysis and data processing, particularly machine learning algorithms and generative AI models.

[1875] A "database" is a digital system for organizing and storing information such as ingredient lists, and includes SQL databases and NoSQL databases.

[1876] A "user terminal" is a device operated by system users, and includes smartphones, PCs, and tablets.

[1877] "Image analysis" refers to the process of analyzing images captured by an image capture device using software or artificial intelligence means.

[1878] The "ingredient list" is a list of ingredients in the refrigerator or pantry, generated based on the analyzed image information.

[1879] "Cooking menu" refers to recipes and cooking methods suggested based on an ingredient list.

[1880] "Opening and closing status" refers to the state and frequency of opening and closing the doors of the refrigerator, pantry, and cupboard.

[1881] "Anomaly detection" is the process of identifying unusual patterns or behaviors and recognizing problems.

[1882] "Push notification" is a function that detects anomalies and sends important information to user devices in real time.

[1883] The present invention is a system that uses an image capture device to manage ingredients stored in a storage device and suggests appropriate cooking menus, and also has a monitoring function for elderly people. Specific embodiments of this system will be described below.

[1884] System Configuration and Operation

[1885] 1. Installation of imaging device

[1886] A user places an image capture device (such as a webcam) in a storage device such as a refrigerator, pantry, or cupboard, and adjusts the angle and position of the device to capture the entire food item in the storage device.

[1887] 2. Take and send images

[1888] The terminal (image capture device) takes pictures of the storage device at a set time. For example, it can be set to take pictures of the inside of the refrigerator every morning at 8:00. The images are connected to the Internet via a Wi-Fi module and sent to a server.

[1889] 3. Image Analysis

[1890] The server receives the image and performs image analysis. It uses artificial intelligence (e.g., GPT-4, TensorFlow, etc.) to identify and recognize ingredients in the image. For example, if "milk, tomato, lettuce" is recognized, it is added to the ingredients list.

[1891] 4. Creating and managing ingredient lists

[1892] The server generates an ingredient list based on the recognized ingredient information and stores it in a database. Duplicate ingredients are automatically omitted. This allows the user device to always receive the latest ingredient list.

[1893] 5. Cooking menu suggestions

[1894] The server generates a cooking menu based on the saved ingredient list. It inputs prompts to a generative AI model (e.g., GPT-4) to suggest appropriate recipes. An example of a prompt is "Please suggest a recommended cooking menu based on the current ingredient list."

[1895] 6. Suggestions for missing ingredients

[1896] If the server detects that a suggested menu item is missing, it lists the ingredients that need to be purchased and notifies the user. For example, if a recipe for "tomato soup" requires "chicken soup base" but is in short supply, the server notifies the user terminal, "You are running low on chicken soup base. Please buy more."

[1897] 7. Monitoring door opening and closing status

[1898] The terminal (image capture device) is attached with a sensor on the door of the storage device to monitor the door opening and closing status. The opening and closing data is periodically recorded and sent to the server.

[1899] 8. Trend Analysis and Anomaly Detection

[1900] The server performs trend analysis based on the received door opening / closing data. It learns normal opening / closing frequencies and patterns, and detects abnormal patterns or cases where the door has not been opened / closed for a certain period of time. For example, if the door is normally opened / closed four times a day, but there has been no opening / closing for more than 24 hours, this is detected as an abnormality. If this abnormality is detected, a push notification is sent to the user's device saying, "The refrigerator has not been opened / closed for more than 24 hours. Please check."

[1901] This allows users to efficiently manage ingredients and receive appropriate cooking menus. It also makes it possible to simultaneously monitor elderly people, improving safety.

[1902] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1903] Step 1: Installing the imaging device

[1904] The user places an image capture device, such as a webcam, on a refrigerator, pantry, or cupboard at an appropriate angle and position so that the entire contents of the storage device are captured. For example, the camera can be attached to the top of the refrigerator and point downward to capture the interior.

[1905] Step 2: Take and send an image

[1906] The terminal (image capture device) automatically captures images in the storage device at a set time. A timer setting (e.g., every morning at 8:00) is given as input. The captured images are connected to the Internet via Wi-Fi and sent to a server. As output, a JPEG image file is generated.

[1907] Step 3: Image analysis

[1908] The server receives the transmitted image and performs image analysis. As input, a JPEG image file is provided. Specifically, artificial intelligence tools (e.g., GPT-4 or TensorFlow) are used to identify and recognize the ingredients in the image. An image processing algorithm is applied to process the data, and as output, a list of ingredient information (e.g., "Milk, Tomato, Lettuce") is generated.

[1909] Step 4: Generate and manage your ingredient list

[1910] The server generates an ingredient list based on the analyzed ingredient information and stores it in a database. The generated ingredient information list is given as input. Data calculations include removing duplicate items and reorganizing the list. An updated ingredient list is generated as output and synchronized with the user's device.

[1911] Step 5: Cooking menu suggestions

[1912] The server generates a cooking menu based on the saved ingredient list. The latest ingredient list is given as input. A prompt is input to the generative AI model (e.g., GPT-4) to suggest an appropriate recipe. For example, the prompt is "Please suggest a recommended cooking menu based on the current ingredient list." As output, multiple cooking menus (e.g., "Tomato soup," "Lettuce salad") are generated and sent to the user device.

[1913] Step 6: Suggest missing ingredients

[1914] The server detects if any ingredients required for the proposed menu are missing. As input, the proposed menu and a list of ingredients are given. As data processing, the ingredients required for the menu are compared with the currently available ingredients. As output, a list of missing ingredients (e.g., "chicken soup base") is generated and notified to the user's terminal.

[1915] Step 7: Monitor the door opening and closing status

[1916] The terminal (image capture device) uses a sensor installed on the door of the storage device to monitor the door's opening and closing status. As input, a timestamp of the door opening and closing is recorded. As data processing, a log of the opening and closing status is generated and sent to a server via the Internet. As output, opening and closing log data is generated.

[1917] Step 8: Trend analysis and anomaly detection

[1918] The server performs trend analysis based on the received opening / closing log data. The door opening / closing log data is given as input. Data calculations learn the normal opening / closing frequency and patterns, and detect abnormal patterns. For example, if the door is normally opened and closed four times a day, but there is no opening / closing for more than 24 hours, this is considered an abnormality. The output is the result of the abnormality detection (e.g., "The refrigerator has not been opened or closed for more than 24 hours. Please check"), and a push notification is sent to the user's device.

[1919] (Application example 1)

[1920] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1921] Conventional systems for managing the contents of refrigerators, pantries, and cupboards have functions such as managing ingredients, suggesting cooking menus, and monitoring opening and closing status, but there are few ways to provide these functions in a visually easy-to-understand manner for elderly people. It is also difficult to immediately visually check information about shortages of ingredients or monitoring functions. Therefore, efficiently managing ingredients and safety at home for elderly people has been a challenge.

[1922] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1923] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the open / close status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the open / close status and notifying a user terminal, means for displaying the ingredient list and the open / close status of the doors on a head-mounted display, and means for visually notifying the user via the information notified and suggested on the head-mounted display. This allows even elderly people to visually manage ingredients, monitor the open / close status, and check safety.

[1924] A "webcam" is a camera device that can capture and transmit video over the Internet.

[1925] "Artificial intelligence means" refers to AI algorithms or software that run on a computer and learn, infer, and analyze data input to it.

[1926] A "grocery list" refers to a list of the types and quantities of ingredients in your refrigerator, pantry, or cupboard.

[1927] "Cooking menu" refers to a list of recipes for dishes that can be made using specific ingredients.

[1928] The "open / close status monitoring means" is a device or system that monitors the opening and closing of refrigerator, pantry, and cupboard doors and acquires the data.

[1929] An "anomaly detection means" is a system or algorithm that determines if an unusual pattern or problem has occurred based on acquired data.

[1930] A "user terminal" is an electronic device used by a user to receive, check, and operate information, and includes smartphones, tablets, and personal computers.

[1931] A "head-mounted display" is a display device that a user wears on their head to display visual information.

[1932] "Notification means" refers to a function or device for conveying specific information to a user, and includes push notifications, audio alarms, etc.

[1933] A "buffer" refers to a memory area that temporarily stores data, enabling high-speed reading and writing of data.

[1934] A "database" is a system or software for systematically storing data and for efficient searching and retrieval.

[1935] A "visual notification means" is a device or system for displaying information so that it can be perceived visually by a user.

[1936] This invention is a system that uses a webcam to capture images of the inside of refrigerators, pantries, and cupboards, analyzes the images, and generates and manages an ingredient list, providing visual information to users, including the elderly. In particular, by combining image analysis using artificial intelligence with a visual notification function using a head-mounted display, it provides a mechanism that allows users to quickly check the status of ingredients and any abnormalities.

[1937] System Configuration

[1938] 1. Webcam

[1939] User: Place a webcam on your refrigerator, pantry, or cupboard, and angle and position it to capture the entire area.

[1940] 2. Image capture and transmission

[1941] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to the server, allowing you to keep up to date with the latest food information.

[1942] 3. Image Analysis

[1943] Server: Uses a generative AI model (such as GPT-4) to analyze the submitted image, recognize ingredients in the image, and generate an ingredient list.

[1944] 4. Creating and managing ingredient lists

[1945] Server: Generates an ingredient list based on the analysis results and stores it in a database. The generated list can be viewed on the user's head-mounted display or other devices.

[1946] 5. Cooking menu suggestions

[1947] Server: Based on the stored ingredient list, the server uses a generative AI model to suggest cooking menus. The suggested menus are sent to the user's head-mounted display, where they can be selected and used.

[1948] 6. Suggestions for missing ingredients

[1949] Server: If any ingredients required for the proposed menu are missing, the server generates a list of those ingredients and notifies the head-mounted display and user device, suggesting that the missing ingredients be purchased.

[1950] Monitoring function

[1951] 7. Monitoring door opening and closing status

[1952] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[1953] 8. Trend Analysis and Anomaly Detection

[1954] Server: Performs trend analysis based on the received door opening / closing data to detect abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, it notifies the head-mounted display and other user devices via push notification.

[1955] Specific examples

[1956] Ingredients management

[1957] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[1958] Device (camera): Take a picture at 8:00 and send it to the server.

[1959] Server: Analyzes the image and generates a list of ingredients (e.g., "milk, tomato, lettuce"). The list is stored in a database and provided to the head-mounted display and other user devices.

[1960] Cooking menu suggestions

[1961] User: Checks the grocery list on the head-mounted display before lunch.

[1962] Server: Based on the ingredients list, suggests a cooking menu (e.g., "tomato soup" or "lettuce salad"). The user selects "tomato soup."

[1963] Server: Detects that the "chicken soup mix" required for the "tomato soup" is missing, and notifies the head-mounted display and other user devices of this fact.

[1964] Monitoring function implementation

[1965] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[1966] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[1967] Server: Detects this anomaly and sends a push notification to the head-mounted display and other user devices. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[1968] Prompt Sentence Examples

[1969] "Analyze an image of the inside of a refrigerator and generate a list of ingredients including milk, tomatoes, and lettuce. Also, tell me what ingredients are missing to make tomato soup."

[1970] This system allows users, including the elderly, to efficiently manage and cook ingredients while ensuring safety.

[1971] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1972] Step 1:

[1973] Webcam installation and image capture settings

[1974] Users can install web cameras on refrigerators, pantries, and cupboards, adjust the angle and position to capture the entire area, and set the timing of the camera captures.

[1975] Input: Physical location of the camera and timing settings.

[1976] Output: After the setup is complete, the camera will take an image at the specified time.

[1977] Step 2:

[1978] Taking and sending images

[1979] The device (camera) takes pictures of the inside of the refrigerator or pantry at a set time and sends them to a server via the Internet.

[1980] Input: The image captured by the camera.

[1981] Data processing: Generation of image files and transmission over the Internet.

[1982] Output: The image file sent to the server.

[1983] Step 3:

[1984] Image analysis

[1985] The server uses a generative AI model (e.g., GPT-4) to analyze the received image, recognize ingredients in the image, and generate an ingredient list.

[1986] Input: The submitted image file.

[1987] Data Computing: Image recognition and ingredient list generation using generative AI models.

[1988] Output: A list of ingredients (e.g. "Milk, Tomato, Lettuce").

[1989] Step 4:

[1990] Save your grocery list

[1991] The server stores the generated ingredient list in a database.

[1992] Input: Ingredient list.

[1993] Data processing: Writing list data to a database.

[1994] Output: Ingredient list stored in database.

[1995] Step 5:

[1996] Cooking menu suggestions

[1997] The server uses a generative AI model to suggest cooking menus based on the stored ingredient list and transmits the results to the head-mounted display and other user devices.

[1998] Input: Saved ingredient list.

[1999] Data Computing: Menu suggestions using generative AI models and results transmission.

[2000] Output: A cooking menu displayed on the user's device (e.g., "Tomato soup" or "Lettuce salad").

[2001] Step 6:

[2002] Suggestions for ingredients that are in short supply

[2003] If any ingredients required for the proposed menu are missing, the server generates a list of the missing ingredients and notifies the head-mounted display and other user terminals of the list.

[2004] Input: Proposed cooking menu and current ingredients list.

[2005] Data calculation: Determine shortages of ingredients and generate notification lists.

[2006] Output: Notification of missing ingredients (e.g. "You are running low on chicken soup base").

[2007] Step 7:

[2008] Door opening / closing status monitoring

[2009] The terminal (camera) monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[2010] Input: Signal from door open / close sensor.

[2011] Data processing: Collection and transmission of door opening and closing data.

[2012] Output: Door open / close data sent to the server.

[2013] Step 8:

[2014] Trend analysis and anomaly detection

[2015] The server performs trend analysis based on the received door opening and closing data to detect abnormal patterns or cases where the door has not been opened or closed within a certain period of time. If an abnormality is detected, a push notification is sent to the user's head-mounted display and other devices.

[2016] Input: Door opening and closing data.

[2017] Data arithmetic: Applying trend analysis and anomaly detection algorithms.

[2018] Output: Push notification if an abnormality is detected (e.g. "The refrigerator has not been opened or closed for more than 24 hours. Please check it.").

[2019] Step 9:

[2020] Visual Notifications

[2021] The server sends various information (ingredient list, cooking menu, missing ingredients, abnormality notification) to the head-mounted display and notifies the user visually.

[2022] Input: Generated information (ingredient list, cooking menu, missing ingredients, abnormality notification).

[2023] Data processing: information format conversion and transmission procedures.

[2024] Output: Information visually displayed on a head-mounted display.

[2025] Through the above processing steps, the user can efficiently manage, cook, and check the safety of ingredients.

[2026] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2027] This system uses a webcam to capture images of the inside of refrigerators, pantries, cupboards, etc., analyzes the images, creates and manages ingredient lists, and also has cooking menu suggestions and monitoring functions, and even recognizes the user's emotions to suggest appropriate menus. The system aims to provide efficient and low-cost support for home cooking and monitoring for the elderly.

[2028] System Configuration

[2029] 1. Webcam

[2030] User: Place webcams in your refrigerator, pantry, or cupboards, angled and positioned to capture the entire area.

[2031] 2. Image capture and transmission

[2032] Terminal (camera): Takes pictures of the inside of the refrigerator and pantry at regular intervals and sends them to a server via the Internet.

[2033] 3. Image Analysis

[2034] Server: Uses artificial intelligence to analyze the received image. The goal of the analysis is to recognize ingredients in the image and generate an ingredient list based on that information.

[2035] 4. Creating and managing ingredient lists

[2036] Server: Generates an ingredient list based on the image analysis results, stores this list in a database, and provides the generated ingredient list to the user's device.

[2037] Cooking menu suggestions

[2038] 5. Cooking menu generation

[2039] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to the user's device, where they can be selected and used.

[2040] 6. Suggesting inadequate ingredients

[2041] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[2042] Monitoring function

[2043] 7. Monitoring door opening and closing status

[2044] Terminal (camera): Monitors whether the refrigerator and pantry doors are open or closed, and periodically sends the data to the server.

[2045] 8. Trend Analysis and Anomaly Detection

[2046] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[2047] Emotion recognition function

[2048] 9. User Emotion Recognition

[2049] Device: Uses a camera and microphone to detect the user's facial expressions and voice, and sends the data to the emotion engine.

[2050] Emotion engine: Analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[2051] 10. Emotion-Based Suggestions

[2052] Server: Based on data from the emotion engine, it generates a cooking menu according to the user's emotional state.

[2053] Server: The server sends the generated menu to the user's device and makes suggestions that match their emotional state. For example, if the user is feeling stressed, a menu with a relaxing effect will be selected.

[2054] 11. Recording and analyzing emotional data

[2055] Server: Periodically records user emotion data and performs trend analysis.

[2056] Server: Accumulates data to provide users with menu preferences and appropriate suggestions based on their past emotional patterns.

[2057] Specific examples

[2058] Ingredient management and emotion recognition

[2059] User: Sets the webcam in the refrigerator to take a picture of the inside of the refrigerator every day at 8 AM.

[2060] Device (camera): Take a picture at 8:00 and send it to the server.

[2061] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[2062] Device: The camera and microphone also detect the user's facial expressions and voice data, and send them to the emotion engine.

[2063] Cooking menu suggestions

[2064] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[2065] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[2066] Emotion-based suggestions

[2067] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[2068] Monitoring function implementation

[2069] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[2070] Server: Performs trend analysis and detects that a server normally opens and closes several times a day, but has not opened or closed for more than 24 hours.

[2071] Server: Detects this anomaly and sends a push notification to the user's device. The push notification displays, "The refrigerator has not been opened or closed for more than 24 hours. Please check."

[2072] This system allows users to efficiently manage ingredients and enjoy appropriate cooking menus, and also makes it easy to monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[2073] The processing flow will be explained below.

[2074] Program processing flow

[2075] Imaging and analysis

[2076] Step 1:

[2077] The device (camera) takes pictures of the refrigerator and pantry at regular intervals each day (e.g., 8:00 a.m.).

[2078] Step 2:

[2079] The captured image is sent from the device (camera) to the server.

[2080] Step 3:

[2081] The server receives the image and temporarily stores it in a buffer.

[2082] Step 4:

[2083] The server uses artificial intelligence such as GPT-4 to analyze the received images.

[2084] Step 5:

[2085] The server identifies ingredients in the image and generates an ingredient list.

[2086] Step 6:

[2087] The generated ingredient list is stored in a database on the server.

[2088] Step 7:

[2089] The server transmits the stored ingredient list to the user's terminal.

[2090] Cooking menu suggestions

[2091] Step 8:

[2092] The user checks the ingredient list on the device.

[2093] Step 9:

[2094] The server uses artificial intelligence to generate a cooking menu based on the stored ingredient list.

[2095] Step 10:

[2096] The server transmits the generated cooking menu to the user's terminal.

[2097] Step 11:

[2098] The user checks the presented cooking menu on the terminal and selects one.

[2099] Suggestions for ingredients that are in short supply

[2100] Step 12:

[2101] The server will list any missing ingredients based on the menu selected.

[2102] Step 13:

[2103] The server sends a list of ingredients that are in short supply to the user's terminal.

[2104] Step 14:

[2105] The user checks the list of ingredients they are running low on and makes the necessary purchases.

[2106] Monitoring function

[2107] Step 15:

[2108] The terminal (camera) constantly monitors whether the refrigerator and pantry doors are open or closed.

[2109] Step 16:

[2110] The terminal (camera) periodically sends door opening / closing information to the server.

[2111] Step 17:

[2112] The server accumulates the opening and closing data received and performs trend analysis.

[2113] Step 18:

[2114] Detects abnormal opening and closing patterns or when the server has not opened or closed for a certain period of time.

[2115] Step 19:

[2116] If the server detects an abnormality, it generates warning information.

[2117] Step 20:

[2118] The server generates a push notification of the warning information to the user's device.

[2119] Step 21:

[2120] The user receives a push notification to check the status.

[2121] Emotion recognition function

[2122] Step 22:

[2123] The device uses a camera and microphone to detect the user's facial expressions and voice and sends the data to the emotion engine.

[2124] Step 23:

[2125] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, happiness).

[2126] Step 24:

[2127] The server generates a cooking menu according to the user's emotional state based on the data from the emotion engine.

[2128] Step 25:

[2129] The server sends the generated menu to the user's device and makes suggestions that match the user's emotional state.

[2130] Step 26:

[2131] The server periodically records the user's emotional data and performs trend analysis.

[2132] Step 27:

[2133] The server accumulates data based on the user's past emotional patterns to provide appropriate suggestions and menus that the user prefers.

[2134] Step 28:

[2135] The device then performs appropriate actions based on the user's emotional state (e.g., suggesting favorite recipes or sending encouraging messages).

[2136] Through the above processing steps, the present invention can efficiently manage ingredients and monitor elderly people all at once, and can also suggest menus that match the user's mood through emotion recognition.

[2137] Example 2

[2138] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2139] Currently, there are systems on the market that manage ingredients in refrigerators and pantries and suggest cooking menus, but these do not detect anomalies based on the user's emotions or the opening and closing status of storage device doors. Therefore, there is a need for systems that can efficiently and low-costly support home cooking and monitor the elderly. In particular, there is a lack of systems that can suggest personalized menus based on the status of ingredients and the user's emotional state.

[2140] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2141] In this invention, the server includes means for installing a webcam in the storage device and capturing images of the inside of the storage device, artificial intelligence means for analyzing the captured images and generating an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the storage device door, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, means for detecting the user's facial expression and voice and recognizing emotions, and means for proposing cooking menus based on the emotion data. This enables flexible and efficient ingredient management, cooking menu suggestions, and abnormality detection based on the user's emotions and storage status.

[2142] "Storage equipment" means a device for storing food or goods, examples of which include refrigerators, pantries, and cupboards.

[2143] A "webcam" is a camera device that captures and transmits images and videos over the Internet.

[2144] The "food list" is a list that lists the types and quantities of food extracted from the analyzed image data.

[2145] "Artificial intelligence means" refers to algorithms and machine learning models used for image analysis, data processing, menu suggestions, etc.

[2146] "Door open / close status" is data indicating the state of the door of the storage device, whether it is open or closed.

[2147] "Anomaly detection" is the process of recognizing behavior that deviates from normal opening and closing patterns and reporting the problem.

[2148] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[2149] "Means for detecting facial expressions and voice" refers to hardware and software for recognizing the user's facial movements and voice and collecting them as data.

[2150] "Means for recognizing emotions" refers to an algorithm or system that analyzes detected facial and voice data to determine the emotional state of the user.

[2151] "Trend analysis" is the process of analyzing collected data to identify trends and patterns over a period of time.

[2152] This invention is a system that manages ingredients stored in storage devices, suggests cooking menus, monitors elderly people, and suggests menus based on their emotions. The system combines webcams, image analysis AI, and emotion recognition technology to provide an efficient and personalized experience.

[2153] Hardware and Software Configuration

[2154] 1. Webcam

[2155] Users place a webcam, such as the Logitech C920, in their refrigerator, pantry, or cupboard, capturing a full view of the contents of the storage device and recording daily changes in the ingredients.

[2156] 2. Image capture and transmission

[2157] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. The software used here is custom firmware, which provides the scheduled shooting and automatic sending functions.

[2158] 3. Image Analysis

[2159] The server uses image recognition artificial intelligence tools such as TensorFlow to analyze the received image, recognize ingredients in the image (e.g., milk, tomato, lettuce), and generate an ingredient list based on that information.

[2160] 4. Creating and managing ingredient lists

[2161] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[2162] 5. Recipe generation and suggestions

[2163] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, recipes for "tomato soup" and "lettuce salad" using "milk," "tomato," and "lettuce" are generated and sent to the user's device.

[2164] 6. Suggestion of necessary ingredients

[2165] If the server is running low on ingredients for a proposed menu, it generates a list of those ingredients and notifies the user. For example, it recognizes that the "chicken soup base" needed for "tomato soup" is missing, and notifies the user of that information.

[2166] 7. Monitoring door opening and closing status

[2167] The device (camera) monitors whether the storage device door is open or closed, and periodically sends the data to the server. The timing of the door opening and closing is recorded and compiled by the server.

[2168] 8. Trend Analysis and Anomaly Detection

[2169] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. When an abnormality is detected, a push notification is sent to the user's device.

[2170] 9. User Emotion Recognition

[2171] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[2172] 10. Emotion-Based Suggestions

[2173] The server generates a cooking menu based on the data from the emotion engine according to the user's emotional state. For example, if the user is feeling depressed, it will suggest a "herbal tea recipe" that has a relaxing effect. The generated menu is sent to the user's device.

[2174] 11. Recording and analyzing emotional data

[2175] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to suggest menus that the user prefers and make appropriate suggestions.

[2176] Examples and prompts

[2177] As a specific example, the following procedure is carried out.

[2178] Ingredient management and emotion recognition

[2179] User: Places a Logitech C920 webcam inside the refrigerator and sets it to automatically record at 8:00 AM every day.

[2180] Example prompt: "Set up your webcam and take a picture of the inside of your refrigerator every day at 8 AM."

[2181] Image capture and transmission

[2182] Device (camera): Take a picture at 8:00 and send it to the server.

[2183] Image analysis

[2184] Server: Analyzes the image and recognizes that there are "milk, tomato, and lettuce." Generates a list of ingredients and saves it in a database.

[2185] Emotion Recognition and Suggestions

[2186] Device (camera and microphone): Detects the user's facial expressions and voice data and sends them to the emotion engine.

[2187] Cooking menu suggestions

[2188] Server: Based on the ingredients list and the analysis results of the emotion engine, suggests menu items such as "tomato soup" and "lettuce salad." The user selects "tomato soup."

[2189] Sample prompt: "We analyzed the image of your refrigerator and detected milk, tomatoes, and lettuce. We've added them to your current ingredient list."

[2190] Suggestion of necessary ingredients

[2191] Server: Detects that the "chicken soup base" required for "tomato soup" is missing and notifies the user device of this fact.

[2192] Emotion-based suggestions

[2193] Server: If the server determines that the user's expression is depressed, it will suggest relaxing "herbal tea recipes" and other similar items.

[2194] Example prompt: "The user's facial expression appears depressed. Suggest a relaxing herbal tea recipe."

[2195] Monitoring function implementation

[2196] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[2197] Trend analysis and anomaly detection

[2198] Server: Conducts trend analysis and detects that the door normally opens and closes several times a day, but has not opened or closed for more than 24 hours. This anomaly is detected and a push notification is sent to the user's device.

[2199] Sample prompt: "The refrigerator hasn't been opened or closed for over 24 hours. Please check."

[2200] This system allows users to efficiently manage ingredients, enjoy appropriate cooking menus, and easily monitor elderly people. In addition, by utilizing emotion recognition, it is possible to suggest menus that match the user's mood, providing a more personalized experience.

[2201] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2202] Processing Steps

[2203] Step 1: Setting up your webcam

[2204] Users place a webcam, such as the Logitech C920, on a refrigerator, pantry, or cupboard and adjust the angle and position to capture the entire interior. For example, place the webcam on top of the refrigerator and adjust it to a position that allows a full view of the interior.

[2205] Input: Webcam installation instructions, refrigerator location

[2206] Output: A webcam placed in a suitable position, ready to record.

[2207] Step 2: Take and send an image

[2208] The device (camera) takes pictures of the storage device at set intervals (e.g., every day at 8:00 AM). The pictures are sent to the server via a wireless connection. Custom firmware is used to implement the scheduled shooting and automatic sending functions.

[2209] Input: Set shooting schedule, webcam location information

[2210] Output: Captured image data, image data transmission to server

[2211] Step 3: Image analysis

[2212] The server analyzes the received image using image recognition AI such as TensorFlow, which identifies ingredients in the image (e.g., milk, tomato, lettuce), and generates an ingredient list based on that information.

[2213] Input: Captured image data

[2214] Output: Ingredient list data as the analysis result

[2215] Step 4: Generate and manage your ingredient list

[2216] The server generates an ingredient list based on the image analysis results and stores it in a database such as MySQL. The generated ingredient list is then provided to the user's smartphone or tablet.

[2217] Input: Ingredient list data as a result of image analysis

[2218] Output: Ingredient list stored in the database, provided to the user's device

[2219] Step 5: Recipe generation and suggestions

[2220] The server uses a generative AI model such as GPT-4 to suggest cooking menus based on the stored ingredient list. For example, a recipe for "tomato soup" or "lettuce salad" using "milk," "tomato," and "lettuce" is generated and sent to the user's device.

[2221] Input: List of ingredients in the database

[2222] Output: Generated recipes, suggestions to user terminal

[2223] Step 6: Suggesting ingredients

[2224] If the server is missing any ingredients needed for the proposed menu, it generates a list of those ingredients and notifies the user terminal. For example, it recognizes that "chicken soup base" needed for "tomato soup" is missing and notifies the user of that information.

[2225] Input: Cooking menu, current ingredients list

[2226] Output: List of missing ingredients, notification to user device

[2227] Step 7: Monitor the door opening and closing status

[2228] The device (camera) monitors the opening and closing status of the storage device door and periodically transmits the data to the server. The timing of the door opening and closing is recorded and transmitted to the server.

[2229] Input: Door open / close status, timestamp

[2230] Output: Open / close data sent to the server

[2231] Step 8: Trend analysis and anomaly detection

[2232] The server analyzes the frequency and patterns of door opening and closing based on the received data, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, a push notification is sent to the user's device.

[2233] Input: Door opening / closing data

[2234] Output: Anomaly detection results, notification to user device

[2235] Step 9: Recognizing User Emotions

[2236] The device (camera and microphone) detects the user's facial expressions and voice and sends them to the emotion engine, which analyzes the received data and recognizes the user's emotional state (e.g., joy, anger, sadness, or happiness).

[2237] Input: facial expression data, voice data

[2238] Output: Emotion recognition result

[2239] Step 10: Emotion-Based Suggestions

[2240] The server generates a cooking menu based on the user's emotional state based on the data from the emotion engine. For example, if the user is feeling depressed, it will suggest a relaxing "herbal tea recipe." The generated menu is sent to the user's device.

[2241] Input: Emotion recognition results, list of ingredients

[2242] Output: Emotion-based cooking menu, sent to user device

[2243] Step 11: Record and analyze emotional data

[2244] The server periodically records the user's emotional data and performs trend analysis. It analyzes the emotional data from the past month to detect the emotional state the user tends to be in at certain times. Based on this data, it accumulates data to provide the user with menus they prefer and make appropriate suggestions.

[2245] Input: Periodically collected emotion data

[2246] Output: Sentiment trend analysis results, personalized menu suggestion data

[2247] (Application example 2)

[2248] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2249] In the past, managing ingredients in the refrigerator or pantry and suggesting cooking menus was often done manually, making it difficult for users to efficiently manage ingredients and select cooking menus. Furthermore, there were issues with systems that were unable to provide services that monitored elderly people or responded to the user's emotions. Furthermore, even in brick-and-mortar stores, there was a demand for automated services based on customers' emotions and food inventory management, but these systems were difficult to implement.

[2250] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2251] In this invention, the server includes means for installing webcams in the refrigerator, pantry, and cupboard to capture images of their interiors, artificial intelligence means for analyzing the captured images to generate an ingredient list, artificial intelligence means for proposing cooking menus based on the ingredient list, means for monitoring the opening and closing status of the refrigerator, pantry, and cupboard doors, means for detecting abnormalities based on the opening and closing status and notifying a user terminal, emotion recognition means for recognizing the user's facial expressions and voice and analyzing the data, and means for proposing cooking menus based on the emotion data obtained from the emotion recognition means. This enables automation of ingredient management in brick-and-mortar stores and the proposing of personalized cooking menus based on the user's emotions.

[2252] A "webcam" is a camera device that captures digital images or video and transmits them in real time over a computer or network.

[2253] The "ingredient list" is a list showing ingredients and items present in the refrigerator, pantry, and cupboard.

[2254] An "artificial intelligence means" is a system or software that uses computer algorithms to analyze data and mimic human intellectual activity.

[2255] "Door opening / closing status" is information indicating the status of the opening and closing of the refrigerator, pantry, and cupboard doors.

[2256] An "anomaly" is a condition or action that deviates from a normal pattern or expected behavior.

[2257] "User terminal" means a computer device operated by a user to receive and display information.

[2258] "Emotion recognition means" refers to a system or software that analyzes a user's facial expressions and voice to identify their emotions.

[2259] "Emotion data" is data that represents the emotional state of the user analyzed from facial expressions and voice.

[2260] A "personalized cooking menu" is a cooking menu suggested to suit the user's individual preferences and condition.

[2261] This system uses a webcam to capture images of food storage areas such as kitchens and stockrooms in brick-and-mortar stores, analyzes the images to generate and manage ingredient lists, and suggests cooking menus based on customer sentiment. This system automates food management in brick-and-mortar stores and improves the customer experience.

[2262] System Configuration

[2263] 1. Webcam installation

[2264] User (store clerk): Install a webcam in the kitchen or stockroom of the physical store and adjust the angle and position to capture the entire scene.

[2265] 2. Image capture and transmission

[2266] Terminal (camera): Takes images at regular intervals (e.g., every two hours) and sends them to a server via the Internet.

[2267] 3. Image Analysis

[2268] Server: Analyzes the received image using AI (e.g., AWS Rekognition or Google Cloud Vision), recognizes the ingredients in the image, and generates an ingredient list.

[2269] 4. Creating and managing ingredient lists

[2270] Server: Stores the generated ingredient list in a database and updates and manages it as needed.

[2271] Cooking menu suggestions

[2272] 5. Cooking menu generation

[2273] Server: Using AI, the server proposes cooking menus based on the stored ingredient list. The proposed menus are sent to and displayed on the smartphones or smart glasses of customers and store staff.

[2274] 6. Suggestions for missing ingredients

[2275] Server: If the ingredients required for the proposed menu are in short supply, the server generates a list of them and notifies the user's device, suggesting that they purchase the missing ingredients.

[2276] Monitoring function

[2277] 7. Monitoring door opening and closing status

[2278] Terminal (camera): Monitors whether refrigerators and pantry doors are open or closed in physical stores, and periodically sends this data to the server.

[2279] 8. Trend Analysis and Anomaly Detection

[2280] Server: Based on the received door opening / closing data, it analyzes trends in the frequency and patterns of opening and closing, and detects abnormal patterns or cases where the door has not been opened for a certain period of time. If an abnormality is detected, it sends a push notification to the user's device.

[2281] Emotion recognition function

[2282] 9. User Emotion Recognition

[2283] Device: Uses a camera and microphone to detect the customer's facial expressions and voice, and sends the data to an emotion engine (e.g., Microsoft Azure Cognitive Services' Face API).

[2284] Server: Analyzes the data received from the emotion engine and recognizes the emotional state of the customer (e.g., joy, anger, sadness, or happiness).

[2285] 10. Emotion-Based Suggestions

[2286] Server: Generates cooking menus based on emotion data and sends them to the customers' and staff's devices. For example, if a customer is feeling stressed, the server suggests menus that will have a relaxing effect.

[2287] 11. Recording and analyzing emotional data

[2288] Server: Periodically records user emotion data and performs trend analysis.

[2289] Server: Accumulates data to determine customers' preferred menu items and make appropriate suggestions based on past emotional patterns.

[2290] Specific examples

[2291] Ingredient management and emotion recognition

[2292] User (Store Staff): Set up a webcam in the kitchen to capture images of the food storage area at 8:00 AM and 2:00 PM every day.

[2293] Terminal (camera): Takes a picture at the set time and sends it to the server.

[2294] Server: Analyzes the image, generates an ingredient list, and stores it in a database.

[2295] Terminal: The camera and microphone also detect the customer's facial expressions and voice data, which are then sent to the emotion engine.

[2296] Cooking menu suggestions

[2297] Server: Based on the ingredients list and the analysis results of the emotion engine, the server proposes multiple menu items and sends them to the user's device. The customer selects a menu item.

[2298] Suggestions for ingredients that are in short supply

[2299] Server: If any ingredients required for the selected menu are missing, it generates a list of them and notifies the user terminal of that fact.

[2300] Emotion-based suggestions

[2301] Server: If a customer's expression is deemed depressed, the server will suggest, for example, a relaxing "herbal tea recipe."

[2302] Monitoring function implementation

[2303] Terminal (camera): Records the opening and closing of the refrigerator door every day and sends the data to the server.

[2304] Server: Conducts trend analysis and detects when th...

Claims

1. A means of installing a webcam in your refrigerator, pantry, or cupboard to capture images of the inside of these items; an artificial intelligence means for analyzing the captured image and generating an ingredient list; an artificial intelligence means for proposing cooking menus based on the ingredient list; A means of monitoring the opening and closing status of refrigerator, pantry and cupboard doors; means for detecting an abnormality based on the opening / closing state and notifying a user terminal of the abnormality; A system including:

2. means for storing the generated ingredient list in a database; a means for suggesting ingredients to be purchased based on the stored ingredient list; The system of claim 1 further comprising:

3. means for receiving and buffering images transmitted from the web camera; artificial intelligence means for analyzing said stored images; means for transmitting the analysis result to a user terminal; The system of claim 1 further comprising:

Images