System

Abstract

An object of a system according to an embodiment is to measure the pace of decrease of ingredients in a refrigerator and make a purchase proposal at an appropriate timing.SOLUTION: A system according to an embodiment includes a food ingredient sensing unit, a decrease pace measurement unit, a purchase proposal generation unit, a position information detection unit, and a price information notification unit. The food sensing unit senses food in the refrigerator. The decreasing pace measurement unit measures the decreasing pace of the foodstuff sensed by the foodstuff sensing unit. The purchase proposal generation unit generates a purchase proposal based on the pace decrease measured by the pace decrease measurement unit. The position information detection unit detects position information of a user. The price information notification unit notifies the price information based on the position information detected by the position information detection unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, it is difficult to detect when food is running low in the refrigerator and suggest purchasing at the appropriate time, which can lead to forgetting to buy something or making unnecessary purchases.

[0005] The system according to the embodiment aims to measure the rate at which food items in a refrigerator are being used up and make purchasing suggestions at the appropriate time. [Means for solving the problem]

[0006] The system according to the embodiment includes an ingredient detection unit, a reduction pace measurement unit, a purchase proposal generation unit, a location information detection unit, and a price information notification unit. The ingredient detection unit detects ingredients in the refrigerator. The reduction pace measurement unit measures the reduction pace of the ingredients detected by the ingredient detection unit. The purchase proposal generation unit generates a purchase proposal based on the reduction pace measured by the reduction pace measurement unit. The location information detection unit detects location information of the user. The price information notification unit notifies price information based on the location information detected by the location information detection unit. [Effects of the Invention]

[0007] The system according to the embodiment can measure the rate at which food items in the refrigerator are being used up and make purchasing suggestions at the appropriate time. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 may have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The food management system according to an embodiment of the present invention automatically detects food items stored in a refrigerator, measures the rate at which they are being used up, and makes purchasing suggestions. Furthermore, by linking with a smartphone and notifying the user of food item price information for each store based on the user's location information, the system prevents the user from forgetting to buy food and provides a service that allows them to purchase food at a lower price. This allows the food management system to efficiently manage food items stored in the refrigerator and improve the user's shopping experience.

[0029] The food ingredient management system according to the embodiment includes an ingredient detection unit, a consumption rate measurement unit, a purchase proposal generation unit, a location information detection unit, and a price information notification unit. The ingredient detection unit detects ingredients in the refrigerator. For example, a camera captures images of the inside of the refrigerator, and the generation AI analyzes the images to identify the type and quantity of ingredients. The sensor detects the presence of ingredients based on weight and location information. For example, the camera captures high-resolution images of the inside of the refrigerator, and the generation AI analyzes the images to identify the type and quantity of ingredients. The sensor measures the weight of ingredients and detects their presence based on location information. The consumption rate measurement unit measures the rate at which ingredients are consumed detected by the ingredient detection unit. For example, it predicts the consumption rate of ingredients based on past data and calculates the rate at which each ingredient is being consumed. The consumption rate measurement unit can also measure the rate at which ingredients are being consumed in real time. For example, it predicts the consumption rate of ingredients based on past data and measures the rate at which ingredients are being consumed in real time. The purchase proposal generation unit generates a purchase proposal based on the consumption rate measured by the consumption rate measurement unit. For example, the purchase suggestion generation unit may send a notification to the smartphone such as, "You're running low on milk. Please buy more the next time you shop." The purchase suggestion generation unit may also customize the suggestion content based on the user's preferences and past purchase history. For example, the suggestion content may be customized based on the user's preferences and past purchase history and notified to the smartphone. The location information detection unit detects the user's location information. For example, the location information detection unit may acquire the user's location information using the smartphone's GPS function. The location information detection unit may also analyze the user's movement patterns and update the location information at the optimal timing. For example, the location information detection unit may acquire the user's location information using the smartphone's GPS function, analyze the movement patterns, and update the location information at the optimal timing. The price information notification unit notifies the user of price information based on the location information detected by the location information detection unit. For example, when the user approaches a supermarket, the unit may send a notification such as, "Milk is on sale at this supermarket." The price information notification unit may also notify the user of price information customized based on the user's purchase history and preferences. For example, the price information notification unit may notify the user of price information customized based on the user's purchase history and preferences.As a result, the food ingredient management system according to the embodiment can improve the efficiency of food ingredient management in the refrigerator and enhance the user's shopping experience. For example, it can prevent forgetting to buy ingredients and purchase the necessary ingredients at the appropriate time. Furthermore, by notifying the user of price information, the user can purchase ingredients at a lower price.

[0030] The food sensor can estimate the freshness and expiration date of food ingredients using a camera. For example, the food sensor uses a camera inside the refrigerator to periodically take images, which are then analyzed by a generation AI. For example, the sensor estimates freshness based on changes in the color and shape of vegetables and identifies food ingredients that are close to their expiration date. The food sensor also uses a generation AI to analyze the labels and packaging information of food ingredients in the refrigerator and automatically read expiration dates. For example, it obtains expiration dates by scanning barcodes or QR codes. To estimate the freshness of food ingredients, the generation AI also performs image analysis to detect discoloration or mold growth on the surface of the food ingredients. For example, it evaluates freshness based on spots and discoloration that appear on the surface of fruit. This allows the freshness and expiration date of food ingredients to be accurately determined.

[0031] The food sensor can monitor the storage status of food ingredients in real time using a temperature sensor and a humidity sensor. The food sensor can monitor the storage status of food ingredients in real time using, for example, a temperature sensor and a humidity sensor installed inside the refrigerator. For example, it can detect fluctuations in temperature and humidity and predict food deterioration. The food sensor can also use AI to analyze data from the temperature and humidity sensors and build a system to evaluate the storage status of food ingredients. For example, it can automatically adjust the refrigerator settings to maintain appropriate storage conditions. The food sensor can also send data from the temperature and humidity sensors to the cloud to remotely monitor the storage status of food ingredients. For example, it can make it possible to check the storage status in real time using a smartphone app. This allows the storage status of food ingredients to be monitored in real time.

[0032] The food sensor can be expanded to the freezer and pantry, enabling centralized management of food ingredients in the home. For example, the food sensor can extend the food sensor function in the refrigerator to the freezer, automatically detecting the type and quantity of frozen foods. For example, cameras and sensors in the freezer can be used to monitor the status of frozen foods. The food sensor can also incorporate food sensor functions in the pantry to centrally manage preserved foods such as dried goods and canned goods. For example, cameras and weight sensors in the pantry can be used to keep track of food inventory. The food sensor can also integrate food data from the refrigerator, freezer, and pantry into the cloud, creating a system for centralized management of all food ingredients in the home. For example, the status of all food ingredients can be checked using a smartphone app. This allows for centralized management of all food ingredients in the home.

[0033] The ingredient sensor stores ingredient detection data in the cloud and makes it accessible from multiple devices, allowing all family members to know the status of ingredients. For example, the ingredient sensor stores ingredient detection data from the refrigerator in the cloud, allowing all family members to access it from their smartphones or tablets. For example, the status of ingredients can be shared through a dedicated app. The ingredient sensor also builds a system that allows all family members to check ingredient inventory in real time based on the ingredient detection data stored in the cloud. For example, it displays ingredient consumption status in a graph. The ingredient sensor also stores ingredient detection data in the cloud, allowing all family members to know the status of ingredients, thereby developing a system that automatically generates shopping lists. For example, it lists and notifies the user of needed ingredients. This allows all family members to know the status of ingredients.

[0034] The reduction pace measurement unit uses the generation AI to compare past consumption data with current trends, allowing for more accurate predictions. For example, the reduction pace measurement unit uses the generation AI to analyze past consumption data and current trends to predict the rate at which an ingredient will be reduced. For example, seasonal consumption patterns can be taken into account to improve prediction accuracy. The reduction pace measurement unit also compares past consumption data with current trends, building a system in which the generation AI updates the rate at which ingredients will be reduced in real time. For example, the prediction can be adjusted according to changes in consumption rate. The reduction pace measurement unit also uses the generation AI to predict the rate at which an ingredient will be reduced based on past consumption data and current trends, and notify the user. For example, an alert can be sent before a specific ingredient runs out. This allows for more accurate predictions by comparing past consumption data with current trends.

[0035] The reduction pace measurement unit can collect data on the user's eating patterns and lifestyle and provide an individually customized prediction. For example, the reduction pace measurement unit collects data on the user's eating patterns and lifestyle and provides an individually customized prediction of the rate at which ingredients will be reduced. For example, the prediction is adjusted based on the frequency and amount of meals. The reduction pace measurement unit also analyzes the user's lifestyle data and builds a system in which the generation AI makes an individually customized prediction of the rate at which ingredients will be reduced. For example, the user's activity level and food preferences are taken into consideration. The reduction pace measurement unit also predicts the rate at which ingredients will be reduced based on the user's eating patterns and lifestyle data and sends an individually customized notification. For example, an alert is sent before a specific ingredient runs out. This makes it possible to provide an individually customized prediction based on the user's eating patterns and lifestyle.

[0036] The consumption pace measurement unit can also be applied to consumables such as beverages and seasonings, allowing for the management of consumables within the home. For example, the consumption pace measurement unit can apply the function of measuring the consumption pace of food ingredients to consumables such as beverages and seasonings, building a system that manages all consumables within the home. For example, it can predict the consumption pace of beverages and make purchase suggestions at the appropriate time. The consumption pace measurement unit also collects consumption data on seasonings and beverages, and the generation AI predicts the consumption pace of consumables based on that data. For example, it can set the consumption pace based on the frequency of seasoning use. The consumption pace measurement unit can also measure the consumption pace of consumables such as beverages and seasonings, developing a system that manages the inventory of all consumables within the home. For example, it can send an alert before a consumable runs out. This allows for the management of all consumables within the home.

[0037] The reduction pace measurement unit can compare the reduction pace data with other households and provide advice based on the average consumption pace. The reduction pace measurement unit, for example, compares the reduction pace data of food ingredients with other households and builds a system that provides advice based on the average consumption pace. For example, the consumption pace is evaluated based on data from households in the same area. The reduction pace measurement unit also collects consumption data from other households, and the generation AI calculates the average consumption pace based on that data. For example, the consumption pace of the same food ingredient is compared and advice is provided. The reduction pace measurement unit also shares the reduction pace data with other households and develops a system that provides advice based on the average consumption pace. For example, advice on saving is given if the consumption pace is slow. This makes it possible to provide advice based on the average consumption pace compared to other households.

[0038] The purchase suggestion generation unit can customize using the generation AI to make personalized suggestions based on the user's past purchase history and preferences. The purchase suggestion generation unit, for example, uses the generation AI to make personalized purchase suggestions based on the user's past purchase history and preferences. For example, it prioritizes suggestions for ingredients that the user frequently purchases. The purchase suggestion generation unit also analyzes the user's purchase history data, and builds a system in which the generation AI makes customized purchase suggestions based on that data. For example, it suggests ingredients of specific brands or types. The purchase suggestion generation unit also uses the generation AI to make personalized purchase suggestions based on the user's preferences and past purchase history, and notifies the smartphone. For example, it suggests ingredients needed for a recipe that the user likes. This makes it possible to make personalized suggestions based on the user's past purchase history and preferences.

[0039] The purchase suggestion generation unit can simultaneously provide nutritional information for ingredients and recipe suggestions, thereby supporting a healthy diet. For example, the purchase suggestion generation unit can simultaneously provide nutritional information for ingredients when making purchase suggestions, thereby supporting the user in maintaining a healthy diet. For example, the vitamin and mineral content can be displayed. The purchase suggestion generation unit can also build a system that provides recipes using the suggested ingredients along with purchase suggestions. For example, it can notify users of recipes that suggest balanced meals. The purchase suggestion generation unit can also simultaneously provide nutritional information for ingredients and recipe suggestions, giving advice to help the user maintain a healthy diet. For example, it can suggest low-calorie and high-protein ingredients. This allows for the simultaneous provision of nutritional information for ingredients and recipe suggestions, thereby supporting a healthy diet.

[0040] The purchase proposal generation unit can also send the proposal to other devices such as a smart speaker or a smart watch, allowing the user to receive the proposal anywhere. For example, the purchase proposal generation unit builds a system that sends the purchase proposal to a smart speaker and notifies the user by voice. For example, the purchase proposal is received by voice while cooking in the kitchen. The purchase proposal generation unit also sends the purchase proposal to a smart watch, allowing the user to receive the proposal even when they are out. For example, the proposal is checked on the smart watch while shopping. The purchase proposal generation unit also develops a system that sends the purchase proposal to multiple devices, allowing the user to receive the proposal anywhere. For example, notifications are sent simultaneously to a smartphone, smart speaker, and smart watch. This allows the user to receive the proposal anywhere.

[0041] The purchase proposal generation unit can send the purchase proposal to the smartphones of all family members and coordinate who will be in charge of shopping. For example, the purchase proposal generation unit builds a system that sends purchase proposals to the smartphones of all family members and coordinates who will be in charge of shopping. For example, a shopping list is shared among family members. The purchase proposal generation unit also sends purchase proposals to the smartphones of all family members and automatically assigns who will be in charge of shopping. For example, it notifies the family member who is closest. The purchase proposal generation unit also develops an app that sends purchase proposals to the smartphones of all family members and coordinates who will be in charge of shopping. For example, the shopping list is updated in real time among family members. This makes it possible to send the purchase proposal to the smartphones of all family members and coordinate who will be in charge of shopping.

[0042] The location information detection unit can use a generation AI to predict sale information for stores the user is likely to visit based on location information and notify the user in advance. For example, the location information detection unit can build a system that uses a generation AI to predict sale information for stores the user is likely to visit based on location information and notify the user in advance. For example, the location information detection unit analyzes the user's movement patterns to provide sale information. The location information detection unit also uses a generation AI to predict sale information based on the user's location information and past purchase history and notify the smartphone. For example, it notifies the user in advance of sale information for specific stores. The location information detection unit can also develop a system that predicts sale information for stores the user is likely to visit in real time based on location information and notifies the user. For example, sale information is sent when the user approaches a nearby store. This allows the user to be notified in advance of sale information for stores the user is likely to visit.

[0043] The location information detection unit can provide customized coupons based on a user's purchasing history and preferences. The location information detection unit, for example, builds a system that provides customized coupons based on a user's purchasing history and preferences. For example, it issues discount coupons for frequently purchased ingredients. The location information detection unit also analyzes purchasing history data, and a generation AI provides customized coupons based on that data. For example, it issues coupons for specific brands or types of ingredients. The location information detection unit also develops a system that notifies a smartphone of customized coupons based on a user's preferences and purchasing history. For example, it provides coupons for ingredients needed for a user's favorite recipe. This makes it possible to provide customized coupons based on a user's purchasing history and preferences.

[0044] The location information detection unit can optimize and suggest an efficient shopping route based on the user's commuting route and daily movement patterns. The location information detection unit, for example, analyzes the user's commuting route and daily movement patterns to build a system that suggests the optimal shopping route. For example, it suggests stores that can be stopped off on the way to work. The location information detection unit also uses generative AI to suggest an efficient shopping route based on the user's movement patterns. For example, it calculates a route that efficiently visits multiple stores. The location information detection unit also develops a system that suggests the optimal shopping route in real time based on the user's commuting route and daily movement patterns. For example, it adjusts the route taking traffic conditions and store opening hours into consideration. This makes it possible to suggest an efficient shopping route based on the user's commuting route and daily movement patterns.

[0045] The location information detection unit can be linked with smart home devices to allow users to receive sale information even when they are at home. For example, the location information detection unit can link location information detection and price information notification with smart home devices to build a system that allows users to receive sale information even when they are at home. For example, sale information can be notified by voice using a smart speaker. The location information detection unit can also link with smart home devices to allow users to receive sale information in real time even when they are at home. For example, sale information can be displayed on a smart display. The location information detection unit can also integrate location information detection and price information notification with smart home devices to develop a system that allows users to receive sale information even when they are at home. For example, sale information can be notified by changing the color of a smart light. This makes it possible to receive sale information even when users are at home.

[0046] The purchase suggestion generation unit uses generation AI to analyze data and learn the user's food ingredient preferences and consumption patterns, enabling more accurate suggestions. The purchase suggestion generation unit, for example, uses generation AI to analyze the user's food ingredient purchase history and learn the preferences and consumption patterns. For example, it identifies frequently purchased ingredients and brands and makes suggestions based on that. The purchase suggestion generation unit also uses generation AI to analyze the user's purchase history data and builds a system that learns consumption patterns. For example, it makes suggestions based on ingredients purchased on specific days of the week or time periods. The purchase suggestion generation unit also uses generation AI to learn the user's food ingredient preferences and consumption patterns and provides more accurate suggestions. For example, it suggests new ingredients and recipes based on past purchase history. This allows the unit to learn the user's food ingredient preferences and consumption patterns and provide more accurate suggestions.

[0047] The purchase suggestion generation unit can make food ingredient suggestions that take into account the user's health condition and nutritional balance. The purchase suggestion generation unit, for example, builds a system that makes food ingredient suggestions that take into account the user's health condition and nutritional balance. For example, suggestions are made based on health checkup results and diet records. The purchase suggestion generation unit also uses generation AI to analyze the user's health condition and nutritional balance and make food ingredient suggestions based on that. For example, if a user is lacking in a particular nutrient, it suggests food ingredients that contain that nutrient. The purchase suggestion generation unit also develops a system that makes food ingredient suggestions that take into account nutritional balance based on the user's health data. For example, it suggests low-calorie food ingredients to a user who is on a diet. This makes it possible to make food ingredient suggestions that take into account the user's health condition and nutritional balance.

[0048] The purchase suggestion generation unit can share purchase history data with other users and promote community-based ingredient suggestions and recipe exchanges. The purchase suggestion generation unit, for example, shares purchase history data with other users and builds a system that promotes community-based ingredient suggestions and recipe exchanges. For example, recipes using the same ingredients are shared. The purchase suggestion generation unit also develops a system that shares users' purchase history data within a community and receives suggestions from other users. For example, it proposes new recipes using specific ingredients. The purchase suggestion generation unit also builds a platform that promotes community-based ingredient suggestions and recipe exchanges based on the purchase history data. For example, users share how to use and store ingredients. This allows purchase history data to be shared with other users and promotes community-based ingredient suggestions and recipe exchanges.

[0049] The purchase suggestion generation unit can make food suggestions that match seasonal special suggestions and events. For example, the purchase suggestion generation unit builds a system that makes food suggestions that match seasonal special suggestions and events based on purchase history data. For example, it suggests food ingredients that match Christmas and Halloween. The purchase suggestion generation unit also uses generation AI to make food suggestions that match seasonal special suggestions and events. For example, it suggests ingredients for barbecues in the summer. The purchase suggestion generation unit also develops a system that makes food suggestions that match seasonal special suggestions and events based on purchase history data. For example, it suggests recipes that use seasonal ingredients. This makes it possible to make food suggestions that match seasonal special suggestions and events.

[0050] The food sensor uses a camera to analyze food label and packaging information and automatically read expiration dates. For example, the food sensor uses a camera inside a refrigerator to periodically take images and analyze those images with a generation AI. For example, the sensor estimates freshness based on changes in the color and shape of vegetables and identifies food items that are close to their expiration date. The food sensor also uses a generation AI to analyze food label and packaging information in the refrigerator and automatically read expiration dates. For example, it obtains expiration dates by scanning barcodes or QR codes. To estimate food freshness, the sensor uses an image analysis system with the generation AI to detect discoloration or mold on the surface of food items. For example, it evaluates freshness based on spots and discoloration that appear on the surface of fruit. This allows the sensor to analyze food label and packaging information and automatically read expiration dates.

[0051] The food sensor uses a camera to detect discoloration and mold growth on the surface of food ingredients and evaluate their freshness. For example, the food sensor uses a camera inside the refrigerator to periodically take images and analyze those images with a generation AI. For example, it estimates freshness based on changes in the color and shape of vegetables and identifies food ingredients that are close to their expiration date. The food sensor also uses a generation AI to analyze the labels and packaging information of food ingredients in the refrigerator and automatically read expiration dates. For example, it obtains expiration dates by scanning barcodes or QR codes. To estimate the freshness of food ingredients, the generation AI also performs image analysis to detect discoloration and mold growth on the surface of food ingredients. For example, it evaluates freshness based on spots and discoloration that appear on the surface of fruit. This makes it possible to detect discoloration and mold growth on the surface of food ingredients and evaluate their freshness.

[0052] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0053] The food sensor can analyze the nutritional value of ingredients in the refrigerator and suggest healthy eating habits to the user. For example, it can use a camera and generative AI to identify the type of ingredient and calculate its nutritional value. The food sensor can also suggest ingredients that take into account the user's health condition and nutritional balance. For example, if a user is lacking in a particular nutrient, it can suggest ingredients that contain that nutrient. The food sensor can also build a system that suggests balanced meals based on the nutritional value of ingredients. This allows users to obtain information to maintain a healthy diet.

[0054] The food sensor can analyze allergen information for food items in the refrigerator and suggest allergy countermeasures to the user. For example, it can use a camera and generative AI to read food labels and extract allergen information. The food sensor can also identify food items containing allergens based on the user's allergy information and issue a warning. For example, it can notify the user if an ingredient containing an allergen is stored in the refrigerator. The food sensor can also build a system that suggests alternative ingredients to prevent allergies based on the allergen information. This allows users to obtain information to take measures against allergies.

[0055] The food sensor can analyze the eco-footprint of ingredients in the refrigerator and suggest environmentally friendly food choices to the user. For example, it can use a camera and generative AI to identify the type of ingredient and calculate the eco-footprint based on its production process and transportation distance. The food sensor can also suggest environmentally friendly ingredients based on the eco-footprint data. For example, it can prioritize locally produced ingredients and organic ingredients. The food sensor can also build a system that uses the eco-footprint data to advise users on environmentally friendly food choices. This allows users to obtain information to make environmentally friendly food choices.

[0056] The food sensor can analyze how food is stored in the refrigerator and suggest optimal storage methods to users. For example, it can use a camera and generative AI to identify the type of food and analyze its storage method. The food sensor can also suggest optimal storage conditions based on the food storage method. For example, it can recommend storing food at a specific temperature and humidity. The food sensor can also build a system that gives users advice on storage methods based on storage method data. This allows users to obtain information to optimize their food storage methods.

[0057] The ingredient detection unit can analyze the calorie information of ingredients in the refrigerator and make suggestions to the user to support calorie management. For example, it can use a camera and generation AI to identify the type of ingredient and calculate its calorie information. The ingredient detection unit can also make ingredient suggestions to support calorie management based on the user's calorie intake. For example, it can prioritize suggestions of low-calorie ingredients. The ingredient detection unit can also build a system that gives calorie management advice to the user based on the calorie information. This allows the user to obtain information to support calorie management.

[0058] The food sensor can analyze the storage period of food ingredients in the refrigerator and suggest the optimal time to consume them to the user. For example, it can use a camera and generative AI to identify the type of food ingredient and calculate its storage period. The food sensor can also suggest the optimal time to consume ingredients based on the storage period data. For example, it can recommend that ingredients with an approaching expiration date be consumed first. The food sensor can also build a system that advises the user on the optimal time to consume ingredients based on the storage period data. This allows the user to obtain information to determine the optimal time to consume ingredients.

[0059] The processing flow of the first embodiment will be briefly explained below.

[0060] Step 1: The food detector detects the food in the refrigerator. For example, a camera takes an image of the inside of the refrigerator, and the generation AI analyzes the image to identify the type and quantity of food. The sensor also detects the presence of food based on weight and location information. Step 2: The consumption rate measurement unit measures the consumption rate of the ingredients detected by the ingredient detection unit. For example, it predicts the consumption rate of ingredients based on past data and calculates the rate at which each ingredient is being consumed. It can also measure the consumption rate of ingredients in real time. Step 3: The purchase suggestion generator generates a purchase suggestion based on the consumption rate measured by the consumption rate measurement unit. For example, it sends a notification to the smartphone saying, "You're running low on milk. Please buy some next time you go shopping." The suggestion content can also be customized based on the user's preferences and past purchase history. Step 4: The location information detection unit detects the user's location information. For example, it can obtain the user's location information using the smartphone's GPS function. It can also analyze the user's movement patterns and update the location information at the optimal timing. Step 5: The price information notification unit notifies the user of price information based on the location information detected by the location information detection unit. For example, when the user approaches a supermarket, it sends a notification such as, "Milk is on sale at this supermarket." It can also notify users of customized price information based on their purchasing history and preferences.

[0061] (Example 2) The food management system according to an embodiment of the present invention automatically detects food items stored in a refrigerator, measures the rate at which they are being used up, and makes purchasing suggestions. Furthermore, by linking with a smartphone and notifying the user of food item price information for each store based on the user's location information, the system prevents the user from forgetting to buy food and provides a service that allows them to purchase food at a lower price. This allows the food management system to efficiently manage food items stored in the refrigerator and improve the user's shopping experience.

[0062] The food ingredient management system according to the embodiment includes an ingredient detection unit, a consumption rate measurement unit, a purchase proposal generation unit, a location information detection unit, and a price information notification unit. The ingredient detection unit detects ingredients in the refrigerator. For example, a camera captures images of the inside of the refrigerator, and the generation AI analyzes the images to identify the type and quantity of ingredients. The sensor detects the presence of ingredients based on weight and location information. For example, the camera captures high-resolution images of the inside of the refrigerator, and the generation AI analyzes the images to identify the type and quantity of ingredients. The sensor measures the weight of ingredients and detects their presence based on location information. The consumption rate measurement unit measures the rate at which ingredients are consumed detected by the ingredient detection unit. For example, it predicts the consumption rate of ingredients based on past data and calculates the rate at which each ingredient is being consumed. The consumption rate measurement unit can also measure the rate at which ingredients are being consumed in real time. For example, it predicts the consumption rate of ingredients based on past data and measures the rate at which ingredients are being consumed in real time. The purchase proposal generation unit generates a purchase proposal based on the consumption rate measured by the consumption rate measurement unit. For example, the purchase suggestion generation unit may send a notification to the smartphone such as, "You're running low on milk. Please buy more the next time you shop." The purchase suggestion generation unit may also customize the suggestion content based on the user's preferences and past purchase history. For example, the suggestion content may be customized based on the user's preferences and past purchase history and notified to the smartphone. The location information detection unit detects the user's location information. For example, the location information detection unit may acquire the user's location information using the smartphone's GPS function. The location information detection unit may also analyze the user's movement patterns and update the location information at the optimal timing. For example, the location information detection unit may acquire the user's location information using the smartphone's GPS function, analyze the movement patterns, and update the location information at the optimal timing. The price information notification unit notifies the user of price information based on the location information detected by the location information detection unit. For example, when the user approaches a supermarket, the unit may send a notification such as, "Milk is on sale at this supermarket." The price information notification unit may also notify the user of price information customized based on the user's purchase history and preferences. For example, the price information notification unit may notify the user of price information customized based on the user's purchase history and preferences.As a result, the food ingredient management system according to the embodiment can improve the efficiency of food ingredient management in the refrigerator and enhance the user's shopping experience. For example, it can prevent forgetting to buy ingredients and purchase the necessary ingredients at the appropriate time. Furthermore, by notifying the user of price information, the user can purchase ingredients at a lower price.

[0063] The food sensor can estimate the freshness and expiration date of food ingredients using a camera. For example, the food sensor uses a camera inside the refrigerator to periodically take images, which are then analyzed by a generation AI. For example, the sensor estimates freshness based on changes in the color and shape of vegetables and identifies food ingredients that are close to their expiration date. The food sensor also uses a generation AI to analyze the labels and packaging information of food ingredients in the refrigerator and automatically read expiration dates. For example, it obtains expiration dates by scanning barcodes or QR codes. To estimate the freshness of food ingredients, the generation AI also performs image analysis to detect discoloration or mold growth on the surface of the food ingredients. For example, it evaluates freshness based on spots and discoloration that appear on the surface of fruit. This allows the freshness and expiration date of food ingredients to be accurately determined.

[0064] The food sensor can monitor the storage status of food ingredients in real time using a temperature sensor and a humidity sensor. The food sensor can monitor the storage status of food ingredients in real time using, for example, a temperature sensor and a humidity sensor installed inside the refrigerator. For example, it can detect fluctuations in temperature and humidity and predict food deterioration. The food sensor can also use AI to analyze data from the temperature and humidity sensors and build a system to evaluate the storage status of food ingredients. For example, it can automatically adjust the refrigerator settings to maintain appropriate storage conditions. The food sensor can also send data from the temperature and humidity sensors to the cloud to remotely monitor the storage status of food ingredients. For example, it can make it possible to check the storage status in real time using a smartphone app. This allows the storage status of food ingredients to be monitored in real time.

[0065] The ingredient detection unit uses the emotion estimation function to analyze the user's facial expression when they open the refrigerator and estimate their preferences or dissatisfaction with a particular ingredient. For example, the ingredient detection unit uses the emotion estimation function to capture the user's facial expression using a camera inside the refrigerator and analyze it. For example, it detects expressions of joy or dissatisfaction when the user opens the refrigerator. The ingredient detection unit also uses the emotion estimation function to analyze the user's facial expression when they take out a particular ingredient and estimate their preference for that ingredient. For example, it evaluates preferences based on smiles or surprised expressions. The ingredient detection unit also accumulates the user's facial expression data and analyzes it using the emotion estimation function to understand the user's long-term preferences or dissatisfaction with a particular ingredient. For example, it identifies ingredients that frequently show dissatisfied expressions. This allows the user's preferences and dissatisfaction to be understood and used for ingredient management.

[0066] The food sensor can be expanded to the freezer and pantry, enabling centralized management of food ingredients in the home. For example, the food sensor can extend the food sensor function in the refrigerator to the freezer, automatically detecting the type and quantity of frozen foods. For example, cameras and sensors in the freezer can be used to monitor the status of frozen foods. The food sensor can also incorporate food sensor functions in the pantry to centrally manage preserved foods such as dried goods and canned goods. For example, cameras and weight sensors in the pantry can be used to keep track of food inventory. The food sensor can also integrate food data from the refrigerator, freezer, and pantry into the cloud, creating a system for centralized management of all food ingredients in the home. For example, the status of all food ingredients can be checked using a smartphone app. This allows for centralized management of all food ingredients in the home.

[0067] The ingredient sensor stores ingredient detection data in the cloud and makes it accessible from multiple devices, allowing all family members to know the status of ingredients. For example, the ingredient sensor stores ingredient detection data from the refrigerator in the cloud, allowing all family members to access it from their smartphones or tablets. For example, the status of ingredients can be shared through a dedicated app. The ingredient sensor also builds a system that allows all family members to check ingredient inventory in real time based on the ingredient detection data stored in the cloud. For example, it displays ingredient consumption status in a graph. The ingredient sensor also stores ingredient detection data in the cloud, allowing all family members to know the status of ingredients, thereby developing a system that automatically generates shopping lists. For example, it lists and notifies the user of needed ingredients. This allows all family members to know the status of ingredients.

[0068] The ingredient detection unit uses the emotion estimation function to analyze the emotion a user feels when they open the refrigerator and can suggest recipes based on their emotions toward specific ingredients. For example, the ingredient detection unit uses the emotion estimation function to capture the user's facial expression using a camera inside the refrigerator and analyze it. For example, it detects expressions of joy or surprise when the user opens the refrigerator. The ingredient detection unit also uses the emotion estimation function to analyze the user's facial expression when they take out a specific ingredient and suggests recipes based on that emotion. For example, it suggests a favorite recipe based on a smiling expression. The ingredient detection unit also accumulates the user's facial expression data and analyzes it using the emotion estimation function to understand the user's long-term emotions toward specific ingredients and suggests recipes based on that. For example, it suggests recipes using ingredients that frequently show expressions of joy. This makes it possible to suggest recipes based on the user's emotions.

[0069] The reduction pace measurement unit uses the generation AI to compare past consumption data with current trends, allowing for more accurate predictions. For example, the reduction pace measurement unit uses the generation AI to analyze past consumption data and current trends to predict the rate at which an ingredient will be reduced. For example, seasonal consumption patterns can be taken into account to improve prediction accuracy. The reduction pace measurement unit also compares past consumption data with current trends, building a system in which the generation AI updates the rate at which ingredients will be reduced in real time. For example, the prediction can be adjusted according to changes in consumption rate. The reduction pace measurement unit also uses the generation AI to predict the rate at which an ingredient will be reduced based on past consumption data and current trends, and notify the user. For example, an alert can be sent before a specific ingredient runs out. This allows for more accurate predictions by comparing past consumption data with current trends.

[0070] The reduction pace measurement unit can collect data on the user's eating patterns and lifestyle and provide an individually customized prediction. For example, the reduction pace measurement unit collects data on the user's eating patterns and lifestyle and provides an individually customized prediction of the rate at which ingredients will be reduced. For example, the prediction is adjusted based on the frequency and amount of meals. The reduction pace measurement unit also analyzes the user's lifestyle data and builds a system in which the generation AI makes an individually customized prediction of the rate at which ingredients will be reduced. For example, the user's activity level and food preferences are taken into consideration. The reduction pace measurement unit also predicts the rate at which ingredients will be reduced based on the user's eating patterns and lifestyle data and sends an individually customized notification. For example, an alert is sent before a specific ingredient runs out. This makes it possible to provide an individually customized prediction based on the user's eating patterns and lifestyle.

[0071] The reduction pace measurement unit uses the emotion estimation function to analyze the emotion of the user when consuming a specific ingredient and predict the consumption pace based on the emotion. For example, the reduction pace measurement unit uses the emotion estimation function to analyze the emotion of the user when consuming a specific ingredient and predict the consumption pace based on the data. For example, the reduction pace measurement unit sets a high consumption pace for ingredients that evoke strong positive emotions. The reduction pace measurement unit also accumulates user emotion data and analyzes it using the emotion estimation function to predict the consumption pace based on the emotion of the user toward a specific ingredient. For example, the reduction pace measurement unit sets a high consumption pace for ingredients that frequently show happy expressions. The reduction pace measurement unit also uses the emotion estimation function to analyze the emotion of the user when consuming a specific ingredient in real time and builds a system that predicts the consumption pace based on the data. For example, the prediction is adjusted according to fluctuations in the consumption pace. This makes it possible to predict the consumption pace based on the user's emotion.

[0072] The consumption pace measurement unit can also be applied to consumables such as beverages and seasonings, allowing for the management of consumables within the home. For example, the consumption pace measurement unit can apply the function of measuring the consumption pace of food ingredients to consumables such as beverages and seasonings, building a system that manages all consumables within the home. For example, it can predict the consumption pace of beverages and make purchase suggestions at the appropriate time. The consumption pace measurement unit also collects consumption data on seasonings and beverages, and the generation AI predicts the consumption pace of consumables based on that data. For example, it can set the consumption pace based on the frequency of seasoning use. The consumption pace measurement unit can also measure the consumption pace of consumables such as beverages and seasonings, developing a system that manages the inventory of all consumables within the home. For example, it can send an alert before a consumable runs out. This allows for the management of all consumables within the home.

[0073] The reduction pace measurement unit can compare the reduction pace data with other households and provide advice based on the average consumption pace. The reduction pace measurement unit, for example, compares the reduction pace data of food ingredients with other households and builds a system that provides advice based on the average consumption pace. For example, the consumption pace is evaluated based on data from households in the same area. The reduction pace measurement unit also collects consumption data from other households, and the generation AI calculates the average consumption pace based on that data. For example, the consumption pace of the same food ingredient is compared and advice is provided. The reduction pace measurement unit also shares the reduction pace data with other households and develops a system that provides advice based on the average consumption pace. For example, advice on saving is given if the consumption pace is slow. This makes it possible to provide advice based on the average consumption pace compared to other households.

[0074] The purchase suggestion generation unit can customize using the generation AI to make personalized suggestions based on the user's past purchase history and preferences. The purchase suggestion generation unit, for example, uses the generation AI to make personalized purchase suggestions based on the user's past purchase history and preferences. For example, it prioritizes suggestions for ingredients that the user frequently purchases. The purchase suggestion generation unit also analyzes the user's purchase history data, and builds a system in which the generation AI makes customized purchase suggestions based on that data. For example, it suggests ingredients of specific brands or types. The purchase suggestion generation unit also uses the generation AI to make personalized purchase suggestions based on the user's preferences and past purchase history, and notifies the smartphone. For example, it suggests ingredients needed for a recipe that the user likes. This makes it possible to make personalized suggestions based on the user's past purchase history and preferences.

[0075] The purchase suggestion generation unit can simultaneously provide nutritional information for ingredients and recipe suggestions, thereby supporting a healthy diet. For example, the purchase suggestion generation unit can simultaneously provide nutritional information for ingredients when making purchase suggestions, thereby supporting the user in maintaining a healthy diet. For example, the vitamin and mineral content can be displayed. The purchase suggestion generation unit can also build a system that provides recipes using the suggested ingredients along with purchase suggestions. For example, it can notify users of recipes that suggest balanced meals. The purchase suggestion generation unit can also simultaneously provide nutritional information for ingredients and recipe suggestions, giving advice to help the user maintain a healthy diet. For example, it can suggest low-calorie and high-protein ingredients. This allows for the simultaneous provision of nutritional information for ingredients and recipe suggestions, thereby supporting a healthy diet.

[0076] The purchase proposal generation unit can use the emotion estimation function to analyze the emotion of the user when receiving the proposal and improve the proposal content. For example, the purchase proposal generation unit uses the emotion estimation function to analyze the emotion of the user when receiving the purchase proposal and improve the proposal content based on the data. For example, the purchase proposal generation unit prioritizes proposals with strong positive emotions. The purchase proposal generation unit also builds a system that continuously improves the content of purchase proposals by accumulating user emotion data and analyzing it with the emotion estimation function. For example, it increases proposals that frequently show happy expressions. The purchase proposal generation unit also uses the emotion estimation function to analyze the emotion of the user when receiving the proposal in real time and dynamically adjusts the proposal content based on the data. For example, it changes the proposal content to match the user's emotion. This makes it possible to improve the proposal content based on the user's emotion.

[0077] The purchase proposal generation unit can also send the proposal to other devices such as a smart speaker or a smart watch, allowing the user to receive the proposal anywhere. For example, the purchase proposal generation unit builds a system that sends the purchase proposal to a smart speaker and notifies the user by voice. For example, the purchase proposal is received by voice while cooking in the kitchen. The purchase proposal generation unit also sends the purchase proposal to a smart watch, allowing the user to receive the proposal even when they are out. For example, the proposal is checked on the smart watch while shopping. The purchase proposal generation unit also develops a system that sends the purchase proposal to multiple devices, allowing the user to receive the proposal anywhere. For example, notifications are sent simultaneously to a smartphone, smart speaker, and smart watch. This allows the user to receive the proposal anywhere.

[0078] The purchase proposal generation unit can send the purchase proposal to the smartphones of all family members and coordinate who will be in charge of shopping. For example, the purchase proposal generation unit builds a system that sends purchase proposals to the smartphones of all family members and coordinates who will be in charge of shopping. For example, a shopping list is shared among family members. The purchase proposal generation unit also sends purchase proposals to the smartphones of all family members and automatically assigns who will be in charge of shopping. For example, it notifies the family member who is closest. The purchase proposal generation unit also develops an app that sends purchase proposals to the smartphones of all family members and coordinates who will be in charge of shopping. For example, the shopping list is updated in real time among family members. This makes it possible to send the purchase proposal to the smartphones of all family members and coordinate who will be in charge of shopping.

[0079] The location information detection unit can use a generation AI to predict sale information for stores the user is likely to visit based on location information and notify the user in advance. For example, the location information detection unit can build a system that uses a generation AI to predict sale information for stores the user is likely to visit based on location information and notify the user in advance. For example, the location information detection unit analyzes the user's movement patterns to provide sale information. The location information detection unit also uses a generation AI to predict sale information based on the user's location information and past purchase history and notify the smartphone. For example, it notifies the user in advance of sale information for specific stores. The location information detection unit can also develop a system that predicts sale information for stores the user is likely to visit in real time based on location information and notifies the user. For example, sale information is sent when the user approaches a nearby store. This allows the user to be notified in advance of sale information for stores the user is likely to visit.

[0080] The location information detection unit can provide customized coupons based on a user's purchasing history and preferences. The location information detection unit, for example, builds a system that provides customized coupons based on a user's purchasing history and preferences. For example, it issues discount coupons for frequently purchased ingredients. The location information detection unit also analyzes purchasing history data, and a generation AI provides customized coupons based on that data. For example, it issues coupons for specific brands or types of ingredients. The location information detection unit also develops a system that notifies a smartphone of customized coupons based on a user's preferences and purchasing history. For example, it provides coupons for ingredients needed for a user's favorite recipe. This makes it possible to provide customized coupons based on a user's purchasing history and preferences.

[0081] The location information detection unit can use the emotion estimation function to analyze the emotions of the user when receiving price information and improve the notification content. For example, the location information detection unit can use the emotion estimation function to analyze the emotions of the user when receiving price information and improve the notification content based on the data. For example, notifications with strong positive emotions can be prioritized. The location information detection unit can also build a system that continuously improves the notification content of price information by accumulating user emotion data and analyzing it with the emotion estimation function. For example, it can increase the number of notifications that frequently show happy expressions. The location information detection unit can also use the emotion estimation function to analyze the emotions of the user when receiving price information in real time and dynamically adjust the notification content based on the data. For example, it can change the notification content to match the user's emotions. This makes it possible to improve the notification content based on the user's emotions.

[0082] The location information detection unit can optimize and suggest an efficient shopping route based on the user's commuting route and daily movement patterns. The location information detection unit, for example, analyzes the user's commuting route and daily movement patterns to build a system that suggests the optimal shopping route. For example, it suggests stores that can be stopped off on the way to work. The location information detection unit also uses generative AI to suggest an efficient shopping route based on the user's movement patterns. For example, it calculates a route that efficiently visits multiple stores. The location information detection unit also develops a system that suggests the optimal shopping route in real time based on the user's commuting route and daily movement patterns. For example, it adjusts the route taking traffic conditions and store opening hours into consideration. This makes it possible to suggest an efficient shopping route based on the user's commuting route and daily movement patterns.

[0083] The location information detection unit can be linked with smart home devices to allow users to receive sale information even when they are at home. For example, the location information detection unit can link location information detection and price information notification with smart home devices to build a system that allows users to receive sale information even when they are at home. For example, sale information can be notified by voice using a smart speaker. The location information detection unit can also link with smart home devices to allow users to receive sale information in real time even when they are at home. For example, sale information can be displayed on a smart display. The location information detection unit can also integrate location information detection and price information notification with smart home devices to develop a system that allows users to receive sale information even when they are at home. For example, sale information can be notified by changing the color of a smart light. This makes it possible to receive sale information even when users are at home.

[0084] The location information detection unit can use the emotion estimation function to analyze the emotions of the user when receiving price information and improve the notification content. For example, the location information detection unit can use the emotion estimation function to analyze the emotions of the user when receiving price information and improve the notification content based on the data. For example, notifications with strong positive emotions can be prioritized. The location information detection unit can also build a system that continuously improves the notification content of price information by accumulating user emotion data and analyzing it with the emotion estimation function. For example, it can increase the number of notifications that frequently show happy expressions. The location information detection unit can also use the emotion estimation function to analyze the emotions of the user when receiving price information in real time and dynamically adjust the notification content based on the data. For example, it can change the notification content to match the user's emotions. This makes it possible to improve the notification content based on the user's emotions.

[0085] The purchase suggestion generation unit uses generation AI to analyze data and learn the user's food ingredient preferences and consumption patterns, enabling more accurate suggestions. The purchase suggestion generation unit, for example, uses generation AI to analyze the user's food ingredient purchase history and learn the preferences and consumption patterns. For example, it identifies frequently purchased ingredients and brands and makes suggestions based on that. The purchase suggestion generation unit also uses generation AI to analyze the user's purchase history data and builds a system that learns consumption patterns. For example, it makes suggestions based on ingredients purchased on specific days of the week or time periods. The purchase suggestion generation unit also uses generation AI to learn the user's food ingredient preferences and consumption patterns and provides more accurate suggestions. For example, it suggests new ingredients and recipes based on past purchase history. This allows the unit to learn the user's food ingredient preferences and consumption patterns and provide more accurate suggestions.

[0086] The purchase suggestion generation unit can make food ingredient suggestions that take into account the user's health condition and nutritional balance. The purchase suggestion generation unit, for example, builds a system that makes food ingredient suggestions that take into account the user's health condition and nutritional balance. For example, suggestions are made based on health checkup results and diet records. The purchase suggestion generation unit also uses generation AI to analyze the user's health condition and nutritional balance and make food ingredient suggestions based on that. For example, if a user is lacking in a particular nutrient, it suggests food ingredients that contain that nutrient. The purchase suggestion generation unit also develops a system that makes food ingredient suggestions that take into account nutritional balance based on the user's health data. For example, it suggests low-calorie food ingredients to a user who is on a diet. This makes it possible to make food ingredient suggestions that take into account the user's health condition and nutritional balance.

[0087] The purchase suggestion generation unit can use the emotion estimation function to analyze the emotion of the user when receiving the suggestion and improve the suggestion content. For example, the purchase suggestion generation unit can use the emotion estimation function to analyze the emotion of the user when receiving the ingredient suggestion and improve the suggestion content based on the data. For example, the purchase suggestion generation unit can prioritize suggestions that show strong positive emotions. The purchase suggestion generation unit can also build a system that continuously improves the content of ingredient suggestions by accumulating user emotion data and analyzing it with the emotion estimation function. For example, the purchase suggestion generation unit can increase suggestions that frequently show happy expressions. The purchase suggestion generation unit can also use the emotion estimation function to analyze the emotion of the user when receiving the suggestion in real time and dynamically adjust the suggestion content based on the data. For example, the suggestion content can be changed to match the user's emotion. This allows the suggestion content to be improved based on the user's emotion.

[0088] The purchase suggestion generation unit can share purchase history data with other users and promote community-based ingredient suggestions and recipe exchanges. The purchase suggestion generation unit, for example, shares purchase history data with other users and builds a system that promotes community-based ingredient suggestions and recipe exchanges. For example, recipes using the same ingredients are shared. The purchase suggestion generation unit also develops a system that shares users' purchase history data within a community and receives suggestions from other users. For example, it proposes new recipes using specific ingredients. The purchase suggestion generation unit also builds a platform that promotes community-based ingredient suggestions and recipe exchanges based on the purchase history data. For example, users share how to use and store ingredients. This allows purchase history data to be shared with other users and promotes community-based ingredient suggestions and recipe exchanges.

[0089] The purchase suggestion generation unit can make food suggestions that match seasonal special suggestions and events. For example, the purchase suggestion generation unit builds a system that makes food suggestions that match seasonal special suggestions and events based on purchase history data. For example, it suggests food ingredients that match Christmas and Halloween. The purchase suggestion generation unit also uses generation AI to make food suggestions that match seasonal special suggestions and events. For example, it suggests ingredients for barbecues in the summer. The purchase suggestion generation unit also develops a system that makes food suggestions that match seasonal special suggestions and events based on purchase history data. For example, it suggests recipes that use seasonal ingredients. This makes it possible to make food suggestions that match seasonal special suggestions and events.

[0090] The purchase suggestion generation unit can use the emotion estimation function to analyze the emotion of the user when receiving the suggestion and improve the suggestion content. For example, the purchase suggestion generation unit can use the emotion estimation function to analyze the emotion of the user when receiving the ingredient suggestion and improve the suggestion content based on the data. For example, the purchase suggestion generation unit can prioritize suggestions that show strong positive emotions. The purchase suggestion generation unit can also build a system that continuously improves the content of ingredient suggestions by accumulating user emotion data and analyzing it with the emotion estimation function. For example, the purchase suggestion generation unit can increase suggestions that frequently show happy expressions. The purchase suggestion generation unit can also use the emotion estimation function to analyze the emotion of the user when receiving the suggestion in real time and dynamically adjust the suggestion content based on the data. For example, the suggestion content can be changed to match the user's emotion. This allows the suggestion content to be improved based on the user's emotion.

[0091] The food sensor uses a camera to analyze food label and packaging information and automatically read expiration dates. For example, the food sensor uses a camera inside a refrigerator to periodically take images and analyze those images with a generation AI. For example, the sensor estimates freshness based on changes in the color and shape of vegetables and identifies food items that are close to their expiration date. The food sensor also uses a generation AI to analyze food label and packaging information in the refrigerator and automatically read expiration dates. For example, it obtains expiration dates by scanning barcodes or QR codes. To estimate food freshness, the sensor uses an image analysis system with the generation AI to detect discoloration or mold on the surface of food items. For example, it evaluates freshness based on spots and discoloration that appear on the surface of fruit. This allows the sensor to analyze food label and packaging information and automatically read expiration dates.

[0092] The food sensor uses a camera to detect discoloration and mold growth on the surface of food ingredients and evaluate their freshness. For example, the food sensor uses a camera inside the refrigerator to periodically take images and analyze those images with a generation AI. For example, it estimates freshness based on changes in the color and shape of vegetables and identifies food ingredients that are close to their expiration date. The food sensor also uses a generation AI to analyze the labels and packaging information of food ingredients in the refrigerator and automatically read expiration dates. For example, it obtains expiration dates by scanning barcodes or QR codes. To estimate the freshness of food ingredients, the generation AI also performs image analysis to detect discoloration and mold growth on the surface of food ingredients. For example, it evaluates freshness based on spots and discoloration that appear on the surface of fruit. This makes it possible to detect discoloration and mold growth on the surface of food ingredients and evaluate their freshness.

[0093] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0094] The food sensor can analyze the nutritional value of ingredients in the refrigerator and suggest healthy eating habits to the user. For example, it can use a camera and generative AI to identify the type of ingredient and calculate its nutritional value. The food sensor can also suggest ingredients that take into account the user's health condition and nutritional balance. For example, if a user is lacking in a particular nutrient, it can suggest ingredients that contain that nutrient. The food sensor can also build a system that suggests balanced meals based on the nutritional value of ingredients. This allows users to obtain information to maintain a healthy diet.

[0095] The food sensor can analyze allergen information for food items in the refrigerator and suggest allergy countermeasures to the user. For example, it can use a camera and generative AI to read food labels and extract allergen information. The food sensor can also identify food items containing allergens based on the user's allergy information and issue a warning. For example, it can notify the user if an ingredient containing an allergen is stored in the refrigerator. The food sensor can also build a system that suggests alternative ingredients to prevent allergies based on the allergen information. This allows users to obtain information to take measures against allergies.

[0096] The ingredient detection unit can use the emotion estimation function to analyze the emotion of the user when opening the refrigerator and optimize the placement of ingredients based on the emotion toward a particular ingredient. For example, ingredients that the user frequently expresses joy in can be placed in a location that makes them easy to access. The ingredient detection unit can also use the emotion estimation function to place ingredients that the user frequently expresses dissatisfaction in a location that is less visible. For example, ingredients that the user dislikes can be placed at the back of the refrigerator. The ingredient detection unit can also build a system that continuously optimizes the placement of ingredients by accumulating user emotion data and analyzing it with the emotion estimation function. This makes it possible to optimize the placement of ingredients based on the user's emotions.

[0097] The food sensor can analyze the eco-footprint of ingredients in the refrigerator and suggest environmentally friendly food choices to the user. For example, it can use a camera and generative AI to identify the type of ingredient and calculate the eco-footprint based on its production process and transportation distance. The food sensor can also suggest environmentally friendly ingredients based on the eco-footprint data. For example, it can prioritize locally produced ingredients and organic ingredients. The food sensor can also build a system that uses the eco-footprint data to advise users on environmentally friendly food choices. This allows users to obtain information to make environmentally friendly food choices.

[0098] The ingredient detection unit can use the emotion estimation function to analyze the emotion expressed when a user opens the refrigerator and manage the expiration dates of ingredients based on the emotion expressed toward a particular ingredient. For example, the ingredient detection unit can prioritize notification of the expiration date of ingredients for which the user frequently displays a happy expression. The ingredient detection unit can also use the emotion estimation function to notify the expiration date of ingredients for which the user displays a dissatisfied expression so as not to miss it. For example, an alert can be sent when the expiration date of an ingredient that the user dislikes is approaching. The ingredient detection unit can also build a system that continuously optimizes ingredient expiration date management by accumulating user emotion data and analyzing it with the emotion estimation function. This makes it possible to manage ingredient expiration dates based on the user's emotions.

[0099] The food sensor can analyze how food is stored in the refrigerator and suggest optimal storage methods to users. For example, it can use a camera and generative AI to identify the type of food and analyze its storage method. The food sensor can also suggest optimal storage conditions based on the food storage method. For example, it can recommend storing food at a specific temperature and humidity. The food sensor can also build a system that gives users advice on storage methods based on storage method data. This allows users to obtain information to optimize their food storage methods.

[0100] The ingredient detection unit can use the emotion estimation function to analyze the emotion expressed when a user opens the refrigerator and adjust the frequency of food purchases based on the emotion expressed toward a particular ingredient. For example, the ingredient detection unit can set a higher purchase frequency for ingredients for which the user frequently displays a happy expression. The ingredient detection unit can also use the emotion estimation function to set a lower purchase frequency for ingredients for which the user frequently displays a dissatisfied expression. For example, the ingredient detection unit can reduce the purchase frequency of ingredients that the user dislikes. The ingredient detection unit can also build a system that accumulates user emotion data and analyzes it with the emotion estimation function to continuously adjust the frequency of food purchases. This makes it possible to adjust the frequency of food purchases based on the user's emotion.

[0101] The ingredient detection unit can analyze the calorie information of ingredients in the refrigerator and make suggestions to the user to support calorie management. For example, it can use a camera and generation AI to identify the type of ingredient and calculate its calorie information. The ingredient detection unit can also make ingredient suggestions to support calorie management based on the user's calorie intake. For example, it can prioritize suggestions of low-calorie ingredients. The ingredient detection unit can also build a system that gives calorie management advice to the user based on the calorie information. This allows the user to obtain information to support calorie management.

[0102] The ingredient detection unit can use the emotion estimation function to analyze the emotion expressed when a user opens the refrigerator and predict ingredient consumption patterns based on the emotion expressed toward a particular ingredient. For example, the ingredient detection unit can set a higher consumption pace for ingredients for which the user frequently displays a happy expression. The ingredient detection unit can also use the emotion estimation function to set a lower consumption pace for ingredients for which the user frequently displays a dissatisfied expression. For example, the ingredient detection unit can set a slower consumption pace for ingredients that the user dislikes. The ingredient detection unit can also build a system that continuously predicts ingredient consumption patterns by accumulating user emotion data and analyzing it with the emotion estimation function. This makes it possible to predict ingredient consumption patterns based on the user's emotions.

[0103] The food sensor can analyze the storage period of food ingredients in the refrigerator and suggest the optimal time to consume them to the user. For example, it can use a camera and generative AI to identify the type of food ingredient and calculate its storage period. The food sensor can also suggest the optimal time to consume ingredients based on the storage period data. For example, it can recommend that ingredients with an approaching expiration date be consumed first. The food sensor can also build a system that advises the user on the optimal time to consume ingredients based on the storage period data. This allows the user to obtain information to determine the optimal time to consume ingredients.

[0104] The processing flow of the second embodiment will be briefly explained below.

[0105] Step 1: The food detector detects the food in the refrigerator. For example, a camera takes an image of the inside of the refrigerator, and the generation AI analyzes the image to identify the type and quantity of food. The sensor also detects the presence of food based on weight and location information. Step 2: The consumption rate measurement unit measures the consumption rate of the ingredients detected by the ingredient detection unit. For example, it predicts the consumption rate of ingredients based on past data and calculates the rate at which each ingredient is being consumed. It can also measure the consumption rate of ingredients in real time. Step 3: The purchase suggestion generator generates a purchase suggestion based on the consumption rate measured by the consumption rate measurement unit. For example, it sends a notification to the smartphone saying, "You're running low on milk. Please buy some next time you go shopping." The suggestion content can also be customized based on the user's preferences and past purchase history. Step 4: The location information detection unit detects the user's location information. For example, it can obtain the user's location information using the smartphone's GPS function. It can also analyze the user's movement patterns and update the location information at the optimal timing. Step 5: The price information notification unit notifies the user of price information based on the location information detected by the location information detection unit. For example, when the user approaches a supermarket, it sends a notification such as, "Milk is on sale at this supermarket." It can also notify users of customized price information based on their purchasing history and preferences.

[0106] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0107] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0108] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0109] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0110] 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0111] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0112] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0113] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0115] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0116] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0117] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0118] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0119] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 may have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0120] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0121] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0122] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0123] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0124] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0125] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0126] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0127] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0128] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0129] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0130] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0131] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0132] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0133] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0134] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 may also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0135] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0136] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0137] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0138] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0139] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0140] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0141] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0142] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0143] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0145] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0146] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0147] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0148] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0149] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0150] In the robot 414, the processor 46 performs the identification process. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 may have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0151] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0152] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0153] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0154] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0155] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0156] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0157] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0158] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0159] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0160] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0161] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0162] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0163] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0164] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0165] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0166] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0167] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0168] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0169] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0170] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0171] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0172] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0173] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. an ingredient detection unit that detects ingredients in the refrigerator; a reduction rate measuring unit for measuring the reduction rate of the ingredients sensed by the ingredient sensing unit; a purchase proposal generation unit that generates a purchase proposal based on the decrease pace measured by the decrease pace measurement unit; a location information detection unit that detects location information of a user; a price information notification unit that notifies price information based on the location information detected by the location information detection unit. A system characterized by:

2. The food ingredient sensing unit Expand to the freezer and pantry to centralize food management in the home 2. The system of claim 1.

3. The decrease pace measuring unit The generative AI is used to compare past consumption data with current trends to make more accurate predictions.

2. The system of claim 1.

4. The purchase proposal generation unit The generative AI customizes and makes personalized suggestions based on the user's past purchase history and preferences.

2. The system of claim 1.

5. The position information detection unit Based on the location information, the generation AI predicts sale information for stores that the user is likely to visit and notifies the user in advance.

2. The system of claim 1.

6. The food ingredient sensing unit Analyzing the facial expression of the user when they open the refrigerator and estimating their preferences and dislikes for specific ingredients 2. The system of claim 1.

Images