system

Abstract

We provide the system. [Solution] Image acquisition method, As a preprocessing means, means for removing unnecessary parts from the image and converting it to a format suitable for analysis, The transmission means includes means for transmitting a pre-processed image to a remote processing device, As an image analysis means, there is a means for analyzing the received image and performing quality evaluation of the object, As a means for generating evaluation information, it includes a means for suggesting appropriate expiration dates and consumption methods based on quality evaluation results, As a means of notifying information, a means of displaying the generated evaluation information to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that food loss occurs and optimal consumption is difficult because consumers cannot appropriately grasp the freshness and nutritional value of food. This problem affects not only ordinary households but also proper inventory management and product provision in the food service industry. Also, since it is difficult for consumers to make diet choices considering nutritional value, there is a risk of leading to an unhealthy diet.

Means for Solving the Problems

[0005] This invention involves acquiring images of food using image acquisition means, and then converting those images into a format suitable for analysis using preprocessing means. Next, the preprocessed images are transmitted to a remote server using transmission means. On the server side, the received images are analyzed by image analysis means to evaluate the quality of the target food. Based on this evaluation, evaluation information generation means provides the user with information such as the expiration date and appropriate consumption methods. Furthermore, the generated evaluation information is displayed on the user terminal using information notification means to support the user in making optimal consumption decisions. In this way, the invention provides a system that enables the reduction of food waste and the realization of healthy food choices.

[0006] "Image acquisition means" refers to a device or function used to acquire images of food.

[0007] "Preprocessing means" refers to a device or function that performs processing to remove unnecessary parts from an acquired image and convert it into a format suitable for analysis.

[0008] "Transmission means" refers to a device or function used to securely and efficiently transmit pre-processed images to a remote server.

[0009] "Image analysis means" refers to a device or function that analyzes image data received on a server and performs quality evaluation of food products.

[0010] "Evaluation information generation means" refers to a device or function that generates information regarding expiration dates and consumption methods based on quality evaluations obtained by image analysis means.

[0011] "Information notification means" refers to a device or function used to notify users of generated information such as expiration dates and consumption methods. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a tagged processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, a tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, a tagged storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, a tagged communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] To implement this invention, the user first takes a photograph of food using a smartphone or other device. A dedicated application is installed on this device, and the user activates the camera through the application and takes a photograph of the food. The captured image is converted into a format suitable for analysis by the device's pre-processing function. During this process, the background is removed and necessary parts are cropped.

[0034] Next, the terminal sends the pre-processed images to a server in the cloud. The server uses image analysis algorithms to analyze the received images and evaluate the freshness and nutritional value of the food. This analysis includes techniques for analyzing image characteristics such as hue and shape. Furthermore, the server generates evaluation information based on the analysis results. This information includes recommended expiration dates, cooking methods, and storage methods.

[0035] The generated evaluation information is sent to the user's device via the information notification function. The device then displays this information to the user through the app. Based on the displayed information, the user can plan to consume food appropriately.

[0036] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image and evaluates its current freshness. The evaluation information generated would include something like, "The apple is fresh and it is recommended to consume it within two days. Refrigeration is the best place to store it." Based on this information, the user can decide when to consume the apple and avoid waste.

[0037] This series of processes ensures that users can always consume fresh food as needed, thereby contributing to a reduction in food waste. Furthermore, the nutritional information provided based on evaluation data helps users make healthy food choices.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The user launches their smartphone's camera app and takes a picture of the food they want to evaluate. The user is expected to adjust the lighting and angle appropriately during shooting to obtain an image best suited for analysis.

[0041] Step 2:

[0042] The device receives the captured image and applies a preprocessing algorithm. This includes removing unwanted backgrounds from the image and cropping it to include only the necessary parts. The image's brightness and contrast are also adjusted to prepare it for optimal analysis.

[0043] Step 3:

[0044] The terminal sends pre-processed image data to the cloud server using a communication protocol. This transmission is encrypted to ensure security.

[0045] Step 4:

[0046] The server runs an AI model to analyze the received images. This model analyzes the color, shape, and surface features of the food in the images to evaluate its freshness and nutritional value.

[0047] Step 5:

[0048] Based on the analysis results, the server generates evaluation information regarding expiration dates, optimal consumption methods, and storage methods. In doing so, it creates personalized recommendations that take into account accumulated data and the user's past history.

[0049] Step 6:

[0050] The server sends the generated evaluation information to the user's terminal. The transmitted data is formatted in an easy-to-read format, ensuring that the content is easy for the user to understand.

[0051] Step 7:

[0052] The device displays the received information to the user. Through notifications and in-app displays, users can check the status of food and recommendations, and make concrete consumption plans.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] Managing food freshness and reducing food waste are critical challenges facing modern consumers. In particular, food waste often arises from a lack of knowledge about expiration dates and proper storage methods, making consumers' lives inconvenient. Providing nutritional information to support healthy food choices is also crucial. Therefore, there is a need to provide systems that enable consumers to properly manage and effectively consume food.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes, as an image analysis means, a means for analyzing received images using a dedicated algorithm and evaluating the freshness and nutritional value of the object; as an evaluation information generation means, a means for presenting recommended expiration dates, storage methods, and cooking methods based on the analysis results; and as an information notification means, a means for displaying the generated evaluation information to the user via a terminal. This makes it possible for consumers to immediately obtain information to properly manage food and make healthy food choices.

[0058] "Image acquisition means" refers to devices or software functions for photographing food, which acquire image data captured by the user.

[0059] "Preprocessing means" refers to a function that removes the background and unnecessary parts from the acquired image and converts the image into a format suitable for analysis.

[0060] "Transmission means" refers to a communication function for securely transmitting pre-processed image data to a remote processing unit.

[0061] "Image analysis means" refers to a function that analyzes received images using specialized algorithms and machine learning models to evaluate the freshness and nutritional value of food.

[0062] "Evaluation information generation means" refers to a function that generates information such as recommended expiration dates, storage methods, and cooking methods for food products based on the results of image analysis.

[0063] "Information notification means" refers to a function that provides generated evaluation information to the user's terminal and displays it to the user.

[0064] The invention will now be described in terms of its embodiments. This invention is a system that operates primarily around a user's terminal and a server in the cloud.

[0065] First, the user uses a smartphone or other device to photograph the food. A dedicated application is installed on the device, and this application controls the camera function to acquire images of the food. While the standard smartphone camera function is used for image acquisition, additional software correction may be performed to further improve accuracy.

[0066] Next, the device preprocesses the acquired image immediately after capture. Here, image processing algorithms are used to remove unwanted background elements and extract only the food portion. Color correction and noise reduction are also performed, and the image is converted into a format suitable for analysis. This preprocessing may utilize known image processing algorithms or image processing libraries from specific manufacturers.

[0067] The pre-processed images are sent from the terminal to a server in the cloud. This transmission uses an internet connection and is encrypted to ensure security.

[0068] Images that reach the server are processed by a dedicated image analysis algorithm. The server analyzes the received image data to estimate the freshness and nutritional value of the food. This analysis uses a generative AI model, which automatically analyzes features such as the hue, shape, and texture of the food.

[0069] Based on the analysis results, the server generates evaluation information. This information includes the recommended expiration date, storage method, and cooking method for the food. This evaluation information is transmitted to the terminal in real time for the user to receive.

[0070] Finally, the device displays the received evaluation information to the user through the application. Based on this information, the user can make plans, for example, on when to consume food and how to store it.

[0071] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image, assesses its freshness, and provides information such as, "It is recommended to consume the apple within two days and store it in the refrigerator." In this way, it becomes easier to plan food consumption.

[0072] An example of a prompt to input into a generative AI model is, "Analyze this picture of an apple and tell me the recommended expiration date and storage method."

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The user launches the application on their device and uses the camera function to take a picture of food. The input at this time is the image within the frame of the food that the user sees through the camera. Specifically, the user saves the image data to the device's memory by pressing the "Capture" button in the app.

[0076] Step 2:

[0077] The device preprocesses the acquired image. The input is the image taken in step 1, and the output is a cleared image of the food with the background removed. Specifically, the device uses an image processing algorithm to perform noise reduction, background removal, and color information correction.

[0078] Step 3:

[0079] The terminal sends the pre-processed image to the server. The input to this transmission is the pre-processed image that will be the output of step 2, and the output is the image data that the cloud server receives. During transmission, the terminal encrypts the data to ensure secure communication.

[0080] Step 4:

[0081] The server analyzes the received images. The input is the pre-processed image data sent in step 3. The server uses a dedicated image analysis algorithm, particularly a generative AI model, to analyze the hue and shape of the food and estimate its freshness and nutritional value. The output is evaluation data of the analysis results.

[0082] Step 5:

[0083] The server generates evaluation information based on the analysis results. The input is the analysis results, which are the output of step 4. Specifically, the server generates text information such as recommended expiration dates, storage methods, and cooking methods, and formats it in a way that is easy for the user to understand. The output is text data as evaluation information.

[0084] Step 6:

[0085] The server sends evaluation information to the terminal. The input for this transmission is the evaluation information generated in step 5. The output is data formatted in a format that can be displayed on the terminal. The server transmits this to the terminal via the internet.

[0086] Step 7:

[0087] The terminal displays the received evaluation information to the user via the application. The input is the evaluation information sent from the server in step 6. Specifically, the terminal utilizes its notification function to display the expiration date and storage method on the user interface. The output is the presentation of information to the user in a visually understandable format.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] Managing food at home is time-consuming, which can lead to a decline in freshness, loss of nutritional value, and unnecessary waste. Furthermore, while there is a need to reduce the burden of daily household tasks and perform them more efficiently, there is a challenge in autonomously managing food.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for presenting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, a means for identifying the object to be photographed in the physical environment, an autonomous means for identifying the object to be photographed in the physical storage device, and a means for periodically scanning the object in the physical storage device based on defined criteria. This automates food management in the home, enabling efficient and waste-free consumption.

[0093] "Image acquisition means" refers to the function of collecting visual data of an object using a photographic device.

[0094] "Preprocessing means" refers to a function that extracts necessary information from acquired images and converts it into a format suitable for analysis.

[0095] "Transmission means" refers to the function of transferring processed data to a remote processing unit via a communication device.

[0096] "Image analysis means" refers to a function that analyzes received visual data and evaluates the characteristics and state of an object.

[0097] "Evaluation information generation means" refers to a function that generates useful information about an object based on the analysis results.

[0098] "Information notification means" refers to a function for transmitting and displaying generated evaluation information to users.

[0099] "Means for identifying objects to be photographed within a physical environment" refers to a system that has the function of identifying and accurately recognizing objects within a storage device.

[0100] "Means for periodically scanning objects within a physical storage device based on defined criteria" refers to a function that continuously monitors objects within a storage device and collects status information according to set conditions.

[0101] The system implementing this invention is designed to automate food management within the home. The system includes a series of processes for image acquisition, preprocessing, image analysis, evaluation information generation, and information notification.

[0102] The server uses Python and TENSORFLOW® to execute the analysis algorithm. A home robot has a built-in camera and periodically photographs food as an image acquisition tool. This robot is based on iRobot and other common home robots and has the ability to autonomously identify food in the refrigerator. The captured images are preprocessed to remove the background and convert them into a format suitable for analysis. This processing includes image cropping and noise reduction.

[0103] Next, the terminal sends the pre-processed image to a cloud server. The server receives the transmitted data and uses image analysis tools to evaluate the condition of the food. Here, the hue, shape, and other visual features of the image are analyzed to evaluate the freshness and nutritional value of the food.

[0104] The server generates evaluation information based on the analysis results. The evaluation information generation means generates information on appropriate expiration dates and storage methods, and sends it to the terminal via the cloud. The terminal displays the received notification to the user in an easy-to-use format and provides recommendations regarding the condition of the food.

[0105] For example, if a user has tomatoes stored in their refrigerator, the robot can take a picture of them, and the server will analyze the image to generate information such as, "The tomatoes are ripe and should be consumed within three days." This promotes the proper consumption of food and reduces waste.

[0106] An example of a prompt for a generative AI model can be set as follows: "Analyze images of food in a refrigerator and recommend freshness assessment and expiration date." By entering this prompt, the system can perform the correct analysis and provide information.

[0107] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0108] Step 1:

[0109] The device uses a home robot's camera to photograph the food inside the refrigerator. The input is a physical image of the food, and the output is digital image data. This data is needed for the following processing.

[0110] Step 2:

[0111] The terminal preprocesses the captured image. The input is the digital image data obtained in step 1. Preprocessing reduces noise and converts the image into a format suitable for analysis by removing the background and cropping the food portion. The output is the preprocessed image.

[0112] Step 3:

[0113] The terminal sends the pre-processed image to the cloud server. The input is the pre-processed image obtained in step 2, and the output is the state information uploaded to the cloud. This state information is used in the next analysis step.

[0114] Step 4:

[0115] The server analyzes the image data received on the cloud. The input is the pre-processed image sent in step 3. The server uses image analysis tools to analyze the visual features of the image, such as hue and shape, and evaluates the freshness and nutritional value of the food. The output is the evaluation result.

[0116] Step 5:

[0117] The server generates evaluation information based on the evaluation results. The input is the evaluation results obtained in step 4. The server generates and outputs information regarding appropriate expiration dates and storage methods. This information is provided to the user.

[0118] Step 6:

[0119] The server sends the generated evaluation information to the terminal. The input is the evaluation information generated in step 5. The output is the evaluation information sent to the terminal.

[0120] Step 7:

[0121] The terminal notifies the user of the received evaluation information. The input is the evaluation information sent in step 6. Based on the evaluation information, the user can properly manage food and create a consumption plan. The output is specific instructions displayed on the terminal's display.

[0122] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0123] This invention evaluates the freshness and nutritional value of food and, using a system equipped with an emotion engine, provides personalized suggestions tailored to the user's emotional state. The system begins with the user taking a photograph of the food using a smartphone or other device, and then recognizes the user's emotions through the emotion engine.

[0124] Specifically, the user takes a picture of food using the device's camera app. During this process, the emotion engine uses the device's front camera and sensors to analyze the user's facial expressions and tone of voice to recognize their emotions. For example, it can obtain information such as whether the user is tired or stressed. The device preprocesses this information and then sends both the image data and emotion data to a cloud server.

[0125] The server analyzes received images and evaluates the freshness and nutritional value of the food. Based on emotional data, it also adjusts the content of the information notifications; for example, suggesting new recipes when the user is in a positive state and recommending simpler cooking methods when they are in a negative state. The evaluation information generation function combines the user's current emotional data with their past usage history to suggest the optimal consumption plan. It can even provide specific suggestions tailored to emotional states, such as "use ingredients with relaxing effects when you're feeling down."

[0126] The server sends the generated evaluation information and suggestions to the terminal, which then displays them to the user. The display is presented in an easy-to-read format that takes the user's emotional state into consideration, encouraging consumer behavior without requiring any user intervention. For example, if the user is feeling stressed, the notification is displayed in a softer tone, and the suggestions are concise to reduce the burden.

[0127] This system allows users to make optimal food choices based on their emotional state at the time, supporting a health-conscious diet. In addition to reducing food waste, using this system can also contribute to users' emotional management.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The user activates their smartphone's camera and takes a picture of the food they want to evaluate. Simultaneously with the photo, the device's front camera and microphone capture the user's facial expressions and voice. This allows the user's emotional state to be recorded at the same time.

[0131] Step 2:

[0132] The device preprocesses the captured images, including cropping food portions and correcting image color. Simultaneously, the emotion engine analyzes the acquired audio and facial expression data, and sets an emotion category (e.g., stress, exhilaration, calmness) in real time.

[0133] Step 3:

[0134] The device pairs pre-processed image data with emotion data and sends it to the cloud server. The transmission is encrypted based on a security protocol.

[0135] Step 4:

[0136] After receiving image data, the server applies an AI image recognition algorithm to analyze the type of food, its freshness, and its nutritional value. It also receives emotional data simultaneously and compares it with a database to enable suggestions that take the user's emotional state into account.

[0137] Step 5:

[0138] Based on the analysis results, the server generates suggestions for expiration dates and cooking methods. Furthermore, it uses emotional data to customize suggestions according to the user's emotional state, for example, "If you're feeling stressed, make a simple and nutritious smoothie."

[0139] Step 6:

[0140] The server sends the generated evaluation information and suggestions to the terminal. The information sent is organized and formatted in an easy-to-understand format.

[0141] Step 7:

[0142] The device notifies the user of the received information and displays it within the app. The displayed content is adjusted according to the user's emotional state and is designed to be received appropriately. Based on this, the user can choose the optimal way to consume food.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Conventional food quality evaluation systems only assess freshness and nutritional value, failing to consider the user's emotional state when making recommendations. Therefore, it was difficult to provide personalized recommendations based on the user's emotional state, making it challenging to improve user satisfaction and convenience. Furthermore, there is a need to optimize food consumption plans in conjunction with emotional states to support a healthy and mentally fulfilling diet.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes an image analysis means for analyzing received images and evaluating the quality of the object, an emotion analysis means for analyzing the user's emotional state and considering the analysis results, and an evaluation information generation means for presenting an appropriate consumption plan and consumption method based on the quality evaluation results and emotion analysis results. This enables personalized food suggestions that take into account the user's emotional state, making it possible to realize healthier and more emotionally considerate food consumption.

[0148] "Image acquisition means" refers to a device or software that acquires an image of an object using a device.

[0149] "Preprocessing means" refers to a device or software that removes unnecessary parts from acquired image data and converts it into a format suitable for analysis.

[0150] "Transmission means" refers to a device or protocol that transmits pre-processed data to a processing device located remotely.

[0151] "Image analysis means" refers to a device or algorithm that analyzes received image data and evaluates the quality and condition of the object from it.

[0152] "Emotional analysis means" refers to a device or program that analyzes a user's facial expressions and voice data to determine their emotional state.

[0153] "Evaluation information generation means" refers to a device or system for formulating and proposing appropriate consumption plans and methods based on quality evaluation results and sentiment analysis results.

[0154] "Information notification means" refers to a device or interface that notifies the user of generated evaluation information visually or audibly.

[0155] This invention provides a system that allows users to evaluate the quality of food and receive personalized consumption suggestions based on their emotional state. The system primarily consists of terminals and servers, each playing a specific role.

[0156] The user first takes a picture of the food using the device's image acquisition mechanism. The device is equipped with a camera application, which allows the user to easily record the food. The captured image is pre-processed to remove unnecessary parts and converted into a format suitable for analysis. Furthermore, the device uses emotion analysis to analyze the user's facial expressions and voice tone through the user's front camera and microphone, and acquires the user's emotion data.

[0157] Subsequently, this data is transmitted to a server via a transmission device. The server has an image analysis device that analyzes the received image data to evaluate the freshness and nutritional value of the food. The server also uses an emotion analysis device to generate evaluation information that takes into account the user's emotional state.

[0158] Based on the user's quality evaluation results and sentiment analysis results, the evaluation information generation system suggests optimal consumption plans and methods. For example, if the user is feeling stressed, the system suggests recipes with short cooking times and ingredients that have a relaxing effect. The server then uses an information notification system to transmit the generated evaluation information to the terminal.

[0159] The device visually displays the received information to the user. The display format is designed to be easy to see and understand, based on the user's emotional state. For example, if the user is feeling down, notifications might be displayed in soft colors and with concise messages.

[0160] Furthermore, an example of a prompt using a generative AI model is, "I'd like to have a relaxing meal today. Please recommend some ingredients." Based on this prompt, the AI ​​will provide appropriate food suggestions.

[0161] These features of the system allow users to make food choices that suit their emotions and health, supporting a healthy and comfortable lifestyle.

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The user launches the camera app on their device and takes a picture of the food they intend to consume. The image of the food is captured by the device's camera as input. Specifically, the user holds the apple up to the camera and presses the shutter button. This image data is then sent to the next processing step.

[0165] Step 2:

[0166] The terminal processes the captured image data using a preprocessing mechanism. Specifically, it removes unwanted backgrounds from the image data and converts it to a resolution suitable for subsequent analysis. The input for this process is the food image acquired in the previous step, and the output is image data in a format suitable for analysis.

[0167] Step 3:

[0168] The device analyzes the user's emotional state using emotion analysis techniques. It uses a front camera and microphone to capture the user's facial expressions and voice as input, and generates emotion data as a result of the analysis. For example, it might capture the user taking a deep breath and output emotion data such as "relaxed."

[0169] Step 4:

[0170] The device transmits pre-processed image data and emotion data to a server in the cloud via a transmission method. The input is the data generated in steps 2 and 3, and the output is the data transmitted to the server. The device uploads the data using a secure protocol.

[0171] Step 5:

[0172] The server analyzes received image data using image analysis tools to evaluate the freshness and nutritional value of food. The input is pre-processed image data, and the analysis results in the output of data regarding food freshness and nutritional value. Machine learning algorithms are applied during this process.

[0173] Step 6:

[0174] The server uses an evaluation information generation mechanism to devise appropriate consumption methods based on input emotion data and analyzed food evaluation data. As output, it generates personalized food suggestions tailored to the user's situation. For example, it might suggest, "If you want to relax, you should try a recipe using herbal tea."

[0175] Step 7:

[0176] The server returns the generated evaluation information and suggestions to the terminal via an information notification system. The input is the generated evaluation information, and the output is the transmission of data to the user terminal.

[0177] Step 8:

[0178] The device displays the received information to the user. The input is evaluation information sent from the server, and as a specific action, a message such as "Let's try these foods today to relax" is displayed. The display is adapted according to the user's emotional state, for example, by using gentle colors.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] In modern living environments, consumers face a variety of stressors, and appropriate food choices are required to address their emotional states. However, providing dietary suggestions tailored to individual emotional states is not easy, and there is a lack of systems that support consumers' mental and physical health while considering food quality and nutritional value.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for suggesting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, an emotion recognition means for analyzing the user's facial expressions and voice to identify their emotional state, and an individualized suggestion means for suggesting a consumption plan suitable for the user based on their emotional state and the quality of the object. This makes it possible to promote appropriate food consumption according to the user's emotional state and support the realization of a healthy diet.

[0184] An "image acquisition means" is a device that captures images of an object or user and acquires them as digital data.

[0185] "Preprocessing means" refers to a device or function that performs processing to remove unnecessary parts from an image and convert it into a format suitable for analysis.

[0186] "Transmission means" refers to a device or function for transmitting pre-processed image data to a remote processing device.

[0187] "Image analysis means" refers to a device or function that analyzes received image data and evaluates the freshness and quality of an object.

[0188] "Evaluation information generation means" refers to a device or function that generates information to suggest appropriate expiration dates and consumption methods based on the results of image analysis.

[0189] An "information notification means" is a device or interface for effectively conveying generated evaluation information to the user.

[0190] "Emotion recognition means" refers to a device or function that analyzes the user's facial expressions and voice to identify their current emotional state.

[0191] "Personalized suggestion means" refers to a device or function that suggests the optimal consumption plan or recipe to the user based on the user's emotional state and food quality information.

[0192] To implement this invention, the user first uses the terminal's camera to take an image of food. The terminal uses a pre-processing means to remove unnecessary parts from this image and convert it into a format suitable for analysis. Next, the pre-processed image data and the user's facial expressions and voice data are collected and transmitted to a remote server by a transmission means.

[0193] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Simultaneously, emotion recognition tools analyze the user's facial expressions and voice data to identify the user's current emotional state. This provides data such as whether the user is fatigued or stressed.

[0194] Based on this data, the evaluation information generation means generates evaluation information and creates information such as expiration dates and appropriate recipes. In particular, the personalized suggestion means, which responds to emotion recognition results, proposes a food consumption plan tailored to the user's current emotional state. For example, if relaxation is needed, it recommends easy-to-make herbal tea or salad.

[0195] The generated evaluation information and personalized suggestions are sent to the device via an information notification system and displayed to the user. The information is displayed on the screen with a tone and design that takes into account the user's emotional state, making it easy to read and understand.

[0196] As a concrete example, when a mother operates the device after returning home, the system senses her fatigue and suggests a simple pasta dish and a relaxing herbal tea. This system helps users to consume food that is appropriate for their emotional state at the time, supporting a health-conscious diet.

[0197] An example of a prompt for a generative AI model would be: "The analyzed emotional state is 'fatigue,' and you have tomatoes and cheese in the refrigerator. Please suggest a simple dish that will help you relax."

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The user takes a picture of food using the device's camera. The input obtained is image data of the food. This activates the image acquisition mechanism, which collects the image as a dataset.

[0201] Step 2:

[0202] The terminal uses preprocessing to remove unnecessary parts from image data and converts it into a format suitable for analysis. The input is raw image data of food, and the output is preprocessed image data with noise removed. Specifically, image filtering and resizing are performed.

[0203] Step 3:

[0204] The terminal transmits pre-processed image data, user facial expressions, and audio data to the server using a transmission method. The input consists of pre-processed images and user audio / video data, which are then transferred to the cloud server.

[0205] Step 4:

[0206] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Pre-processed image data is the input, and data evaluating freshness and nutritional value is the output. Here, data calculations such as image recognition and quality evaluation are performed.

[0207] Step 5:

[0208] The server uses emotion recognition to analyze the user's voice and video data and identify their emotional state. The input is data indicating the user's emotions, and the output is data representing the corresponding emotional state. Emotional states are detected through voice tone analysis and facial expression recognition.

[0209] Step 6:

[0210] The server uses an evaluation information generation mechanism to generate an appropriate consumption plan based on the acquired quality evaluation and emotional state. The input is freshness evaluation data and emotional state, and the output is the proposed consumption plan. Here, a generation AI model is used to create the prompt text.

[0211] Step 7:

[0212] The server transmits the generated consumption plan and evaluation information to the terminal via an information notification system and displays it to the user. The input is the generated consumption plan, and the output is a visual or auditory notification. Specific actions update the user interface and provide information to the user.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] To implement this invention, the user first takes a photograph of food using a smartphone or other device. A dedicated application is installed on this device, and the user activates the camera through the application and takes a photograph of the food. The captured image is converted into a format suitable for analysis by the device's pre-processing function. During this process, the background is removed and necessary parts are cropped.

[0230] Next, the terminal sends the pre-processed images to a server in the cloud. The server uses image analysis algorithms to analyze the received images and evaluate the freshness and nutritional value of the food. This analysis includes techniques for analyzing image characteristics such as hue and shape. Furthermore, the server generates evaluation information based on the analysis results. This information includes recommended expiration dates, cooking methods, and storage methods.

[0231] The generated evaluation information is sent to the user's device via the information notification function. The device then displays this information to the user through the app. Based on the displayed information, the user can plan to consume food appropriately.

[0232] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image and evaluates its current freshness. The evaluation information generated would include something like, "The apple is fresh and it is recommended to consume it within two days. Refrigeration is the best place to store it." Based on this information, the user can decide when to consume the apple and avoid waste.

[0233] This series of processes ensures that users can always consume fresh food as needed, thereby contributing to a reduction in food waste. Furthermore, the nutritional information provided based on evaluation data helps users make healthy food choices.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] The user launches their smartphone's camera app and takes a picture of the food they want to evaluate. The user is expected to adjust the lighting and angle appropriately during shooting to obtain an image best suited for analysis.

[0237] Step 2:

[0238] The device receives the captured image and applies a preprocessing algorithm. This includes removing unwanted backgrounds from the image and cropping it to include only the necessary parts. The image's brightness and contrast are also adjusted to prepare it for optimal analysis.

[0239] Step 3:

[0240] The terminal sends pre-processed image data to the cloud server using a communication protocol. This transmission is encrypted to ensure security.

[0241] Step 4:

[0242] The server runs an AI model to analyze the received images. This model analyzes the color, shape, and surface features of the food in the images to evaluate its freshness and nutritional value.

[0243] Step 5:

[0244] Based on the analysis results, the server generates evaluation information regarding expiration dates, optimal consumption methods, and storage methods. In doing so, it creates personalized recommendations that take into account accumulated data and the user's past history.

[0245] Step 6:

[0246] The server sends the generated evaluation information to the user's terminal. The transmitted data is formatted in an easy-to-read format, ensuring that the content is easy for the user to understand.

[0247] Step 7:

[0248] The device displays the received information to the user. Through notifications and in-app displays, users can check the status of food and recommendations, and make concrete consumption plans.

[0249] (Example 1)

[0250] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0251] Managing food freshness and reducing food waste are critical challenges facing modern consumers. In particular, food waste often arises from a lack of knowledge about expiration dates and proper storage methods, making consumers' lives inconvenient. Providing nutritional information to support healthy food choices is also crucial. Therefore, there is a need to provide systems that enable consumers to properly manage and effectively consume food.

[0252] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0253] In this invention, the server includes, as an image analysis means, a means for analyzing received images using a dedicated algorithm and evaluating the freshness and nutritional value of the object; as an evaluation information generation means, a means for presenting recommended expiration dates, storage methods, and cooking methods based on the analysis results; and as an information notification means, a means for displaying the generated evaluation information to the user via a terminal. This makes it possible for consumers to immediately obtain information to properly manage food and make healthy food choices.

[0254] "Image acquisition means" refers to devices or software functions for photographing food, which acquire image data captured by the user.

[0255] "Preprocessing means" refers to a function that removes the background and unnecessary parts from the acquired image and converts the image into a format suitable for analysis.

[0256] "Transmission means" refers to a communication function for securely transmitting pre-processed image data to a remote processing unit.

[0257] "Image analysis means" refers to a function that analyzes received images using specialized algorithms and machine learning models to evaluate the freshness and nutritional value of food.

[0258] "Evaluation information generation means" refers to a function that generates information such as recommended expiration dates, storage methods, and cooking methods for food products based on the results of image analysis.

[0259] "Information notification means" refers to a function that provides generated evaluation information to the user's terminal and displays it to the user.

[0260] The invention will now be described in terms of its embodiments. This invention is a system that operates primarily around a user's terminal and a server in the cloud.

[0261] First, the user uses a smartphone or other device to photograph the food. A dedicated application is installed on the device, and this application controls the camera function to acquire images of the food. While the standard smartphone camera function is used for image acquisition, additional software correction may be performed to further improve accuracy.

[0262] Next, the device preprocesses the acquired image immediately after capture. Here, image processing algorithms are used to remove unwanted background elements and extract only the food portion. Color correction and noise reduction are also performed, and the image is converted into a format suitable for analysis. This preprocessing may utilize known image processing algorithms or image processing libraries from specific manufacturers.

[0263] The pre-processed images are sent from the terminal to a server in the cloud. This transmission uses an internet connection and is encrypted to ensure security.

[0264] Images that reach the server are processed by a dedicated image analysis algorithm. The server analyzes the received image data to estimate the freshness and nutritional value of the food. This analysis uses a generative AI model, which automatically analyzes features such as the hue, shape, and texture of the food.

[0265] Based on the analysis results, the server generates evaluation information. This information includes the recommended expiration date, storage method, and cooking method for the food. This evaluation information is transmitted to the terminal in real time for the user to receive.

[0266] Finally, the device displays the received evaluation information to the user through the application. Based on this information, the user can make plans, for example, on when to consume food and how to store it.

[0267] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image, assesses its freshness, and provides information such as, "It is recommended to consume the apple within two days and store it in the refrigerator." In this way, it becomes easier to plan food consumption.

[0268] An example of a prompt to input into a generative AI model is, "Analyze this picture of an apple and tell me the recommended expiration date and storage method."

[0269] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0270] Step 1:

[0271] The user launches the application on their device and uses the camera function to take a picture of food. The input at this time is the image within the frame of the food that the user sees through the camera. Specifically, the user saves the image data to the device's memory by pressing the "Capture" button in the app.

[0272] Step 2:

[0273] The device preprocesses the acquired image. The input is the image taken in step 1, and the output is a cleared image of the food with the background removed. Specifically, the device uses an image processing algorithm to perform noise reduction, background removal, and color information correction.

[0274] Step 3:

[0275] The terminal sends the pre-processed image to the server. The input to this transmission is the pre-processed image that will be the output of step 2, and the output is the image data that the cloud server receives. During transmission, the terminal encrypts the data to ensure secure communication.

[0276] Step 4:

[0277] The server analyzes the received images. The input is the pre-processed image data sent in step 3. The server uses a dedicated image analysis algorithm, particularly a generative AI model, to analyze the hue and shape of the food and estimate its freshness and nutritional value. The output is evaluation data of the analysis results.

[0278] Step 5:

[0279] The server generates evaluation information based on the analysis results. The input is the analysis results, which are the output of step 4. Specifically, the server generates text information such as recommended expiration dates, storage methods, and cooking methods, and formats it in a way that is easy for the user to understand. The output is text data as evaluation information.

[0280] Step 6:

[0281] The server sends evaluation information to the terminal. The input for this transmission is the evaluation information generated in step 5. The output is data formatted in a format that can be displayed on the terminal. The server transmits this to the terminal via the internet.

[0282] Step 7:

[0283] The terminal displays the received evaluation information to the user via the application. The input is the evaluation information sent from the server in step 6. Specifically, the terminal utilizes its notification function to display the expiration date and storage method on the user interface. The output is the presentation of information to the user in a visually understandable format.

[0284] (Application Example 1)

[0285] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0286] Food management at home is time-consuming. As a result, the freshness of food may decrease, the nutritional value may be lost, and there may even be wasteful disposal. In addition, while there is a need for means to reduce the burden of daily tasks within the home and perform them efficiently, there is a problem that it is difficult to perform food management autonomously.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0288] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from the image and converting it into a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and performing a quality evaluation of the object, an evaluation information generation means for presenting an appropriate expiration date and consumption method based on the quality evaluation result, an information notification means for displaying the generated evaluation information to the user, a photographing object identification means in the physical environment for autonomously identifying the photographing object in the physical storage device, and a means for periodically scanning the object in the physical storage device based on a defined criterion. As a result, food management within the home is automated, enabling efficient and waste-free consumption.

[0289] The "image acquisition means" refers to the function of collecting visual data of an object using a photographing device.

[0290] The "preprocessing means" is a function that extracts necessary information from the acquired image and performs a process of converting it into a format suitable for analysis.

[0291] "Transmission means" refers to the function of transferring processed data to a remote processing unit via a communication device.

[0292] "Image analysis means" refers to a function that analyzes received visual data and evaluates the characteristics and state of an object.

[0293] "Evaluation information generation means" refers to a function that generates useful information about an object based on the analysis results.

[0294] "Information notification means" refers to a function for transmitting and displaying generated evaluation information to users.

[0295] "Means for identifying objects to be photographed within a physical environment" refers to a system that has the function of identifying and accurately recognizing objects within a storage device.

[0296] "Means for periodically scanning objects within a physical storage device based on defined criteria" refers to a function that continuously monitors objects within a storage device and collects status information according to set conditions.

[0297] The system implementing this invention is designed to automate food management within the home. The system includes a series of processes for image acquisition, preprocessing, image analysis, evaluation information generation, and information notification.

[0298] The server uses Python and TensorFlow to execute the analysis algorithm. A home robot has a built-in camera and periodically photographs food as an image acquisition tool. This robot is based on iRobot and other common home robots and has the ability to autonomously identify food in the refrigerator. The captured images are preprocessed to remove the background and convert them into a format suitable for analysis. This processing includes image cropping and noise reduction.

[0299] Next, the terminal sends the pre-processed image to a cloud server. The server receives the transmitted data and uses image analysis tools to evaluate the condition of the food. Here, the hue, shape, and other visual features of the image are analyzed to evaluate the freshness and nutritional value of the food.

[0300] The server generates evaluation information based on the analysis results. The evaluation information generation means generates information on appropriate expiration dates and storage methods, and sends it to the terminal via the cloud. The terminal displays the received notification to the user in an easy-to-use format and provides recommendations regarding the condition of the food.

[0301] For example, if a user has tomatoes stored in their refrigerator, the robot can take a picture of them, and the server will analyze the image to generate information such as, "The tomatoes are ripe and should be consumed within three days." This promotes the proper consumption of food and reduces waste.

[0302] An example of a prompt for a generative AI model can be set as follows: "Analyze images of food in a refrigerator and recommend freshness assessment and expiration date." By entering this prompt, the system can perform the correct analysis and provide information.

[0303] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0304] Step 1:

[0305] The device uses a home robot's camera to photograph the food inside the refrigerator. The input is a physical image of the food, and the output is digital image data. This data is needed for the following processing.

[0306] Step 2:

[0307] The terminal preprocesses the captured image. The input is the digital image data obtained in step 1. In preprocessing, the background is removed and the food part is cropped to reduce noise and convert it into a form suitable for analysis. The output is the preprocessed image.

[0308] Step 3:

[0309] The terminal sends the preprocessed image to the cloud server. The input is the preprocessed image obtained in step 2, and the output is the status information uploaded to the cloud. This status information is used in the next analysis step.

[0310] Step 4:

[0311] The server analyzes the image data received on the cloud. The input is the preprocessed image sent in step 3. The server uses image analysis means to analyze visual features such as the hue and shape of the image and evaluate the freshness and nutritional value of the food. The output is the evaluation result.

[0312] Step 5:

[0313] The server generates evaluation information based on the evaluation result. The input is the evaluation result obtained in step 4. The server generates and outputs information regarding an appropriate expiration date and storage method. This information is provided to the user.

[0314] Step 6:

[0315] The server sends the generated evaluation information to the terminal. The input is the evaluation information generated in step 5. The output is the evaluation information sent to the terminal.

[0316] Step 7:

[0317] The terminal notifies the user of the received evaluation information. The input is the evaluation information sent in step 6. Based on the evaluation information, the user can properly manage food and create a consumption plan. The output is specific instructions displayed on the terminal's display.

[0318] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0319] This invention evaluates the freshness and nutritional value of food and, using a system equipped with an emotion engine, provides personalized suggestions tailored to the user's emotional state. The system begins with the user taking a photograph of the food using a smartphone or other device, and then recognizes the user's emotions through the emotion engine.

[0320] Specifically, the user takes a picture of food using the device's camera app. During this process, the emotion engine uses the device's front camera and sensors to analyze the user's facial expressions and tone of voice to recognize their emotions. For example, it can obtain information such as whether the user is tired or stressed. The device preprocesses this information and then sends both the image data and emotion data to a cloud server.

[0321] The server analyzes received images and evaluates the freshness and nutritional value of the food. Based on emotional data, it also adjusts the content of the information notifications; for example, suggesting new recipes when the user is in a positive state and recommending simpler cooking methods when they are in a negative state. The evaluation information generation function combines the user's current emotional data with their past usage history to suggest the optimal consumption plan. It can even provide specific suggestions tailored to emotional states, such as "use ingredients with relaxing effects when you're feeling down."

[0322] The server sends the generated evaluation information and suggestions to the terminal, which then displays them to the user. The display is presented in an easy-to-read format that takes the user's emotional state into consideration, encouraging consumer behavior without requiring any user intervention. For example, if the user is feeling stressed, the notification is displayed in a softer tone, and the suggestions are concise to reduce the burden.

[0323] This system allows users to make optimal food choices based on their emotional state at the time, supporting a health-conscious diet. In addition to reducing food waste, using this system can also contribute to users' emotional management.

[0324] The following describes the processing flow.

[0325] Step 1:

[0326] The user activates their smartphone's camera and takes a picture of the food they want to evaluate. Simultaneously with the photo, the device's front camera and microphone capture the user's facial expressions and voice. This allows the user's emotional state to be recorded at the same time.

[0327] Step 2:

[0328] The device preprocesses the captured images, including cropping food portions and correcting image color. Simultaneously, the emotion engine analyzes the acquired audio and facial expression data, and sets an emotion category (e.g., stress, exhilaration, calmness) in real time.

[0329] Step 3:

[0330] The device pairs pre-processed image data with emotion data and sends it to the cloud server. The transmission is encrypted based on a security protocol.

[0331] Step 4:

[0332] After receiving image data, the server applies an AI image recognition algorithm to analyze the type of food, its freshness, and its nutritional value. It also receives emotional data simultaneously and compares it with a database to enable suggestions that take the user's emotional state into account.

[0333] Step 5:

[0334] Based on the analysis results, the server generates suggestions for expiration dates and cooking methods. Furthermore, it uses emotional data to customize suggestions according to the user's emotional state, for example, "If you're feeling stressed, make a simple and nutritious smoothie."

[0335] Step 6:

[0336] The server sends the generated evaluation information and suggestions to the terminal. The information sent is organized and formatted in an easy-to-understand format.

[0337] Step 7:

[0338] The device notifies the user of the received information and displays it within the app. The displayed content is adjusted according to the user's emotional state and is designed to be received appropriately. Based on this, the user can choose the optimal way to consume food.

[0339] (Example 2)

[0340] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0341] Conventional food quality evaluation systems only assess freshness and nutritional value, failing to consider the user's emotional state when making recommendations. Therefore, it was difficult to provide personalized recommendations based on the user's emotional state, making it challenging to improve user satisfaction and convenience. Furthermore, there is a need to optimize food consumption plans in conjunction with emotional states to support a healthy and mentally fulfilling diet.

[0342] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0343] In this invention, the server includes an image analysis means for analyzing received images and evaluating the quality of the object, an emotion analysis means for analyzing the user's emotional state and considering the analysis results, and an evaluation information generation means for presenting an appropriate consumption plan and consumption method based on the quality evaluation results and emotion analysis results. This enables personalized food suggestions that take into account the user's emotional state, making it possible to realize healthier and more emotionally considerate food consumption.

[0344] "Image acquisition means" refers to a device or software that acquires an image of an object using a device.

[0345] "Preprocessing means" refers to a device or software that removes unnecessary parts from acquired image data and converts it into a format suitable for analysis.

[0346] "Transmission means" refers to a device or protocol that transmits pre-processed data to a processing device located remotely.

[0347] "Image analysis means" refers to a device or algorithm that analyzes received image data and evaluates the quality and condition of the object from it.

[0348] "Emotional analysis means" refers to a device or program that analyzes a user's facial expressions and voice data to determine their emotional state.

[0349] "Evaluation information generation means" refers to a device or system for formulating and proposing appropriate consumption plans and methods based on quality evaluation results and sentiment analysis results.

[0350] "Information notification means" refers to a device or interface that notifies the user of generated evaluation information visually or audibly.

[0351] This invention provides a system that allows users to evaluate the quality of food and receive personalized consumption suggestions based on their emotional state. The system primarily consists of terminals and servers, each playing a specific role.

[0352] The user first takes a picture of the food using the device's image acquisition mechanism. The device is equipped with a camera application, which allows the user to easily record the food. The captured image is pre-processed to remove unnecessary parts and converted into a format suitable for analysis. Furthermore, the device uses emotion analysis to analyze the user's facial expressions and voice tone through the user's front camera and microphone, and acquires the user's emotion data.

[0353] Subsequently, this data is transmitted to a server via a transmission device. The server has an image analysis device that analyzes the received image data to evaluate the freshness and nutritional value of the food. The server also uses an emotion analysis device to generate evaluation information that takes into account the user's emotional state.

[0354] Based on the user's quality evaluation results and sentiment analysis results, the evaluation information generation system suggests optimal consumption plans and methods. For example, if the user is feeling stressed, the system suggests recipes with short cooking times and ingredients that have a relaxing effect. The server then uses an information notification system to transmit the generated evaluation information to the terminal.

[0355] The device visually displays the received information to the user. The display format is designed to be easy to see and understand, based on the user's emotional state. For example, if the user is feeling down, notifications might be displayed in soft colors and with concise messages.

[0356] Furthermore, an example of a prompt using a generative AI model is, "I'd like to have a relaxing meal today. Please recommend some ingredients." Based on this prompt, the AI ​​will provide appropriate food suggestions.

[0357] These features of the system allow users to make food choices that suit their emotions and health, supporting a healthy and comfortable lifestyle.

[0358] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0359] Step 1:

[0360] The user launches the camera app on their device and takes a picture of the food they intend to consume. The image of the food is captured by the device's camera as input. Specifically, the user holds the apple up to the camera and presses the shutter button. This image data is then sent to the next processing step.

[0361] Step 2:

[0362] The terminal processes the captured image data using a preprocessing mechanism. Specifically, it removes unwanted backgrounds from the image data and converts it to a resolution suitable for subsequent analysis. The input for this process is the food image acquired in the previous step, and the output is image data in a format suitable for analysis.

[0363] Step 3:

[0364] The device analyzes the user's emotional state using emotion analysis techniques. It uses a front camera and microphone to capture the user's facial expressions and voice as input, and generates emotion data as a result of the analysis. For example, it might capture the user taking a deep breath and output emotion data such as "relaxed."

[0365] Step 4:

[0366] The device transmits pre-processed image data and emotion data to a server in the cloud via a transmission method. The input is the data generated in steps 2 and 3, and the output is the data transmitted to the server. The device uploads the data using a secure protocol.

[0367] Step 5:

[0368] The server analyzes received image data using image analysis tools to evaluate the freshness and nutritional value of food. The input is pre-processed image data, and the analysis results in the output of data regarding food freshness and nutritional value. Machine learning algorithms are applied during this process.

[0369] Step 6:

[0370] The server uses an evaluation information generation mechanism to devise appropriate consumption methods based on input emotion data and analyzed food evaluation data. As output, it generates personalized food suggestions tailored to the user's situation. For example, it might suggest, "If you want to relax, you should try a recipe using herbal tea."

[0371] Step 7:

[0372] The server returns the generated evaluation information and suggestions to the terminal via an information notification system. The input is the generated evaluation information, and the output is the transmission of data to the user terminal.

[0373] Step 8:

[0374] The device displays the received information to the user. The input is evaluation information sent from the server, and as a specific action, a message such as "Let's try these foods today to relax" is displayed. The display is adapted according to the user's emotional state, for example, by using gentle colors.

[0375] (Application Example 2)

[0376] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0377] In modern living environments, consumers face a variety of stressors, and appropriate food choices are required to address their emotional states. However, providing dietary suggestions tailored to individual emotional states is not easy, and there is a lack of systems that support consumers' mental and physical health while considering food quality and nutritional value.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0379] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for suggesting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, an emotion recognition means for analyzing the user's facial expressions and voice to identify their emotional state, and an individualized suggestion means for suggesting a consumption plan suitable for the user based on their emotional state and the quality of the object. This makes it possible to promote appropriate food consumption according to the user's emotional state and support the realization of a healthy diet.

[0380] An "image acquisition means" is a device that captures images of an object or user and acquires them as digital data.

[0381] "Preprocessing means" refers to a device or function that performs processing to remove unnecessary parts from an image and convert it into a format suitable for analysis.

[0382] "Transmission means" refers to a device or function for transmitting pre-processed image data to a remote processing device.

[0383] "Image analysis means" refers to a device or function that analyzes received image data and evaluates the freshness and quality of an object.

[0384] "Evaluation information generation means" refers to a device or function that generates information to suggest appropriate expiration dates and consumption methods based on the results of image analysis.

[0385] An "information notification means" is a device or interface for effectively conveying generated evaluation information to the user.

[0386] "Emotion recognition means" refers to a device or function that analyzes the user's facial expressions and voice to identify their current emotional state.

[0387] "Personalized suggestion means" refers to a device or function that suggests the optimal consumption plan or recipe to the user based on the user's emotional state and food quality information.

[0388] To implement this invention, the user first uses the terminal's camera to take an image of food. The terminal uses a pre-processing means to remove unnecessary parts from this image and convert it into a format suitable for analysis. Next, the pre-processed image data and the user's facial expressions and voice data are collected and transmitted to a remote server by a transmission means.

[0389] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Simultaneously, emotion recognition tools analyze the user's facial expressions and voice data to identify the user's current emotional state. This provides data such as whether the user is fatigued or stressed.

[0390] Based on this data, the evaluation information generation means generates evaluation information and creates information such as expiration dates and appropriate recipes. In particular, the personalized suggestion means, which responds to emotion recognition results, proposes a food consumption plan tailored to the user's current emotional state. For example, if relaxation is needed, it recommends easy-to-make herbal tea or salad.

[0391] The generated evaluation information and personalized suggestions are sent to the device via an information notification system and displayed to the user. The information is displayed on the screen with a tone and design that takes into account the user's emotional state, making it easy to read and understand.

[0392] As a concrete example, when a mother operates the device after returning home, the system senses her fatigue and suggests a simple pasta dish and a relaxing herbal tea. This system helps users to consume food that is appropriate for their emotional state at the time, supporting a health-conscious diet.

[0393] An example of a prompt for a generative AI model would be: "The analyzed emotional state is 'fatigue,' and you have tomatoes and cheese in the refrigerator. Please suggest a simple dish that will help you relax."

[0394] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0395] Step 1:

[0396] The user takes a picture of food using the device's camera. The input obtained is image data of the food. This activates the image acquisition mechanism, which collects the image as a dataset.

[0397] Step 2:

[0398] The terminal uses preprocessing to remove unnecessary parts from image data and converts it into a format suitable for analysis. The input is raw image data of food, and the output is preprocessed image data with noise removed. Specifically, image filtering and resizing are performed.

[0399] Step 3:

[0400] The terminal transmits pre-processed image data, user facial expressions, and audio data to the server using a transmission method. The input consists of pre-processed images and user audio / video data, which are then transferred to the cloud server.

[0401] Step 4:

[0402] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Pre-processed image data is the input, and data evaluating freshness and nutritional value is the output. Here, data calculations such as image recognition and quality evaluation are performed.

[0403] Step 5:

[0404] The server uses emotion recognition to analyze the user's voice and video data and identify their emotional state. The input is data indicating the user's emotions, and the output is data representing the corresponding emotional state. Emotional states are detected through voice tone analysis and facial expression recognition.

[0405] Step 6:

[0406] The server uses an evaluation information generation mechanism to generate an appropriate consumption plan based on the acquired quality evaluation and emotional state. The input is freshness evaluation data and emotional state, and the output is the proposed consumption plan. Here, a generation AI model is used to create the prompt text.

[0407] Step 7:

[0408] The server transmits the generated consumption plan and evaluation information to the terminal via an information notification system and displays it to the user. The input is the generated consumption plan, and the output is a visual or auditory notification. Specific actions update the user interface and provide information to the user.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0410] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0412] [Third Embodiment]

[0413] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0414] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0415] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0417] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0419] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0420] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0421] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0424] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0425] To implement this invention, the user first takes a photograph of food using a smartphone or other device. A dedicated application is installed on this device, and the user activates the camera through the application and takes a photograph of the food. The captured image is converted into a format suitable for analysis by the device's pre-processing function. During this process, the background is removed and necessary parts are cropped.

[0426] Next, the terminal sends the pre-processed images to a server in the cloud. The server uses image analysis algorithms to analyze the received images and evaluate the freshness and nutritional value of the food. This analysis includes techniques for analyzing image characteristics such as hue and shape. Furthermore, the server generates evaluation information based on the analysis results. This information includes recommended expiration dates, cooking methods, and storage methods.

[0427] The generated evaluation information is sent to the user's device via the information notification function. The device then displays this information to the user through the app. Based on the displayed information, the user can plan to consume food appropriately.

[0428] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image and evaluates its current freshness. The evaluation information generated would include something like, "The apple is fresh and it is recommended to consume it within two days. Refrigeration is the best place to store it." Based on this information, the user can decide when to consume the apple and avoid waste.

[0429] This series of processes ensures that users can always consume fresh food as needed, thereby contributing to a reduction in food waste. Furthermore, the nutritional information provided based on evaluation data helps users make healthy food choices.

[0430] The following describes the processing flow.

[0431] Step 1:

[0432] The user launches their smartphone's camera app and takes a picture of the food they want to evaluate. The user is expected to adjust the lighting and angle appropriately during shooting to obtain an image best suited for analysis.

[0433] Step 2:

[0434] The device receives the captured image and applies a preprocessing algorithm. This includes removing unwanted backgrounds from the image and cropping it to include only the necessary parts. The image's brightness and contrast are also adjusted to prepare it for optimal analysis.

[0435] Step 3:

[0436] The terminal sends pre-processed image data to the cloud server using a communication protocol. This transmission is encrypted to ensure security.

[0437] Step 4:

[0438] The server runs an AI model to analyze the received images. This model analyzes the color, shape, and surface features of the food in the images to evaluate its freshness and nutritional value.

[0439] Step 5:

[0440] Based on the analysis results, the server generates evaluation information regarding expiration dates, optimal consumption methods, and storage methods. In doing so, it creates personalized recommendations that take into account accumulated data and the user's past history.

[0441] Step 6:

[0442] The server sends the generated evaluation information to the user's terminal. The transmitted data is formatted in an easy-to-read format, ensuring that the content is easy for the user to understand.

[0443] Step 7:

[0444] The device displays the received information to the user. Through notifications and in-app displays, users can check the status of food and recommendations, and make concrete consumption plans.

[0445] (Example 1)

[0446] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0447] Managing food freshness and reducing food waste are critical challenges facing modern consumers. In particular, food waste often arises from a lack of knowledge about expiration dates and proper storage methods, making consumers' lives inconvenient. Providing nutritional information to support healthy food choices is also crucial. Therefore, there is a need to provide systems that enable consumers to properly manage and effectively consume food.

[0448] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0449] In this invention, the server includes, as an image analysis means, a means for analyzing received images using a dedicated algorithm and evaluating the freshness and nutritional value of the object; as an evaluation information generation means, a means for presenting recommended expiration dates, storage methods, and cooking methods based on the analysis results; and as an information notification means, a means for displaying the generated evaluation information to the user via a terminal. This makes it possible for consumers to immediately obtain information to properly manage food and make healthy food choices.

[0450] "Image acquisition means" refers to devices or software functions for photographing food, which acquire image data captured by the user.

[0451] "Preprocessing means" refers to a function that removes the background and unnecessary parts from the acquired image and converts the image into a format suitable for analysis.

[0452] "Transmission means" refers to a communication function for securely transmitting pre-processed image data to a remote processing unit.

[0453] "Image analysis means" refers to a function that analyzes received images using specialized algorithms and machine learning models to evaluate the freshness and nutritional value of food.

[0454] "Evaluation information generation means" refers to a function that generates information such as recommended expiration dates, storage methods, and cooking methods for food products based on the results of image analysis.

[0455] "Information notification means" refers to a function that provides generated evaluation information to the user's terminal and displays it to the user.

[0456] The invention will now be described in terms of its embodiments. This invention is a system that operates primarily around a user's terminal and a server in the cloud.

[0457] First, the user uses a smartphone or other device to photograph the food. A dedicated application is installed on the device, and this application controls the camera function to acquire images of the food. While the standard smartphone camera function is used for image acquisition, additional software correction may be performed to further improve accuracy.

[0458] Next, the device preprocesses the acquired image immediately after capture. Here, image processing algorithms are used to remove unwanted background elements and extract only the food portion. Color correction and noise reduction are also performed, and the image is converted into a format suitable for analysis. This preprocessing may utilize known image processing algorithms or image processing libraries from specific manufacturers.

[0459] The pre-processed images are sent from the terminal to a server in the cloud. This transmission uses an internet connection and is encrypted to ensure security.

[0460] Images that reach the server are processed by a dedicated image analysis algorithm. The server analyzes the received image data to estimate the freshness and nutritional value of the food. This analysis uses a generative AI model, which automatically analyzes features such as the hue, shape, and texture of the food.

[0461] Based on the analysis results, the server generates evaluation information. This information includes the recommended expiration date, storage method, and cooking method for the food. This evaluation information is transmitted to the terminal in real time for the user to receive.

[0462] Finally, the device displays the received evaluation information to the user through the application. Based on this information, the user can make plans, for example, on when to consume food and how to store it.

[0463] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image, assesses its freshness, and provides information such as, "It is recommended to consume the apple within two days and store it in the refrigerator." In this way, it becomes easier to plan food consumption.

[0464] An example of a prompt to input into a generative AI model is, "Analyze this picture of an apple and tell me the recommended expiration date and storage method."

[0465] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0466] Step 1:

[0467] The user launches the application on their device and uses the camera function to take a picture of food. The input at this time is the image within the frame of the food that the user sees through the camera. Specifically, the user saves the image data to the device's memory by pressing the "Capture" button in the app.

[0468] Step 2:

[0469] The device preprocesses the acquired image. The input is the image taken in step 1, and the output is a cleared image of the food with the background removed. Specifically, the device uses an image processing algorithm to perform noise reduction, background removal, and color information correction.

[0470] Step 3:

[0471] The terminal sends the pre-processed image to the server. The input to this transmission is the pre-processed image that will be the output of step 2, and the output is the image data that the cloud server receives. During transmission, the terminal encrypts the data to ensure secure communication.

[0472] Step 4:

[0473] The server analyzes the received images. The input is the pre-processed image data sent in step 3. The server uses a dedicated image analysis algorithm, particularly a generative AI model, to analyze the hue and shape of the food and estimate its freshness and nutritional value. The output is evaluation data of the analysis results.

[0474] Step 5:

[0475] The server generates evaluation information based on the analysis results. The input is the analysis results, which are the output of step 4. Specifically, the server generates text information such as recommended expiration dates, storage methods, and cooking methods, and formats it in a way that is easy for the user to understand. The output is text data as evaluation information.

[0476] Step 6:

[0477] The server sends evaluation information to the terminal. The input for this transmission is the evaluation information generated in step 5. The output is data formatted in a format that can be displayed on the terminal. The server transmits this to the terminal via the internet.

[0478] Step 7:

[0479] The terminal displays the received evaluation information to the user via the application. The input is the evaluation information sent from the server in step 6. Specifically, the terminal utilizes its notification function to display the expiration date and storage method on the user interface. The output is the presentation of information to the user in a visually understandable format.

[0480] (Application Example 1)

[0481] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0482] Managing food at home is time-consuming, which can lead to a decline in freshness, loss of nutritional value, and unnecessary waste. Furthermore, while there is a need to reduce the burden of daily household tasks and perform them more efficiently, there is a challenge in autonomously managing food.

[0483] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0484] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for presenting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, a means for identifying the object to be photographed in the physical environment, an autonomous means for identifying the object to be photographed in the physical storage device, and a means for periodically scanning the object in the physical storage device based on defined criteria. This automates food management in the home, enabling efficient and waste-free consumption.

[0485] "Image acquisition means" refers to the function of collecting visual data of an object using a photographic device.

[0486] "Preprocessing means" refers to a function that extracts necessary information from acquired images and converts it into a format suitable for analysis.

[0487] "Transmission means" refers to the function of transferring processed data to a remote processing unit via a communication device.

[0488] "Image analysis means" refers to a function that analyzes received visual data and evaluates the characteristics and state of an object.

[0489] "Evaluation information generation means" refers to a function that generates useful information about an object based on the analysis results.

[0490] "Information notification means" refers to a function for transmitting and displaying generated evaluation information to users.

[0491] "Means for identifying objects to be photographed within a physical environment" refers to a system that has the function of identifying and accurately recognizing objects within a storage device.

[0492] "Means for periodically scanning objects within a physical storage device based on defined criteria" refers to a function that continuously monitors objects within a storage device and collects status information according to set conditions.

[0493] The system implementing this invention is designed to automate food management within the home. The system includes a series of processes for image acquisition, preprocessing, image analysis, evaluation information generation, and information notification.

[0494] The server uses Python and TensorFlow to execute the analysis algorithm. A home robot has a built-in camera and periodically photographs food as an image acquisition tool. This robot is based on iRobot and other common home robots and has the ability to autonomously identify food in the refrigerator. The captured images are preprocessed to remove the background and convert them into a format suitable for analysis. This processing includes image cropping and noise reduction.

[0495] Next, the terminal sends the pre-processed image to a cloud server. The server receives the transmitted data and uses image analysis tools to evaluate the condition of the food. Here, the hue, shape, and other visual features of the image are analyzed to evaluate the freshness and nutritional value of the food.

[0496] The server generates evaluation information based on the analysis results. The evaluation information generation means generates information on appropriate expiration dates and storage methods, and sends it to the terminal via the cloud. The terminal displays the received notification to the user in an easy-to-use format and provides recommendations regarding the condition of the food.

[0497] For example, if a user has tomatoes stored in their refrigerator, the robot can take a picture of them, and the server will analyze the image to generate information such as, "The tomatoes are ripe and should be consumed within three days." This promotes the proper consumption of food and reduces waste.

[0498] An example of a prompt for a generative AI model can be set as follows: "Analyze images of food in a refrigerator and recommend freshness assessment and expiration date." By entering this prompt, the system can perform the correct analysis and provide information.

[0499] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0500] Step 1:

[0501] The device uses a home robot's camera to photograph the food inside the refrigerator. The input is a physical image of the food, and the output is digital image data. This data is needed for the following processing.

[0502] Step 2:

[0503] The terminal preprocesses the captured image. The input is the digital image data obtained in step 1. Preprocessing reduces noise and converts the image into a format suitable for analysis by removing the background and cropping the food portion. The output is the preprocessed image.

[0504] Step 3:

[0505] The terminal sends the pre-processed image to the cloud server. The input is the pre-processed image obtained in step 2, and the output is the state information uploaded to the cloud. This state information is used in the next analysis step.

[0506] Step 4:

[0507] The server analyzes the image data received on the cloud. The input is the pre-processed image sent in step 3. The server uses image analysis tools to analyze the visual features of the image, such as hue and shape, and evaluates the freshness and nutritional value of the food. The output is the evaluation result.

[0508] Step 5:

[0509] The server generates evaluation information based on the evaluation results. The input is the evaluation results obtained in step 4. The server generates and outputs information regarding appropriate expiration dates and storage methods. This information is provided to the user.

[0510] Step 6:

[0511] The server sends the generated evaluation information to the terminal. The input is the evaluation information generated in step 5. The output is the evaluation information sent to the terminal.

[0512] Step 7:

[0513] The terminal notifies the user of the received evaluation information. The input is the evaluation information sent in step 6. Based on the evaluation information, the user can properly manage food and create a consumption plan. The output is specific instructions displayed on the terminal's display.

[0514] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0515] This invention evaluates the freshness and nutritional value of food and, using a system equipped with an emotion engine, provides personalized suggestions tailored to the user's emotional state. The system begins with the user taking a photograph of the food using a smartphone or other device, and then recognizes the user's emotions through the emotion engine.

[0516] Specifically, the user takes a picture of food using the device's camera app. During this process, the emotion engine uses the device's front camera and sensors to analyze the user's facial expressions and tone of voice to recognize their emotions. For example, it can obtain information such as whether the user is tired or stressed. The device preprocesses this information and then sends both the image data and emotion data to a cloud server.

[0517] The server analyzes received images and evaluates the freshness and nutritional value of the food. Based on emotional data, it also adjusts the content of the information notifications; for example, suggesting new recipes when the user is in a positive state and recommending simpler cooking methods when they are in a negative state. The evaluation information generation function combines the user's current emotional data with their past usage history to suggest the optimal consumption plan. It can even provide specific suggestions tailored to emotional states, such as "use ingredients with relaxing effects when you're feeling down."

[0518] The server sends the generated evaluation information and suggestions to the terminal, which then displays them to the user. The display is presented in an easy-to-read format that takes the user's emotional state into consideration, encouraging consumer behavior without requiring any user intervention. For example, if the user is feeling stressed, the notification is displayed in a softer tone, and the suggestions are concise to reduce the burden.

[0519] This system allows users to make optimal food choices based on their emotional state at the time, supporting a health-conscious diet. In addition to reducing food waste, using this system can also contribute to users' emotional management.

[0520] The following describes the processing flow.

[0521] Step 1:

[0522] The user activates their smartphone's camera and takes a picture of the food they want to evaluate. Simultaneously with the photo, the device's front camera and microphone capture the user's facial expressions and voice. This allows the user's emotional state to be recorded at the same time.

[0523] Step 2:

[0524] The device preprocesses the captured images, including cropping food portions and correcting image color. Simultaneously, the emotion engine analyzes the acquired audio and facial expression data, and sets an emotion category (e.g., stress, exhilaration, calmness) in real time.

[0525] Step 3:

[0526] The device pairs pre-processed image data with emotion data and sends it to the cloud server. The transmission is encrypted based on a security protocol.

[0527] Step 4:

[0528] After receiving image data, the server applies an AI image recognition algorithm to analyze the type of food, its freshness, and its nutritional value. It also receives emotional data simultaneously and compares it with a database to enable suggestions that take the user's emotional state into account.

[0529] Step 5:

[0530] Based on the analysis results, the server generates suggestions for expiration dates and cooking methods. Furthermore, it uses emotional data to customize suggestions according to the user's emotional state, for example, "If you're feeling stressed, make a simple and nutritious smoothie."

[0531] Step 6:

[0532] The server sends the generated evaluation information and suggestions to the terminal. The information sent is organized and formatted in an easy-to-understand format.

[0533] Step 7:

[0534] The device notifies the user of the received information and displays it within the app. The displayed content is adjusted according to the user's emotional state and is designed to be received appropriately. Based on this, the user can choose the optimal way to consume food.

[0535] (Example 2)

[0536] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0537] Conventional food quality evaluation systems only assess freshness and nutritional value, failing to consider the user's emotional state when making recommendations. Therefore, it was difficult to provide personalized recommendations based on the user's emotional state, making it challenging to improve user satisfaction and convenience. Furthermore, there is a need to optimize food consumption plans in conjunction with emotional states to support a healthy and mentally fulfilling diet.

[0538] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0539] In this invention, the server includes an image analysis means for analyzing received images and evaluating the quality of the object, an emotion analysis means for analyzing the user's emotional state and considering the analysis results, and an evaluation information generation means for presenting an appropriate consumption plan and consumption method based on the quality evaluation results and emotion analysis results. This enables personalized food suggestions that take into account the user's emotional state, making it possible to realize healthier and more emotionally considerate food consumption.

[0540] "Image acquisition means" refers to a device or software that acquires an image of an object using a device.

[0541] "Preprocessing means" refers to a device or software that removes unnecessary parts from acquired image data and converts it into a format suitable for analysis.

[0542] "Transmission means" refers to a device or protocol that transmits pre-processed data to a processing device located remotely.

[0543] "Image analysis means" refers to a device or algorithm that analyzes received image data and evaluates the quality and condition of the object from it.

[0544] "Emotional analysis means" refers to a device or program that analyzes a user's facial expressions and voice data to determine their emotional state.

[0545] "Evaluation information generation means" refers to a device or system for formulating and proposing appropriate consumption plans and methods based on quality evaluation results and sentiment analysis results.

[0546] "Information notification means" refers to a device or interface that notifies the user of generated evaluation information visually or audibly.

[0547] This invention provides a system that allows users to evaluate the quality of food and receive personalized consumption suggestions based on their emotional state. The system primarily consists of terminals and servers, each playing a specific role.

[0548] The user first takes a picture of the food using the device's image acquisition mechanism. The device is equipped with a camera application, which allows the user to easily record the food. The captured image is pre-processed to remove unnecessary parts and converted into a format suitable for analysis. Furthermore, the device uses emotion analysis to analyze the user's facial expressions and voice tone through the user's front camera and microphone, and acquires the user's emotion data.

[0549] Subsequently, this data is transmitted to a server via a transmission device. The server has an image analysis device that analyzes the received image data to evaluate the freshness and nutritional value of the food. The server also uses an emotion analysis device to generate evaluation information that takes into account the user's emotional state.

[0550] Based on the user's quality evaluation results and sentiment analysis results, the evaluation information generation system suggests optimal consumption plans and methods. For example, if the user is feeling stressed, the system suggests recipes with short cooking times and ingredients that have a relaxing effect. The server then uses an information notification system to transmit the generated evaluation information to the terminal.

[0551] The device visually displays the received information to the user. The display format is designed to be easy to see and understand, based on the user's emotional state. For example, if the user is feeling down, notifications might be displayed in soft colors and with concise messages.

[0552] Furthermore, an example of a prompt using a generative AI model is, "I'd like to have a relaxing meal today. Please recommend some ingredients." Based on this prompt, the AI ​​will provide appropriate food suggestions.

[0553] These features of the system allow users to make food choices that suit their emotions and health, supporting a healthy and comfortable lifestyle.

[0554] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0555] Step 1:

[0556] The user launches the camera app on their device and takes a picture of the food they intend to consume. The image of the food is captured by the device's camera as input. Specifically, the user holds the apple up to the camera and presses the shutter button. This image data is then sent to the next processing step.

[0557] Step 2:

[0558] The terminal processes the captured image data using a preprocessing mechanism. Specifically, it removes unwanted backgrounds from the image data and converts it to a resolution suitable for subsequent analysis. The input for this process is the food image acquired in the previous step, and the output is image data in a format suitable for analysis.

[0559] Step 3:

[0560] The device analyzes the user's emotional state using emotion analysis techniques. It uses a front camera and microphone to capture the user's facial expressions and voice as input, and generates emotion data as a result of the analysis. For example, it might capture the user taking a deep breath and output emotion data such as "relaxed."

[0561] Step 4:

[0562] The device transmits pre-processed image data and emotion data to a server in the cloud via a transmission method. The input is the data generated in steps 2 and 3, and the output is the data transmitted to the server. The device uploads the data using a secure protocol.

[0563] Step 5:

[0564] The server analyzes received image data using image analysis tools to evaluate the freshness and nutritional value of food. The input is pre-processed image data, and the analysis results in the output of data regarding food freshness and nutritional value. Machine learning algorithms are applied during this process.

[0565] Step 6:

[0566] The server uses an evaluation information generation mechanism to devise appropriate consumption methods based on input emotion data and analyzed food evaluation data. As output, it generates personalized food suggestions tailored to the user's situation. For example, it might suggest, "If you want to relax, you should try a recipe using herbal tea."

[0567] Step 7:

[0568] The server returns the generated evaluation information and suggestions to the terminal via an information notification system. The input is the generated evaluation information, and the output is the transmission of data to the user terminal.

[0569] Step 8:

[0570] The device displays the received information to the user. The input is evaluation information sent from the server, and as a specific action, a message such as "Let's try these foods today to relax" is displayed. The display is adapted according to the user's emotional state, for example, by using gentle colors.

[0571] (Application Example 2)

[0572] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0573] In modern living environments, consumers face a variety of stressors, and appropriate food choices are required to address their emotional states. However, providing dietary suggestions tailored to individual emotional states is not easy, and there is a lack of systems that support consumers' mental and physical health while considering food quality and nutritional value.

[0574] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0575] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for suggesting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, an emotion recognition means for analyzing the user's facial expressions and voice to identify their emotional state, and an individualized suggestion means for suggesting a consumption plan suitable for the user based on their emotional state and the quality of the object. This makes it possible to promote appropriate food consumption according to the user's emotional state and support the realization of a healthy diet.

[0576] An "image acquisition means" is a device that captures images of an object or user and acquires them as digital data.

[0577] "Preprocessing means" refers to a device or function that performs processing to remove unnecessary parts from an image and convert it into a format suitable for analysis.

[0578] "Transmission means" refers to a device or function for transmitting pre-processed image data to a remote processing device.

[0579] "Image analysis means" refers to a device or function that analyzes received image data and evaluates the freshness and quality of an object.

[0580] "Evaluation information generation means" refers to a device or function that generates information to suggest appropriate expiration dates and consumption methods based on the results of image analysis.

[0581] An "information notification means" is a device or interface for effectively conveying generated evaluation information to the user.

[0582] "Emotion recognition means" refers to a device or function that analyzes the user's facial expressions and voice to identify their current emotional state.

[0583] "Personalized suggestion means" refers to a device or function that suggests the optimal consumption plan or recipe to the user based on the user's emotional state and food quality information.

[0584] To implement this invention, the user first uses the terminal's camera to take an image of food. The terminal uses a pre-processing means to remove unnecessary parts from this image and convert it into a format suitable for analysis. Next, the pre-processed image data and the user's facial expressions and voice data are collected and transmitted to a remote server by a transmission means.

[0585] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Simultaneously, emotion recognition tools analyze the user's facial expressions and voice data to identify the user's current emotional state. This provides data such as whether the user is fatigued or stressed.

[0586] Based on this data, the evaluation information generation means generates evaluation information and creates information such as expiration dates and appropriate recipes. In particular, the personalized suggestion means, which responds to emotion recognition results, proposes a food consumption plan tailored to the user's current emotional state. For example, if relaxation is needed, it recommends easy-to-make herbal tea or salad.

[0587] The generated evaluation information and personalized suggestions are sent to the device via an information notification system and displayed to the user. The information is displayed on the screen with a tone and design that takes into account the user's emotional state, making it easy to read and understand.

[0588] As a concrete example, when a mother operates the device after returning home, the system senses her fatigue and suggests a simple pasta dish and a relaxing herbal tea. This system helps users to consume food that is appropriate for their emotional state at the time, supporting a health-conscious diet.

[0589] An example of a prompt for a generative AI model would be: "The analyzed emotional state is 'fatigue,' and you have tomatoes and cheese in the refrigerator. Please suggest a simple dish that will help you relax."

[0590] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0591] Step 1:

[0592] The user takes a picture of food using the device's camera. The input obtained is image data of the food. This activates the image acquisition mechanism, which collects the image as a dataset.

[0593] Step 2:

[0594] The terminal uses preprocessing to remove unnecessary parts from image data and converts it into a format suitable for analysis. The input is raw image data of food, and the output is preprocessed image data with noise removed. Specifically, image filtering and resizing are performed.

[0595] Step 3:

[0596] The terminal transmits pre-processed image data, user facial expressions, and audio data to the server using a transmission method. The input consists of pre-processed images and user audio / video data, which are then transferred to the cloud server.

[0597] Step 4:

[0598] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Pre-processed image data is the input, and data evaluating freshness and nutritional value is the output. Here, data calculations such as image recognition and quality evaluation are performed.

[0599] Step 5:

[0600] The server uses emotion recognition to analyze the user's voice and video data and identify their emotional state. The input is data indicating the user's emotions, and the output is data representing the corresponding emotional state. Emotional states are detected through voice tone analysis and facial expression recognition.

[0601] Step 6:

[0602] The server uses an evaluation information generation mechanism to generate an appropriate consumption plan based on the acquired quality evaluation and emotional state. The input is freshness evaluation data and emotional state, and the output is the proposed consumption plan. Here, a generation AI model is used to create the prompt text.

[0603] Step 7:

[0604] The server transmits the generated consumption plan and evaluation information to the terminal via an information notification system and displays it to the user. The input is the generated consumption plan, and the output is a visual or auditory notification. Specific actions update the user interface and provide information to the user.

[0605] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0606] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0607] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0608] [Fourth Embodiment]

[0609] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0610] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0611] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0612] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0613] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0614] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0615] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0616] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0617] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0618] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0619] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0620] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0621] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] To implement this invention, the user first takes a photograph of food using a smartphone or other device. A dedicated application is installed on this device, and the user activates the camera through the application and takes a photograph of the food. The captured image is converted into a format suitable for analysis by the device's pre-processing function. During this process, the background is removed and necessary parts are cropped.

[0623] Next, the terminal sends the pre-processed images to a server in the cloud. The server uses image analysis algorithms to analyze the received images and evaluate the freshness and nutritional value of the food. This analysis includes techniques for analyzing image characteristics such as hue and shape. Furthermore, the server generates evaluation information based on the analysis results. This information includes recommended expiration dates, cooking methods, and storage methods.

[0624] The generated evaluation information is sent to the user's device via the information notification function. The device then displays this information to the user through the app. Based on the displayed information, the user can plan to consume food appropriately.

[0625] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image and evaluates its current freshness. The evaluation information generated would include something like, "The apple is fresh and it is recommended to consume it within two days. Refrigeration is the best place to store it." Based on this information, the user can decide when to consume the apple and avoid waste.

[0626] This series of processes ensures that users can always consume fresh food as needed, thereby contributing to a reduction in food waste. Furthermore, the nutritional information provided based on evaluation data helps users make healthy food choices.

[0627] The following describes the processing flow.

[0628] Step 1:

[0629] The user launches their smartphone's camera app and takes a picture of the food they want to evaluate. The user is expected to adjust the lighting and angle appropriately during shooting to obtain an image best suited for analysis.

[0630] Step 2:

[0631] The device receives the captured image and applies a preprocessing algorithm. This includes removing unwanted backgrounds from the image and cropping it to include only the necessary parts. The image's brightness and contrast are also adjusted to prepare it for optimal analysis.

[0632] Step 3:

[0633] The terminal sends pre-processed image data to the cloud server using a communication protocol. This transmission is encrypted to ensure security.

[0634] Step 4:

[0635] The server runs an AI model to analyze the received images. This model analyzes the color, shape, and surface features of the food in the images to evaluate its freshness and nutritional value.

[0636] Step 5:

[0637] Based on the analysis results, the server generates evaluation information regarding expiration dates, optimal consumption methods, and storage methods. In doing so, it creates personalized recommendations that take into account accumulated data and the user's past history.

[0638] Step 6:

[0639] The server sends the generated evaluation information to the user's terminal. The transmitted data is formatted in an easy-to-read format, ensuring that the content is easy for the user to understand.

[0640] Step 7:

[0641] The device displays the received information to the user. Through notifications and in-app displays, users can check the status of food and recommendations, and make concrete consumption plans.

[0642] (Example 1)

[0643] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] Managing food freshness and reducing food waste are critical challenges facing modern consumers. In particular, food waste often arises from a lack of knowledge about expiration dates and proper storage methods, making consumers' lives inconvenient. Providing nutritional information to support healthy food choices is also crucial. Therefore, there is a need to provide systems that enable consumers to properly manage and effectively consume food.

[0645] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0646] In this invention, the server includes, as an image analysis means, a means for analyzing received images using a dedicated algorithm and evaluating the freshness and nutritional value of the object; as an evaluation information generation means, a means for presenting recommended expiration dates, storage methods, and cooking methods based on the analysis results; and as an information notification means, a means for displaying the generated evaluation information to the user via a terminal. This makes it possible for consumers to immediately obtain information to properly manage food and make healthy food choices.

[0647] "Image acquisition means" refers to devices or software functions for photographing food, which acquire image data captured by the user.

[0648] "Preprocessing means" refers to a function that removes the background and unnecessary parts from the acquired image and converts the image into a format suitable for analysis.

[0649] "Transmission means" refers to a communication function for securely transmitting pre-processed image data to a remote processing unit.

[0650] "Image analysis means" refers to a function that analyzes received images using specialized algorithms and machine learning models to evaluate the freshness and nutritional value of food.

[0651] "Evaluation information generation means" refers to a function that generates information such as recommended expiration dates, storage methods, and cooking methods for food products based on the results of image analysis.

[0652] "Information notification means" refers to a function that provides generated evaluation information to the user's terminal and displays it to the user.

[0653] The invention will now be described in terms of its embodiments. This invention is a system that operates primarily around a user's terminal and a server in the cloud.

[0654] First, the user uses a smartphone or other device to photograph the food. A dedicated application is installed on the device, and this application controls the camera function to acquire images of the food. While the standard smartphone camera function is used for image acquisition, additional software correction may be performed to further improve accuracy.

[0655] Next, the device preprocesses the acquired image immediately after capture. Here, image processing algorithms are used to remove unwanted background elements and extract only the food portion. Color correction and noise reduction are also performed, and the image is converted into a format suitable for analysis. This preprocessing may utilize known image processing algorithms or image processing libraries from specific manufacturers.

[0656] The pre-processed images are sent from the terminal to a server in the cloud. This transmission uses an internet connection and is encrypted to ensure security.

[0657] Images that reach the server are processed by a dedicated image analysis algorithm. The server analyzes the received image data to estimate the freshness and nutritional value of the food. This analysis uses a generative AI model, which automatically analyzes features such as the hue, shape, and texture of the food.

[0658] Based on the analysis results, the server generates evaluation information. This information includes the recommended expiration date, storage method, and cooking method for the food. This evaluation information is transmitted to the terminal in real time for the user to receive.

[0659] Finally, the device displays the received evaluation information to the user through the application. Based on this information, the user can make plans, for example, on when to consume food and how to store it.

[0660] As a concrete example, consider a case where a user takes a picture of an apple. The server analyzes the color and texture of the apple's skin from the image, assesses its freshness, and provides information such as, "It is recommended to consume the apple within two days and store it in the refrigerator." In this way, it becomes easier to plan food consumption.

[0661] An example of a prompt to input into a generative AI model is, "Analyze this picture of an apple and tell me the recommended expiration date and storage method."

[0662] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0663] Step 1:

[0664] The user launches the application on their device and uses the camera function to take a picture of food. The input at this time is the image within the frame of the food that the user sees through the camera. Specifically, the user saves the image data to the device's memory by pressing the "Capture" button in the app.

[0665] Step 2:

[0666] The device preprocesses the acquired image. The input is the image taken in step 1, and the output is a cleared image of the food with the background removed. Specifically, the device uses an image processing algorithm to perform noise reduction, background removal, and color information correction.

[0667] Step 3:

[0668] The terminal sends the pre-processed image to the server. The input to this transmission is the pre-processed image that will be the output of step 2, and the output is the image data that the cloud server receives. During transmission, the terminal encrypts the data to ensure secure communication.

[0669] Step 4:

[0670] The server analyzes the received images. The input is the pre-processed image data sent in step 3. The server uses a dedicated image analysis algorithm, particularly a generative AI model, to analyze the hue and shape of the food and estimate its freshness and nutritional value. The output is evaluation data of the analysis results.

[0671] Step 5:

[0672] The server generates evaluation information based on the analysis results. The input is the analysis results, which are the output of step 4. Specifically, the server generates text information such as recommended expiration dates, storage methods, and cooking methods, and formats it in a way that is easy for the user to understand. The output is text data as evaluation information.

[0673] Step 6:

[0674] The server sends evaluation information to the terminal. The input for this transmission is the evaluation information generated in step 5. The output is data formatted in a format that can be displayed on the terminal. The server transmits this to the terminal via the internet.

[0675] Step 7:

[0676] The terminal displays the received evaluation information to the user via the application. The input is the evaluation information sent from the server in step 6. Specifically, the terminal utilizes its notification function to display the expiration date and storage method on the user interface. The output is the presentation of information to the user in a visually understandable format.

[0677] (Application Example 1)

[0678] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0679] Managing food at home is time-consuming, which can lead to a decline in freshness, loss of nutritional value, and unnecessary waste. Furthermore, while there is a need to reduce the burden of daily household tasks and perform them more efficiently, there is a challenge in autonomously managing food.

[0680] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0681] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for presenting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, a means for identifying the object to be photographed in the physical environment, an autonomous means for identifying the object to be photographed in the physical storage device, and a means for periodically scanning the object in the physical storage device based on defined criteria. This automates food management in the home, enabling efficient and waste-free consumption.

[0682] "Image acquisition means" refers to the function of collecting visual data of an object using a photographic device.

[0683] "Preprocessing means" refers to a function that extracts necessary information from acquired images and converts it into a format suitable for analysis.

[0684] "Transmission means" refers to the function of transferring processed data to a remote processing unit via a communication device.

[0685] "Image analysis means" refers to a function that analyzes received visual data and evaluates the characteristics and state of an object.

[0686] "Evaluation information generation means" refers to a function that generates useful information about an object based on the analysis results.

[0687] "Information notification means" refers to a function for transmitting and displaying generated evaluation information to users.

[0688] "Means for identifying objects to be photographed within a physical environment" refers to a system that has the function of identifying and accurately recognizing objects within a storage device.

[0689] "Means for periodically scanning objects within a physical storage device based on defined criteria" refers to a function that continuously monitors objects within a storage device and collects status information according to set conditions.

[0690] The system implementing this invention is designed to automate food management within the home. The system includes a series of processes for image acquisition, preprocessing, image analysis, evaluation information generation, and information notification.

[0691] The server uses Python and TensorFlow to execute the analysis algorithm. A home robot has a built-in camera and periodically photographs food as an image acquisition tool. This robot is based on iRobot and other common home robots and has the ability to autonomously identify food in the refrigerator. The captured images are preprocessed to remove the background and convert them into a format suitable for analysis. This processing includes image cropping and noise reduction.

[0692] Next, the terminal sends the pre-processed image to a cloud server. The server receives the transmitted data and uses image analysis tools to evaluate the condition of the food. Here, the hue, shape, and other visual features of the image are analyzed to evaluate the freshness and nutritional value of the food.

[0693] The server generates evaluation information based on the analysis results. The evaluation information generation means generates information on appropriate expiration dates and storage methods, and sends it to the terminal via the cloud. The terminal displays the received notification to the user in an easy-to-use format and provides recommendations regarding the condition of the food.

[0694] For example, if a user has tomatoes stored in their refrigerator, the robot can take a picture of them, and the server will analyze the image to generate information such as, "The tomatoes are ripe and should be consumed within three days." This promotes the proper consumption of food and reduces waste.

[0695] An example of a prompt for a generative AI model can be set as follows: "Analyze images of food in a refrigerator and recommend freshness assessment and expiration date." By entering this prompt, the system can perform the correct analysis and provide information.

[0696] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0697] Step 1:

[0698] The device uses a home robot's camera to photograph the food inside the refrigerator. The input is a physical image of the food, and the output is digital image data. This data is needed for the following processing.

[0699] Step 2:

[0700] The terminal preprocesses the captured image. The input is the digital image data obtained in step 1. Preprocessing reduces noise and converts the image into a format suitable for analysis by removing the background and cropping the food portion. The output is the preprocessed image.

[0701] Step 3:

[0702] The terminal sends the pre-processed image to the cloud server. The input is the pre-processed image obtained in step 2, and the output is the state information uploaded to the cloud. This state information is used in the next analysis step.

[0703] Step 4:

[0704] The server analyzes the image data received on the cloud. The input is the pre-processed image sent in step 3. The server uses image analysis tools to analyze the visual features of the image, such as hue and shape, and evaluates the freshness and nutritional value of the food. The output is the evaluation result.

[0705] Step 5:

[0706] The server generates evaluation information based on the evaluation results. The input is the evaluation results obtained in step 4. The server generates and outputs information regarding appropriate expiration dates and storage methods. This information is provided to the user.

[0707] Step 6:

[0708] The server sends the generated evaluation information to the terminal. The input is the evaluation information generated in step 5. The output is the evaluation information sent to the terminal.

[0709] Step 7:

[0710] The terminal notifies the user of the received evaluation information. The input is the evaluation information sent in step 6. Based on the evaluation information, the user can properly manage food and create a consumption plan. The output is specific instructions displayed on the terminal's display.

[0711] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0712] This invention evaluates the freshness and nutritional value of food and, using a system equipped with an emotion engine, provides personalized suggestions tailored to the user's emotional state. The system begins with the user taking a photograph of the food using a smartphone or other device, and then recognizes the user's emotions through the emotion engine.

[0713] Specifically, the user takes a picture of food using the device's camera app. During this process, the emotion engine uses the device's front camera and sensors to analyze the user's facial expressions and tone of voice to recognize their emotions. For example, it can obtain information such as whether the user is tired or stressed. The device preprocesses this information and then sends both the image data and emotion data to a cloud server.

[0714] The server analyzes received images and evaluates the freshness and nutritional value of the food. Based on emotional data, it also adjusts the content of the information notifications; for example, suggesting new recipes when the user is in a positive state and recommending simpler cooking methods when they are in a negative state. The evaluation information generation function combines the user's current emotional data with their past usage history to suggest the optimal consumption plan. It can even provide specific suggestions tailored to emotional states, such as "use ingredients with relaxing effects when you're feeling down."

[0715] The server sends the generated evaluation information and suggestions to the terminal, which then displays them to the user. The display is presented in an easy-to-read format that takes the user's emotional state into consideration, encouraging consumer behavior without requiring any user intervention. For example, if the user is feeling stressed, the notification is displayed in a softer tone, and the suggestions are concise to reduce the burden.

[0716] This system allows users to make optimal food choices based on their emotional state at the time, supporting a health-conscious diet. In addition to reducing food waste, using this system can also contribute to users' emotional management.

[0717] The following describes the processing flow.

[0718] Step 1:

[0719] The user activates their smartphone's camera and takes a picture of the food they want to evaluate. Simultaneously with the photo, the device's front camera and microphone capture the user's facial expressions and voice. This allows the user's emotional state to be recorded at the same time.

[0720] Step 2:

[0721] The device preprocesses the captured images, including cropping food portions and correcting image color. Simultaneously, the emotion engine analyzes the acquired audio and facial expression data, and sets an emotion category (e.g., stress, exhilaration, calmness) in real time.

[0722] Step 3:

[0723] The device pairs pre-processed image data with emotion data and sends it to the cloud server. The transmission is encrypted based on a security protocol.

[0724] Step 4:

[0725] After receiving image data, the server applies an AI image recognition algorithm to analyze the type of food, its freshness, and its nutritional value. It also receives emotional data simultaneously and compares it with a database to enable suggestions that take the user's emotional state into account.

[0726] Step 5:

[0727] Based on the analysis results, the server generates suggestions for expiration dates and cooking methods. Furthermore, it uses emotional data to customize suggestions according to the user's emotional state, for example, "If you're feeling stressed, make a simple and nutritious smoothie."

[0728] Step 6:

[0729] The server sends the generated evaluation information and suggestions to the terminal. The information sent is organized and formatted in an easy-to-understand format.

[0730] Step 7:

[0731] The device notifies the user of the received information and displays it within the app. The displayed content is adjusted according to the user's emotional state and is designed to be received appropriately. Based on this, the user can choose the optimal way to consume food.

[0732] (Example 2)

[0733] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0734] Conventional food quality evaluation systems only assess freshness and nutritional value, failing to consider the user's emotional state when making recommendations. Therefore, it was difficult to provide personalized recommendations based on the user's emotional state, making it challenging to improve user satisfaction and convenience. Furthermore, there is a need to optimize food consumption plans in conjunction with emotional states to support a healthy and mentally fulfilling diet.

[0735] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0736] In this invention, the server includes an image analysis means for analyzing received images and evaluating the quality of the object, an emotion analysis means for analyzing the user's emotional state and considering the analysis results, and an evaluation information generation means for presenting an appropriate consumption plan and consumption method based on the quality evaluation results and emotion analysis results. This enables personalized food suggestions that take into account the user's emotional state, making it possible to realize healthier and more emotionally considerate food consumption.

[0737] "Image acquisition means" refers to a device or software that acquires an image of an object using a device.

[0738] "Preprocessing means" refers to a device or software that removes unnecessary parts from acquired image data and converts it into a format suitable for analysis.

[0739] "Transmission means" refers to a device or protocol that transmits pre-processed data to a processing device located remotely.

[0740] "Image analysis means" refers to a device or algorithm that analyzes received image data and evaluates the quality and condition of the object from it.

[0741] "Emotional analysis means" refers to a device or program that analyzes a user's facial expressions and voice data to determine their emotional state.

[0742] "Evaluation information generation means" refers to a device or system for formulating and proposing appropriate consumption plans and methods based on quality evaluation results and sentiment analysis results.

[0743] "Information notification means" refers to a device or interface that notifies the user of generated evaluation information visually or audibly.

[0744] This invention provides a system that allows users to evaluate the quality of food and receive personalized consumption suggestions based on their emotional state. The system primarily consists of terminals and servers, each playing a specific role.

[0745] The user first takes a picture of the food using the device's image acquisition mechanism. The device is equipped with a camera application, which allows the user to easily record the food. The captured image is pre-processed to remove unnecessary parts and converted into a format suitable for analysis. Furthermore, the device uses emotion analysis to analyze the user's facial expressions and voice tone through the user's front camera and microphone, and acquires the user's emotion data.

[0746] Subsequently, this data is transmitted to a server via a transmission device. The server has an image analysis device that analyzes the received image data to evaluate the freshness and nutritional value of the food. The server also uses an emotion analysis device to generate evaluation information that takes into account the user's emotional state.

[0747] Based on the user's quality evaluation results and sentiment analysis results, the evaluation information generation system suggests optimal consumption plans and methods. For example, if the user is feeling stressed, the system suggests recipes with short cooking times and ingredients that have a relaxing effect. The server then uses an information notification system to transmit the generated evaluation information to the terminal.

[0748] The device visually displays the received information to the user. The display format is designed to be easy to see and understand, based on the user's emotional state. For example, if the user is feeling down, notifications might be displayed in soft colors and with concise messages.

[0749] Furthermore, an example of a prompt using a generative AI model is, "I'd like to have a relaxing meal today. Please recommend some ingredients." Based on this prompt, the AI ​​will provide appropriate food suggestions.

[0750] These features of the system allow users to make food choices that suit their emotions and health, supporting a healthy and comfortable lifestyle.

[0751] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0752] Step 1:

[0753] The user launches the camera app on their device and takes a picture of the food they intend to consume. The image of the food is captured by the device's camera as input. Specifically, the user holds the apple up to the camera and presses the shutter button. This image data is then sent to the next processing step.

[0754] Step 2:

[0755] The terminal processes the captured image data using a preprocessing mechanism. Specifically, it removes unwanted backgrounds from the image data and converts it to a resolution suitable for subsequent analysis. The input for this process is the food image acquired in the previous step, and the output is image data in a format suitable for analysis.

[0756] Step 3:

[0757] The device analyzes the user's emotional state using emotion analysis techniques. It uses a front camera and microphone to capture the user's facial expressions and voice as input, and generates emotion data as a result of the analysis. For example, it might capture the user taking a deep breath and output emotion data such as "relaxed."

[0758] Step 4:

[0759] The device transmits pre-processed image data and emotion data to a server in the cloud via a transmission method. The input is the data generated in steps 2 and 3, and the output is the data transmitted to the server. The device uploads the data using a secure protocol.

[0760] Step 5:

[0761] The server analyzes received image data using image analysis tools to evaluate the freshness and nutritional value of food. The input is pre-processed image data, and the analysis results in the output of data regarding food freshness and nutritional value. Machine learning algorithms are applied during this process.

[0762] Step 6:

[0763] The server uses an evaluation information generation mechanism to devise appropriate consumption methods based on input emotion data and analyzed food evaluation data. As output, it generates personalized food suggestions tailored to the user's situation. For example, it might suggest, "If you want to relax, you should try a recipe using herbal tea."

[0764] Step 7:

[0765] The server returns the generated evaluation information and suggestions to the terminal via an information notification system. The input is the generated evaluation information, and the output is the transmission of data to the user terminal.

[0766] Step 8:

[0767] The device displays the received information to the user. The input is evaluation information sent from the server, and as a specific action, a message such as "Let's try these foods today to relax" is displayed. The display is adapted according to the user's emotional state, for example, by using gentle colors.

[0768] (Application Example 2)

[0769] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0770] In modern living environments, consumers face a variety of stressors, and appropriate food choices are required to address their emotional states. However, providing dietary suggestions tailored to individual emotional states is not easy, and there is a lack of systems that support consumers' mental and physical health while considering food quality and nutritional value.

[0771] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0772] In this invention, the server includes an image acquisition means, a preprocessing means for removing unnecessary parts from an image and converting it to a format suitable for analysis, a transmission means for transmitting the preprocessed image to a remote processing device, an image analysis means for analyzing the received image and evaluating the quality of the object, an evaluation information generation means for suggesting an appropriate expiration date and consumption method based on the quality evaluation results, an information notification means for displaying the generated evaluation information to the user, an emotion recognition means for analyzing the user's facial expressions and voice to identify their emotional state, and an individualized suggestion means for suggesting a consumption plan suitable for the user based on their emotional state and the quality of the object. This makes it possible to promote appropriate food consumption according to the user's emotional state and support the realization of a healthy diet.

[0773] An "image acquisition means" is a device that captures images of an object or user and acquires them as digital data.

[0774] "Preprocessing means" refers to a device or function that performs processing to remove unnecessary parts from an image and convert it into a format suitable for analysis.

[0775] "Transmission means" refers to a device or function for transmitting pre-processed image data to a remote processing device.

[0776] "Image analysis means" refers to a device or function that analyzes received image data and evaluates the freshness and quality of an object.

[0777] "Evaluation information generation means" refers to a device or function that generates information to suggest appropriate expiration dates and consumption methods based on the results of image analysis.

[0778] An "information notification means" is a device or interface for effectively conveying generated evaluation information to the user.

[0779] "Emotion recognition means" refers to a device or function that analyzes the user's facial expressions and voice to identify their current emotional state.

[0780] "Personalized suggestion means" refers to a device or function that suggests the optimal consumption plan or recipe to the user based on the user's emotional state and food quality information.

[0781] To implement this invention, the user first uses the terminal's camera to take an image of food. The terminal uses a pre-processing means to remove unnecessary parts from this image and convert it into a format suitable for analysis. Next, the pre-processed image data and the user's facial expressions and voice data are collected and transmitted to a remote server by a transmission means.

[0782] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Simultaneously, emotion recognition tools analyze the user's facial expressions and voice data to identify the user's current emotional state. This provides data such as whether the user is fatigued or stressed.

[0783] Based on this data, the evaluation information generation means generates evaluation information and creates information such as expiration dates and appropriate recipes. In particular, the personalized suggestion means, which responds to emotion recognition results, proposes a food consumption plan tailored to the user's current emotional state. For example, if relaxation is needed, it recommends easy-to-make herbal tea or salad.

[0784] The generated evaluation information and personalized suggestions are sent to the device via an information notification system and displayed to the user. The information is displayed on the screen with a tone and design that takes into account the user's emotional state, making it easy to read and understand.

[0785] As a concrete example, when a mother operates the device after returning home, the system senses her fatigue and suggests a simple pasta dish and a relaxing herbal tea. This system helps users to consume food that is appropriate for their emotional state at the time, supporting a health-conscious diet.

[0786] An example of a prompt for a generative AI model would be: "The analyzed emotional state is 'fatigue,' and you have tomatoes and cheese in the refrigerator. Please suggest a simple dish that will help you relax."

[0787] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0788] Step 1:

[0789] The user takes a picture of food using the device's camera. The input obtained is image data of the food. This activates the image acquisition mechanism, which collects the image as a dataset.

[0790] Step 2:

[0791] The terminal uses preprocessing to remove unnecessary parts from image data and converts it into a format suitable for analysis. The input is raw image data of food, and the output is preprocessed image data with noise removed. Specifically, image filtering and resizing are performed.

[0792] Step 3:

[0793] The terminal transmits pre-processed image data, user facial expressions, and audio data to the server using a transmission method. The input consists of pre-processed images and user audio / video data, which are then transferred to the cloud server.

[0794] Step 4:

[0795] The server analyzes the received image data using image analysis tools to evaluate the freshness and quality of the food. Pre-processed image data is the input, and data evaluating freshness and nutritional value is the output. Here, data calculations such as image recognition and quality evaluation are performed.

[0796] Step 5:

[0797] The server uses emotion recognition to analyze the user's voice and video data and identify their emotional state. The input is data indicating the user's emotions, and the output is data representing the corresponding emotional state. Emotional states are detected through voice tone analysis and facial expression recognition.

[0798] Step 6:

[0799] The server uses an evaluation information generation mechanism to generate an appropriate consumption plan based on the acquired quality evaluation and emotional state. The input is freshness evaluation data and emotional state, and the output is the proposed consumption plan. Here, a generation AI model is used to create the prompt text.

[0800] Step 7:

[0801] The server transmits the generated consumption plan and evaluation information to the terminal via an information notification system and displays it to the user. The input is the generated consumption plan, and the output is a visual or auditory notification. Specific actions update the user interface and provide information to the user.

[0802] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0803] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0805] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0806] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0807] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0808] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0809] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0810] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0811] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0812] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0813] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0814] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0815] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0816] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0817] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0818] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0819] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0820] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0821] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0822] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0823] The following is further disclosed regarding the embodiments described above.

[0824] (Claim 1)

[0825] Image acquisition method,

[0826] As a preprocessing means, means for removing unnecessary parts from the image and converting it to a format suitable for analysis,

[0827] The transmission means includes means for transmitting a pre-processed image to a remote processing device,

[0828] As an image analysis means, there is a means for analyzing the received image and performing quality evaluation of the object,

[0829] As a means for generating evaluation information, it includes a means for suggesting appropriate expiration dates and consumption methods based on quality evaluation results,

[0830] As a means of notifying information, a means of displaying the generated evaluation information to the user,

[0831] A system that includes this.

[0832] (Claim 2)

[0833] The system according to claim 1, further comprising means for predicting changes in freshness over time and periodically notifying the user.

[0834] (Claim 3)

[0835] The system according to claim 1, further comprising means for providing personalized suggestions that take into account the user's past usage history and preferences.

[0836] "Example 1"

[0837] (Claim 1)

[0838] As a means of acquiring images, there is a means of using a device that allows the user to photograph food,

[0839] As a preprocessing method, the means include removing unnecessary parts of the image and converting it into a format suitable for analysis,

[0840] The transmission means includes means for securely transmitting pre-processed images to a remote processing unit,

[0841] As an image analysis method, it involves analyzing the received image using a dedicated algorithm to evaluate the freshness and nutritional value of the object,

[0842] As a means for generating evaluation information, it includes a means for presenting recommended expiration dates, storage methods, and cooking methods based on the analysis results,

[0843] As a means of notifying information, a means of displaying the generated evaluation information to the user via a terminal,

[0844] A system that includes this.

[0845] (Claim 2)

[0846] The system according to claim 1, further comprising means for predicting changes in freshness over time and periodically notifying the user's device.

[0847] (Claim 3)

[0848] The system according to claim 1, further comprising means for providing personalized food consumption suggestions, taking into account the user's past usage history and preferences.

[0849] "Application Example 1"

[0850] (Claim 1)

[0851] Image acquisition method,

[0852] As a preprocessing means, means for removing unnecessary parts from the image and converting it to a format suitable for analysis,

[0853] The transmission means includes means for transmitting a pre-processed image to a remote processing device,

[0854] As an image analysis means, there is a means for analyzing the received image and performing quality evaluation of the object,

[0855] As a means for generating evaluation information, it includes a means for suggesting appropriate expiration dates and consumption methods based on quality evaluation results,

[0856] As a means of notifying information, a means of displaying the generated evaluation information to the user,

[0857] As a means for identifying a subject to be photographed within a physical environment, it includes a means for autonomously identifying a subject to be photographed within a physical storage device,

[0858] A means for periodically scanning objects within a physical storage device based on defined criteria,

[0859] A system that includes this.

[0860] (Claim 2)

[0861] The system according to claim 1, further comprising means for predicting changes in freshness over time and periodically notifying the user.

[0862] (Claim 3)

[0863] The system according to claim 1, further comprising means for providing personalized suggestions that take into account the user's past usage history and preferences.

[0864] "Example 2 of combining an emotion engine"

[0865] (Claim 1)

[0866] Image acquisition method,

[0867] As a preprocessing means, means for removing unnecessary parts from the image and converting it to a format suitable for analysis,

[0868] The transmission means includes means for transmitting pre-processed images and emotion data to a remote processing unit,

[0869] As an image analysis means, there is a means for analyzing the received image and performing quality evaluation of the object,

[0870] As a means of emotion analysis, there is a means to analyze the user's emotional state and consider the analysis results,

[0871] As a means for generating evaluation information, it includes a means for presenting appropriate consumption plans and consumption methods based on quality evaluation results and sentiment analysis results,

[0872] As a means of notifying information, a means of displaying the generated evaluation information in a format that is appropriate to the user's emotional state,

[0873] A system that includes this.

[0874] (Claim 2)

[0875] The system according to claim 1, further comprising means for predicting changes in freshness over time and periodically notifying the user.

[0876] (Claim 3)

[0877] The system according to claim 1, further comprising means for providing personalized suggestions, taking into account the user's past usage history and emotional state.

[0878] "Application example 2 when combining with an emotional engine"

[0879] (Claim 1)

[0880] Image acquisition method,

[0881] As a preprocessing means, means for removing unnecessary parts from the image and converting it to a format suitable for analysis,

[0882] The transmission means includes means for transmitting a pre-processed image to a remote processing device,

[0883] As an image analysis means, there is a means for analyzing the received image and performing quality evaluation of the object,

[0884] As a means for generating evaluation information, it includes a means for suggesting appropriate expiration dates and consumption methods based on quality evaluation results,

[0885] As a means of notifying information, a means of displaying the generated evaluation information to the user,

[0886] As a means of emotion recognition, it includes a means of analyzing the user's facial expressions and voice to identify their emotional state,

[0887] As a means of providing personalized suggestions, a means of presenting a consumption plan suitable for the user based on their emotional state and the quality of the object,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, further comprising means for predicting changes in freshness over time and periodically notifying the user.

[0891] (Claim 3)

[0892] The system according to claim 1, further comprising means for providing personalized suggestions that take into account the user's past usage history and preferences. [Explanation of symbols]

[0893] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Image acquisition method, As a preprocessing means, means for removing unnecessary parts from the image and converting it to a format suitable for analysis, The transmission means includes means for transmitting a pre-processed image to a remote processing device, As an image analysis means, there is a means for analyzing the received image and performing quality evaluation of the object, As a means for generating evaluation information, it includes a means for suggesting appropriate expiration dates and consumption methods based on quality evaluation results, As a means of notifying information, a means of displaying the generated evaluation information to the user, A system that includes this.

2. The system according to claim 1, further comprising means for predicting changes in freshness over time and periodically notifying the user.

3. The system according to claim 1, further comprising means for providing personalized suggestions that take into account the user's past usage history and preferences.

Images