System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring an image of a food material photographed by a user; means for transmitting the acquired image to a server; means for recognizing a food material by analyzing the image in the server; means for providing the recognized food material information to the user; means for generating a recipe based on the provided food material information; and means for providing the generated recipe to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been an increasing demand for recipe creation systems that allow users to effectively utilize ingredients. However, the process of recognizing ingredients and creating recipes based on that information must be performed quickly and accurately, and current systems lack the means to efficiently perform these processes. As a result, users often find it difficult to effectively use ingredients. Therefore, an objective of the present invention is to provide a system that allows users to easily recognize ingredients and create recipes based on those ingredients. [Means for solving the problem]

[0005] The system of the present invention includes a means for acquiring an image of an ingredient photographed by a user, a means for transmitting the acquired image to a server, a means for analyzing the image in the server to recognize the ingredient, a means for providing the user with information about the recognized ingredient, a means for generating a recipe based on the provided information about the ingredient, and a means for providing the user with the generated recipe. This allows a user to easily recognize an ingredient and generate a recipe based on that ingredient simply by photographing an image of the ingredient.

[0006] An "ingredient" is a substance used as an ingredient in a dish.

[0007] An "image" is data that visually records the shape and color of an object.

[0008] A "user" is a person who operates the system to recognize ingredients and generate recipes.

[0009] A "terminal" is an electronic device used by a user, which is a computer device that has a camera function and is connected to the Internet.

[0010] A "server" is a remote computer system that receives, analyzes, and processes data sent from a terminal.

[0011] "Means of acquisition" refers to the functions and applications that allow the terminal to take and record images.

[0012] The "transmitting means" is a communication function for transmitting the acquired image data to the server.

[0013] The "means for analyzing and recognizing ingredients" is a process in which the server uses an image recognition algorithm to identify ingredients from an image.

[0014] "Means for providing" refers to the function by which the server transmits the recognition results and generated recipes to the terminal in a format that can be viewed by the user.

[0015] The "means for generating a recipe" is the process of planning an appropriate cooking method based on the acquired ingredient information and outputting it as a detailed recipe.

[0016] A "generated recipe" is a proposal that specifically shows the steps and ingredients for cooking using the ingredients used by the user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0039] Ingredient recognition processing

[0040] 1. Acquire images of ingredients:

[0041] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[0042] 2. Send image data to the server:

[0043] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[0044] 3. Recognizing ingredients using image recognition models:

[0045] The server inputs the received image data into an image recognition algorithm (for example, a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[0046] 4. Providing ingredients information to users:

[0047] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[0048] Handling recipe generation

[0049] 1. Request to generate a recipe:

[0050] The user uses the device interface and instructs the device to "create a recipe using tomatoes."

[0051] 2. Send the request to the server:

[0052] The terminal sends a request including ingredient information (tomato) to the server.

[0053] 3. Give instructions to the generative AI:

[0054] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes."

[0055] 4. Generate the recipe:

[0056] Generative AI generates recipes based on instructions. For example, it generates the following recipes:

[0057] Tomato and mozzarella salad

[0058] material:

[0059] Tomatoes: 2

[0060] Mozzarella cheese: 100g

[0061] Basil leaves: appropriate amount

[0062] Olive oil: 2 tablespoons

[0063] Salt: 1 / 2 teaspoon

[0064] Black pepper: appropriate amount

[0065] Instructions:

[0066] 1. Slice the tomatoes.

[0067] 2. Slice the mozzarella cheese.

[0068] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0069] 4. Garnish with basil leaves.

[0070] 5. Drizzle with olive oil and season with salt and black pepper.

[0071] 5. Providing recipe data to users:

[0072] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[0073] In this way, the system of the present invention provides users with the ability to easily recognize ingredients and create recipes, helping them make the most of the ingredients they have on hand. This allows users to make their daily cooking activities more efficient and creative. This system can reduce ingredient waste and enrich users' cooking repertoire.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] A user takes a photo of an ingredient using a device such as a smartphone or tablet, and the device uses its camera function to acquire image data of the ingredient.

[0077] Step 2:

[0078] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[0079] Step 3:

[0080] The server temporarily stores the received image data, which is then prepared for analysis.

[0081] Step 4:

[0082] The server feeds the image data into an image recognition model, which uses deep learning algorithms to identify ingredients in the image.

[0083] Step 5:

[0084] The server analyzes the output of the image recognition model and recognizes the food. For example, if the model identifies a food as a "tomato," the server will recognize the food as a tomato.

[0085] Step 6:

[0086] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[0087] Step 7:

[0088] The device parses the received ingredient information and displays "Recognized ingredient: Tomato" to the user. Based on this information, the user prepares for the next operation.

[0089] Step 8:

[0090] The user uses the terminal interface to say "generate a recipe using tomatoes," which is treated as an appropriate command within the application.

[0091] Step 9:

[0092] The device sends a request to the server to "create a recipe using tomatoes." This request includes ingredient information (tomatoes).

[0093] Step 10:

[0094] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes."

[0095] Step 11:

[0096] The generative AI uses an internal natural language generation algorithm to generate tomato-based recipes, including details such as the title, ingredients, and cooking instructions.

[0097] Step 12:

[0098] The server formats the generated recipe into JSON format and sends it to the device.

[0099] Step 13:

[0100] The device parses the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in a layout that the user can view. For example, a detailed recipe for "Tomato and Mozzarella Salad" is displayed.

[0101] Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] In today's world, users need to efficiently utilize the ingredients they use on a daily basis and quickly obtain a variety of recipes based on those ingredients. However, users often have to spend a lot of time and effort to find appropriate recipes for the ingredients they have on hand. Furthermore, existing recipe suggestion systems often struggle to flexibly accommodate the specific ingredients a user has and are unable to meet individual needs. Therefore, an effective means is needed to reduce ingredient waste and support users' cooking activities.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image and recognizing ingredients in the server, means for providing the recognized ingredient information to the user, means for generating a prompt sentence based on the provided ingredient information and generating a recipe using a generative AI model, and means for providing the generated recipe to the user, thereby enabling the user to quickly obtain a variety of specific recipes based on ingredients on hand.

[0107] "User" refers to an individual who utilizes the system to request ingredient recognition and recipe generation.

[0108] "Terminal" refers to a hardware device that a user uses to operate the system, and specifically includes a smartphone or tablet.

[0109] "Server" refers to a remote computer system that analyzes image data and generates recipes.

[0110] "Means for acquiring images" refers to the function that allows a user to take a photo of an ingredient using a terminal.

[0111] "Means for transmitting to a server" refers to the function of transmitting image data captured from a terminal to a server via the Internet.

[0112] "Means for analyzing images and recognizing ingredients" refers to the algorithms and processes for analyzing image data received by the server and identifying the corresponding ingredients.

[0113] "Means for providing ingredient information" refers to the function of the server returning the ingredient information identified by the server to the terminal and notifying the user.

[0114] "Means for generating a prompt sentence" refers to a function that generates an input sentence required for recipe generation based on the identified ingredient information.

[0115] "Means for generating a recipe using a generative AI model" refers to a function that uses a generative AI model (e.g., a natural language generation model) to generate a specific recipe based on a prompt sentence.

[0116] The "means for providing the generated recipe to the user" refers to a function for providing the generated recipe to the terminal and displaying it to the user.

[0117] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0118] First, this system uses a device (such as a smartphone or tablet) for users to take photos of ingredients. The image data of the ingredients taken by the user is sent from the device and stored on a server. The device then transmits this data via the HTTP protocol.

[0119] The server then uses a machine learning model (e.g., an image recognition model using deep learning) to analyze the received image data. This model often uses algorithms such as ResNet or YOLO. The server uses this model to analyze the image data and identify the ingredients. For example, the server may identify the ingredient in the image taken by the user as a "tomato."

[0120] The server sends the analysis results back to the device to provide to the user, and the device displays this information to the user, such as "Recognized ingredient: Tomato."

[0121] Next, the user makes a request through the device interface to generate a recipe based on specific ingredients, for example, "Generate a recipe using tomatoes." The device sends this request to the server, which generates a prompt, such as "Generate a simple and delicious recipe using tomatoes."

[0122] The server then uses a generative AI model (e.g., a natural language generation model such as GPT-4®) to create a recipe based on the generated prompt. The model takes the prompt as input and generates a detailed recipe, such as a "Tomato and Mozzarella Salad" with specific, detailed instructions:

[0123] Tomato and mozzarella salad

[0124] material:

[0125] Tomatoes: 2

[0126] Mozzarella cheese: 100g

[0127] Basil leaves: appropriate amount

[0128] Olive oil: 2 tablespoons

[0129] Salt: 1 / 2 teaspoon

[0130] Black pepper: appropriate amount

[0131] Instructions:

[0132] 1. Slice the tomatoes.

[0133] 2. Slice the mozzarella cheese.

[0134] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0135] 4. Garnish with basil leaves.

[0136] 5. Drizzle with olive oil and season with salt and black pepper.

[0137] The server sends the generated recipe back to the device, and the device provides this recipe information to the user, who can check the recipe on the device screen and use it in their cooking activities.

[0138] In this way, the system of the present invention supports the user's cooking activities by recognizing ingredients on hand and providing specific and detailed recipes based on those ingredients, allowing the user to cook efficiently and creatively without wasting ingredients.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1: Get an image of the ingredient

[0141] A user takes a photo of an ingredient using a device such as a smartphone or tablet. Specifically, the user opens a camera app and takes a clear image of the ingredient (e.g., a tomato). The input data is an image file of the tomato, and this image is saved on the device.

[0142] Step 2: Send image data to the server

[0143] The image data captured by the device is sent to a server via the Internet. Specifically, an HTTP POST request is used to send the image file to a specific API endpoint on the server. The input data is the saved image file of the tomato, and the output data is the image file passed to the server.

[0144] Step 3: Analyze the image and recognize the ingredients

[0145] The server analyzes the received image data using a deep learning model. Specifically, it runs machine learning algorithms such as ResNet and YOLO to recognize ingredients in the image. The input data is the image file of the tomato that was sent, and the output data is the ingredient information recognized as "tomato."

[0146] Step 4: Provide ingredients information to users

[0147] The server returns the recognition results to the device, which then displays this information to the user. Specifically, the server returns the information "Ingredient: Tomato" in JSON format, and the device analyzes this information and displays "Recognized ingredient: Tomato" on the screen. The input data is the recognized ingredient information, and the output data is the ingredient information displayed to the user.

[0148] Step 5: Request Recipe Generation

[0149] The user issues a command through the device interface to "create a recipe using tomatoes." Specifically, the user taps a button in the application to send the request. The input data is the user's command, and the output data is the creation request.

[0150] Step 6: Send the request to the server

[0151] The terminal sends a user request to the server. This request contains a specific instruction: "Generate a recipe using tomatoes." The input data is the generation request obtained from the user, and the output data is the request sent to the server.

[0152] Step 7: Generate the prompt

[0153] The server generates the prompt required to generate a recipe based on the provided ingredient information. For example, a prompt in the format "Please generate a simple and delicious recipe using tomatoes" is created. The input data is the ingredient information and the generation request, and the output data is the generated prompt.

[0154] Step 8: Generate recipes using generative AI models

[0155] The server uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. Specifically, the prompt is input into the model, and a detailed recipe (e.g., "Tomato and mozzarella salad") is generated. The input data is the prompt, and the output data is the generated recipe.

[0156] Step 9: Provide the generated recipe to the user

[0157] The server returns the generated recipe to the device, and the device provides this recipe information to the user. Specifically, the server returns detailed recipe information in JSON format, and the device parses this information and displays it on the screen. The input data is the generated recipe, and the output data is the recipe information displayed to the user.

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] Today's consumers have a growing need for easy access to recipe information that effectively utilizes ingredients purchased at supermarkets. However, conventional methods are limited in suggesting recipes based on specific ingredients, and lack means to provide information on other related ingredients available in the store and their locations. This makes it difficult for consumers to efficiently purchase the ingredients they need and smoothly carry out cooking activities.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for acquiring images of ingredients photographed by a user, means for transmitting the acquired images to the server, means for analyzing the images to recognize ingredients, means for providing the user with the recognized ingredient information, means for generating a recipe based on the provided ingredient information, means for providing the generated recipe to the user, means for indicating the location of related ingredients based on ingredient information photographed in the store, and means for generating a recipe based on available ingredients in conjunction with store inventory information. This allows consumers to obtain location information for necessary ingredients based on information on ingredients they wish to purchase in the store, and easily obtain recipes that make the most of the ingredients they have purchased.

[0163] A "user" is a consumer who uses the system to take photos of ingredients and obtain recipe information.

[0164] "Ingredients" are food ingredients that are photographed by the user and are recognized by the system.

[0165] An "image" refers to visual information obtained by a user photographing an ingredient.

[0166] A "server" is a computer device that receives images sent by users and performs analysis and data processing.

[0167] "Means" refers to a method or apparatus for performing a particular function or process.

[0168] "Analysis" refers to the process in which the server analyzes the image data received and recognizes the ingredients.

[0169] "Recognition" is the process of identifying specific ingredients from the image analyzed by the server.

[0170] A "recipe" is a list of cooking instructions and necessary ingredients generated based on recognized ingredient information.

[0171] "Providing" refers to sending information from the server to the user and the user receiving that information.

[0172] A "store" refers to a physical sales location such as a supermarket where users purchase ingredients.

[0173] "Location of related ingredients" is location information of the sales area in the store that sells other ingredients related to the ingredient photographed by the user.

[0174] "Store inventory information" is data on the inventory status of ingredients currently being sold in the store.

[0175] "Integration" refers to the sharing of information and collaboration between different systems and data sources.

[0176] "Available ingredients" refers to food ingredients that are currently available in the store, i.e., available for purchase.

[0177] A "machine learning model" refers to an algorithm that automatically learns from large amounts of data and performs data analysis and predictions.

[0178] A "natural language generation model" is a model that generates linguistically natural sentences based on text data.

[0179] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0180] System configuration

[0181] 1. User Device:

[0182] A smartphone, tablet, etc. is a device that allows users to take photos of ingredients and request the system to create a recipe.

[0183] 2. Server:

[0184] A high-performance computer device, the server processes image data sent by users, recognizes ingredients, and generates recipes.

[0185] 3. Machine learning models:

[0186] Image recognition algorithms built using deep learning frameworks such as TENSORFLOW (registered trademark) and PyTorch, used to recognize ingredients from captured images.

[0187] 4. Natural Language Generation Models:

[0188] It uses generative AI (Natural Language Processing model) to generate recipes based on ingredients.

[0189] System Operation Overview

[0190] 1. Ingredient Recognition Process:

[0191] The user takes a photo of the food using their smartphone and sends the image data to the server.

[0192] The server inputs the received image data into an image recognition algorithm to recognize the ingredients.

[0193] The server returns the recognized ingredient information to the user terminal and displays it to the user.

[0194] 2. Recipe generation process:

[0195] The user requests recipe creation based on the provided ingredient information.

[0196] The server issues instructions to the generative AI model to generate the recipe.

[0197] The server transmits the generated recipe to the user terminal and displays it to the user.

[0198] 3. Providing store information:

[0199] Based on the food information photographed in the store, the location of related ingredients is displayed.

[0200] It works with store inventory information to generate recipes based on available ingredients.

[0201] Program processing flow

[0202] The server first receives image data sent from the user's device. Next, this image data is input into a machine learning model (using TensorFlow or PyTorch) to recognize ingredients. The recognition result (e.g., "tomato") is sent back to the user's device. The user then requests recipe generation based on the provided information. The server generates a recipe by inputting a prompt statement, such as "Please generate a recipe using tomatoes," into the generative AI model. The generated recipe is sent to the user's device and provided to the user. It can also be linked to in-store inventory information to show the user where related ingredients can be purchased.

[0203] Specific examples

[0204] A user takes a picture of a tomato with their smartphone at a supermarket and uses the app to obtain ingredient information. The server receives the image and uses a machine learning model to recognize it as a "tomato." When the user then requests, "Please generate a recipe using tomatoes," the server uses a generative AI model to generate a recipe, such as "Tomato and Mozzarella Salad," and provides the user with such a recipe. Additionally, by providing information on the location and inventory of mozzarella cheese in the store, the server can efficiently purchase the ingredients they need.

[0205] In this way, it is possible to support the user's cooking activities, reduce food waste, and enrich the user's cooking repertoire.

[0206] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0207] Step 1:

[0208] The user takes a photo of an ingredient with their smartphone. The input is image data acquired from the smartphone camera. The user takes a photo of an ingredient (e.g., a tomato) and inputs the image data into the application.

[0209] Step 2:

[0210] The device sends the captured image data to the server. The input is the image data stored on the smartphone, and the output is the image data sent to the server via the Internet. The device sends the image data to the server using the HTTP protocol.

[0211] Step 3:

[0212] The server analyzes the received image data and recognizes the ingredients. The input is the image data sent to the server, and the output is the recognized ingredient information (e.g., "tomato"). The server inputs the image data into a machine learning model built using TensorFlow or PyTorch to recognize the ingredients. Specifically, a deep learning algorithm analyzes pixel information in the image and identifies the ingredients.

[0213] Step 4:

[0214] The server returns the recognized ingredient information to the user's device. The input is the recognized ingredient information, and the output is the ingredient information sent to the user's device. The server again uses the HTTP protocol to send the recognition result to the device, and information such as "tomato" is displayed to the user.

[0215] Step 5:

[0216] The user requests a recipe to be generated based on the provided ingredient information. The input is the recognized ingredient information and the user's request, and the output is a recipe generation request. The user uses the app interface to make a request such as "Please generate a recipe using tomatoes."

[0217] Step 6:

[0218] The server instructs the generative AI model to generate a recipe. The input is the user's request and recognized ingredient information, and the output is the generated recipe. The server uses a natural language generation model (generative AI model) to generate a recipe based on the ingredients. Specifically, the prompt sentence "Please generate a recipe using tomatoes" is input into the model, and the generated text data is output as the recipe.

[0219] Step 7:

[0220] The server sends the generated recipe to the user's terminal. The input is the generated recipe data, and the output is the recipe information sent to the user's terminal. The server again uses the HTTP protocol to send the generated recipe to the user's terminal, allowing the user to view the recipe.

[0221] Step 8:

[0222] The server shows the location of related ingredients based on the ingredient information photographed in the store. The input is ingredient information photographed in the store, and the output is location information of related ingredients. The server connects to the store's inventory database to identify the location of available ingredients and provide it to the user.

[0223] Step 9:

[0224] The server works with store inventory information to generate recipes based on available ingredients. The input is store inventory information and recognized ingredient information, and the output is a recipe. The server generates recipes using a generative AI model while taking inventory information into account, and provides them to the user.

[0225] Through the above processing steps, the user can take a photo of the ingredients they want to purchase in the store and, based on that information, efficiently obtain the location of the ingredients and cooking recipes.

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[0228] Ingredient recognition processing

[0229] 1. Acquire images of ingredients:

[0230] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[0231] 2. Send image data to the server:

[0232] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[0233] 3. Recognizing ingredients using image recognition models:

[0234] The server inputs the received image data into an image recognition algorithm (e.g., a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[0235] 4. Providing ingredients information to users:

[0236] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[0237] Emotion recognition processing

[0238] 1. Get user sentiment:

[0239] The user uses the device's camera to capture their facial expressions, and the device sends this image data to the emotion engine.

[0240] 2. Analyze emotion data:

[0241] The emotion engine in the device analyzes the image data and recognizes the user's emotion. For example, if the user is smiling, the emotion information of "happiness" is acquired.

[0242] 3. Send emotion information to the server:

[0243] The device sends the acquired emotion information to the server via a network, such as the HTTP protocol.

[0244] Handling recipe generation

[0245] 1. Request to generate a recipe:

[0246] The user uses the device interface to instruct "Create a recipe using tomatoes." At this time, the acquired emotional information is also sent to the server.

[0247] 2. Send the request to the server:

[0248] The terminal sends a request including ingredient information (tomato) and emotion information (e.g., happiness) to the server.

[0249] 3. Give instructions to the generative AI:

[0250] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[0251] 4. Generate the recipe:

[0252] Generative AI generates recipes based on instructions. For example, if the user's emotion is "happy," the generated recipe might include brightly colored ingredients and easy-to-prepare recipes. The generated recipe might include:

[0253] Tomato and mozzarella salad

[0254] material:

[0255] Tomatoes: 2

[0256] Mozzarella cheese: 100g

[0257] Basil leaves: appropriate amount

[0258] Olive oil: 2 tablespoons

[0259] Salt: 1 / 2 teaspoon

[0260] Black pepper: appropriate amount

[0261] Instructions:

[0262] 1. Slice the tomatoes.

[0263] 2. Slice the mozzarella cheese.

[0264] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0265] 4. Garnish with basil leaves.

[0266] 5. Drizzle with olive oil and season with salt and black pepper.

[0267] 5. Providing recipe data to users:

[0268] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[0269] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[0270] The processing flow will be explained below.

[0271] Ingredient recognition processing

[0272] Step 1:

[0273] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato in a refrigerator.

[0274] Step 2:

[0275] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[0276] Step 3:

[0277] The server temporarily stores the received image data, which is then prepared for analysis.

[0278] Step 4:

[0279] The server feeds the image data into an image recognition model (e.g., a deep learning algorithm) that has been trained to identify ingredients.

[0280] Step 5:

[0281] The server analyzes the output of the image recognition model and recognizes ingredients. For example, the model identifies "tomato" from an image.

[0282] Step 6:

[0283] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[0284] Step 7:

[0285] The device analyzes the received ingredient information and displays "Recognized ingredient: Tomato" to the user.

[0286] Emotion engine processing

[0287] Step 8:

[0288] The user uses the camera function of the device to capture a picture of their own facial expression. For example, the user uses the camera of a smartphone to take a picture of their own face.

[0289] Step 9:

[0290] The device sends the captured facial image data to the emotion engine, which is installed on the device.

[0291] Step 10:

[0292] The emotion engine analyzes the image data and recognizes the user's emotions. For example, it obtains emotional information such as "happiness" from the user's facial expression.

[0293] Step 11:

[0294] The device transmits the acquired emotional information to the server, where it is encoded and the necessary security measures are applied.

[0295] Handling recipe generation

[0296] Step 12:

[0297] The user uses the device interface to instruct "Create a recipe using tomatoes." This instruction also includes the user's emotional information.

[0298] Step 13:

[0299] The terminal sends a request including ingredient information (tomato) and emotion information (happiness) to the server.

[0300] Step 14:

[0301] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[0302] Step 15:

[0303] The generative AI uses an internal natural language generation algorithm to generate recipes using tomatoes. For example, if the user's emotion is "happy," the generated recipes will have brightly colored ingredients and be easy to prepare.

[0304] Step 16:

[0305] The server formats the generated recipe into JSON format and sends it to the device.

[0306] Step 17:

[0307] The device analyzes the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in an appropriate layout. For example, "Tomato and Mozzarella Salad" is displayed in detail.

[0308] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[0309] Example 2

[0310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] Conventional cooking assistance systems provide a function to recognize ingredients available to the user and suggest recipes based on them. However, these systems cannot take into account personal circumstances such as the user's emotions and physical condition, making it difficult to suggest personalized meals. Therefore, there is a growing need for a system that provides more personalized recipes that are tailored to the user's emotions and circumstances.

[0312] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring the user's emotion, means for transmitting the acquired emotion information to the server, means for generating a recipe based on the provided ingredient information and emotion information, and means for providing the generated recipe to the user. This enables personalized recipe suggestions that take into account the user's emotion and situation in addition to ingredient information.

[0313] "Means for acquiring images" is a function that allows a user to take an image of an ingredient using the camera of the terminal and acquire that data.

[0314] A "server" is a computer system that receives and processes data sent from a terminal via a network.

[0315] The "analysis means" refers to an algorithm and process for analyzing the received image data of ingredients using a machine learning model and recognizing them as specific ingredients.

[0316] The "means for providing ingredient information" is a function that returns information about the recognized ingredients to the terminal and visually presents it to the user.

[0317] The "means for acquiring emotions" is a function for analyzing an image of a user's facial expression and recognizing the user's emotions.

[0318] The "means for transmitting emotion information to a server" is a communication function for transmitting acquired emotion data from the terminal to a server.

[0319] The "means for generating a recipe" is a process of creating a recipe using a generative AI model based on the acquired ingredient information and emotion information.

[0320] The "means for providing the generated recipe" is a function for transmitting the generated recipe from the server to the terminal and displaying it to the user.

[0321] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[0322] Ingredient recognition processing

[0323] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they have taken out of the refrigerator. This photo data is sent by the device to a server using the HTTP protocol. The server inputs the received image data into an image recognition model using deep learning (for example, TensorFlow or PyTorch) for analysis. The model identifies the ingredient from the image and recognizes it as a tomato. The server returns the recognized ingredient information to the device, and the user sees a message on their device screen saying "Recognized ingredient: Tomato."

[0324] Emotion recognition processing

[0325] The user takes a photo of their face using the device's camera function. The device then sends this image data to the built-in emotion engine. The emotion engine analyzes the facial expression and recognizes the user's emotion. For example, if the user is smiling, the emotion engine identifies the emotion information as "happiness." This emotion information is then sent by the device to the server. The HTTP protocol is used for transmission.

[0326] Handling recipe generation

[0327] The user uses the device's application interface to instruct "Generate a recipe using tomatoes." In response to this instruction, the acquired ingredient information (tomatoes) and emotion information (happiness) are sent together to the server. The device sends the transmission data to the server via the HTTP protocol. The server then sends a prompt to a natural language generation model (e.g., GPT-3 (registered trademark)) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotion."

[0328] Generative AI generates recipes based on prompts. For example, if the user's emotion is "happy," the generated recipes will feature brightly colored ingredients and easy-to-prepare recipes. The generated recipes might include:

[0329] Tomato and mozzarella salad

[0330] material:

[0331] Tomatoes: 2

[0332] Mozzarella cheese: 100g

[0333] Basil leaves: appropriate amount

[0334] Olive oil: 2 tablespoons

[0335] Salt: 1 / 2 teaspoon

[0336] Black pepper: appropriate amount

[0337] Instructions:

[0338] 1. Slice the tomatoes.

[0339] 2. Slice the mozzarella cheese.

[0340] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0341] 4. Garnish with basil leaves.

[0342] 5. Drizzle with olive oil and season with salt and black pepper.

[0343] The server returns the generated recipe data to the device. The device then displays the recipe to the user and provides detailed cooking instructions. Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it becomes possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[0344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0345] Step 1:

[0346] A user takes a photo of an ingredient using a device such as a smartphone or tablet.

[0347] Input: Photo data of ingredients taken by the user

[0348] Specific operation: The user opens the device's camera app and takes a photo of an ingredient such as a tomato.

[0349] Output: Image data stored in the device's storage

[0350] Step 2:

[0351] The terminal transmits the captured image data to the server.

[0352] Input: Image data stored in the device's storage

[0353] Specific operation: The terminal sends image data to the server via an Internet connection using the HTTP protocol.

[0354] Output: Image data received by the server

[0355] Step 3:

[0356] The image data received by the server is input into a deep learning model to recognize ingredients.

[0357] Input: Image data received by the server

[0358] How it works: The server uses deep learning frameworks such as TensorFlow and PyTorch to apply image recognition algorithms and identify ingredients in the image as tomatoes.

[0359] Output: Recognized ingredient information (e.g., tomato)

[0360] Step 4:

[0361] The server returns the recognized ingredient information to the terminal.

[0362] Input: Recognized ingredient information

[0363] Specific operation: The server sends the recognized ingredient information (e.g., "tomato") to the terminal via the HTTP protocol.

[0364] Output: Ingredient information displayed on the device (e.g. "Recognized ingredient: Tomato")

[0365] Step 5:

[0366] The user takes a photo of their face using the camera function of the device.

[0367] Input: A photo of the user's face

[0368] Specific operation: The user opens the device's camera app and takes a photo of their face.

[0369] Output: Facial photo data stored in the device storage

[0370] Step 6:

[0371] The device sends the captured facial photo to the emotion engine for analysis.

[0372] Input: Facial photo data stored on the device

[0373] How it works: The device's built-in emotion engine analyzes the face photo and recognizes the user's emotion based on their facial expression. For example, if the user is smiling, the emotion information of "happiness" is obtained.

[0374] Output: Recognized emotion information (e.g., "happiness")

[0375] Step 7:

[0376] The terminal transmits the acquired emotion information to the server.

[0377] Input: Recognized emotion information

[0378] Specific operation: The device sends emotional information (e.g., "happiness") to the server via the HTTP protocol.

[0379] Output: Emotion information received by the server

[0380] Step 8:

[0381] A user requests recipe generation using the terminal's application interface.

[0382] Input: User's creation request (e.g. "Create a recipe using tomatoes")

[0383] Specific operation: The user enters the instruction "Create a recipe using tomatoes" on the device's application screen and presses the send button.

[0384] Output: The generated request sent to the server (e.g., containing the information "tomato" and "happiness")

[0385] Step 9:

[0386] The server generates a recipe based on a natural language generation model.

[0387] Input: The generation request received by the server (ingredients and emotion information)

[0388] Specific operation: The server sends a prompt to the generative AI model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotions," and instructs it to generate the recipe.

[0389] Output: The generated recipe

[0390] Step 10:

[0391] The server returns the generated recipe to the device.

[0392] Input: Generated recipe

[0393] Specific operation: The server sends the generated recipe to the terminal via the HTTP protocol.

[0394] Output: The generated recipe (e.g., a tomato and mozzarella salad recipe) displayed in the terminal.

[0395] The above is the specific flow of the system's program processing. Through this series of processes, users can easily obtain personalized recipes based on ingredients and emotions.

[0396] (Application example 2)

[0397] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] In many modern households, daily ingredient management and meal planning are major issues that require time and effort. Furthermore, personalized recipe suggestions tailored to the user's mood can be difficult to implement on one's own, and finding the right recipe and the necessary ingredients can be a challenge. Furthermore, cooking decisions can be difficult when ingredients are not on hand. There is a need to solve these challenges and make users' cooking experiences richer and more convenient.

[0399] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by the user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring and analyzing the user's emotion, means for generating a recipe based on the provided ingredient information and the acquired emotion information, means for providing the generated recipe to the user, and means for ordering ingredients based on the ingredient information. This allows the user to easily acquire an optimal recipe based on the ingredients they have on hand and their emotion, and also enables them to quickly order the necessary ingredients.

[0400] "Ingredient image" is image data of food that the user has taken a picture of.

[0401] A "server" is a computer system that sends and receives data over a network and provides processing and storage functions.

[0402] "Analyzing images" refers to identifying the contents of food ingredients from the photographed images and processing the data to identify the ingredients.

[0403] The "means for recognizing ingredients" refers to a method or device for identifying the type of ingredient using image analysis technology.

[0404] "Ingredient information" refers to data related to the recognized ingredient, specifically information such as the name and type of ingredient.

[0405] The "means for providing to the user" is a method or device for displaying the obtained information to the user.

[0406] "Acquiring emotions" refers to collecting data to identify the user's emotional state from facial expression data such as face and voice.

[0407] A "means for analyzing emotions" is a method or device for analyzing acquired emotion data to identify the user's emotional state.

[0408] The "means for generating a recipe" is a method or device for creating cooking instructions and ingredients based on the recognized ingredient information and emotion information.

[0409] A "means for ordering ingredients" is a system or device for ordering the ingredients you need online.

[0410] The system of this invention recognizes images of ingredients taken by users and acquires and analyzes the user's emotions to provide optimal recipes for individual users and enable them to order those ingredients. The detailed configuration and processing procedures of this system are described below.

[0411] Hardware and software used

[0412] In the present invention, the following hardware and software are used.

[0413] Hardware:

[0414] Smartphones with camera functions (e.g., iPhone (registered trademark) 12, Samsung Galaxy series)

[0415] Data processing server (e.g., AWS (registered trademark) EC2 instance)

[0416] software:

[0417] Image recognition model: A deep learning model for recognizing ingredients (e.g., using PyTorch)

[0418] Emotion Recognition Engine: A machine learning model to identify emotions from user facial expressions (e.g., using Huggingface Transformers)

[0419] Recipe generation engine: A natural language generation model (e.g., OpenAI® GPT-3) to generate recipes.

[0420] Network protocol: Data communication uses HTTP and REST API

[0421] System processing overview

[0422] 1. How to get images of ingredients:

[0423] The user uses the smartphone camera to take a photo of an ingredient (e.g., a tomato) in their hand.

[0424] 2. How to send images to the server:

[0425] The captured image data is sent from the smartphone to a server via the Internet using the HTTP protocol.

[0426] 3. Methods for recognizing ingredients by analyzing images:

[0427] The server inputs the received image data into a deep learning-based image recognition model to identify the food item, for example, an image of a tomato.

[0428] 4. Means for providing recognized ingredient information to the user:

[0429] The server sends the recognized ingredient information (e.g., "tomato") back to the user's device, and the user's screen displays "Recognized ingredient: tomato."

[0430] 5. Means of acquiring and analyzing user emotions:

[0431] A user takes a picture of their face using a smartphone camera. The captured image data is sent to an emotion recognition engine for analysis. For example, if the user is smiling, emotion information of "happiness" is acquired.

[0432] 6. A method for generating recipes based on provided ingredient information and acquired emotion information:

[0433] The server creates a recipe generation prompt for the generative AI model based on the recognized ingredient information and emotion information. An example of a prompt sentence is as follows:

[0434] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[0435] A generative AI model (e.g., GPT-3) generates a recipe based on this prompt. For example, the generated recipe might look like this:

[0436] Tomato and mozzarella salad

[0437] material:

[0438] Tomatoes: 2

[0439] Mozzarella cheese: 100g

[0440] Basil leaves: appropriate amount

[0441] Olive oil: 2 tablespoons

[0442] Salt: 1 / 2 teaspoon

[0443] Black pepper: appropriate amount

[0444] Instructions:

[0445] 1. Slice the tomatoes.

[0446] 2. Slice the mozzarella cheese.

[0447] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0448] 4. Garnish with basil leaves.

[0449] 5. Drizzle with olive oil and season with salt and black pepper.

[0450] 7. How to provide the generated recipe to the user:

[0451] The generated recipe is sent back from the server to the user's terminal and displayed so that the user can view it.

[0452] 8. How to order ingredients based on ingredient information:

[0453] Based on the information about the ingredients needed, the user orders ingredients through the food delivery service's API. The server sends this information to the food delivery system, which processes the order.

[0454] This allows users to easily obtain recipes based on the ingredients they have on hand and their current mood, and quickly order the ingredients they need.

[0455] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0456] Step 1:

[0457] The user uses the smartphone camera to take a photo of the ingredients they have on hand, and the captured image data is then stored on the device.

[0458] Input: Ingredient image

[0459] Output: Image data

[0460] Step 2:

[0461] The terminal sends the acquired image data to the server using the HTTP protocol.

[0462] Input: Image data

[0463] Output: Image data transferred to the server

[0464] Step 3:

[0465] The server inputs the received image data into a deep learning-based image recognition model, which analyzes the image and recognizes the ingredients.

[0466] Input: Image data

[0467] Output: Recognized ingredient information (e.g., "tomato")

[0468] Step 4:

[0469] The server returns the recognized ingredient information to the user's terminal, which displays this information on the user's screen.

[0470] Input: Recognized ingredient information

[0471] Output: The device screen will display "Recognized ingredient: Tomato"

[0472] Step 5:

[0473] The user takes a picture of their face using the smartphone camera and the image data is acquired by the terminal.

[0474] Input: An image of the user's face

[0475] Output: Face image data

[0476] Step 6:

[0477] The device sends the acquired facial image data to the server using the HTTP protocol.

[0478] Input: Facial image data

[0479] Output: Facial image data transferred to the server

[0480] Step 7:

[0481] The server inputs the received facial image data into an emotion recognition engine, which analyzes the image and identifies the user's emotion.

[0482] Input: Facial image data

[0483] Output: Recognized emotion information (e.g., "happiness")

[0484] Step 8:

[0485] The server creates a prompt for recipe generation based on the recognized ingredients and emotion information. An example prompt is as follows:

[0486] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[0487] Input: Recognized ingredient information, emotion information

[0488] Output: prompt statement

[0489] Step 9:

[0490] The server inputs prompts into the generative AI model to generate recipes, which include specific ingredients and steps.

[0491] Input: prompt statement

[0492] Output: The generated recipe (e.g. "Tomato and Mozzarella Salad")

[0493] Step 10:

[0494] The server returns the generated recipe to the user's terminal, which displays the recipe on the user's screen.

[0495] Input: Generated recipe

[0496] Output: The recipe is displayed on the terminal screen.

[0497] Step 11:

[0498] The server then sends an order for ingredients to the food delivery service based on the recognized ingredient information. The order calls an API using the HTTP protocol.

[0499] Input: Recognized ingredient information

[0500] Output: Place an order to a food delivery service

[0501] This allows users to get recipes based on the ingredients they have on hand and their current mood, and then quickly order the ingredients they need.

[0502] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0503] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0504] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0505] [Second embodiment]

[0506] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0507] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0508] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0509] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0510] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0511] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0512] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0513] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0514] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0515] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0516] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0517] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0518] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0519] Ingredient recognition processing

[0520] 1. Acquire images of ingredients:

[0521] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[0522] 2. Send image data to the server:

[0523] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[0524] 3. Recognizing ingredients using image recognition models:

[0525] The server inputs the received image data into an image recognition algorithm (for example, a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[0526] 4. Providing ingredients information to users:

[0527] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[0528] Handling recipe generation

[0529] 1. Request to generate a recipe:

[0530] The user uses the device interface and instructs the device to "create a recipe using tomatoes."

[0531] 2. Send the request to the server:

[0532] The terminal sends a request including ingredient information (tomato) to the server.

[0533] 3. Give instructions to the generative AI:

[0534] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes."

[0535] 4. Generate the recipe:

[0536] Generative AI generates recipes based on instructions. For example, it generates the following recipes:

[0537] Tomato and mozzarella salad

[0538] material:

[0539] Tomatoes: 2

[0540] Mozzarella cheese: 100g

[0541] Basil leaves: appropriate amount

[0542] Olive oil: 2 tablespoons

[0543] Salt: 1 / 2 teaspoon

[0544] Black pepper: appropriate amount

[0545] Instructions:

[0546] 1. Slice the tomatoes.

[0547] 2. Slice the mozzarella cheese.

[0548] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0549] 4. Garnish with basil leaves.

[0550] 5. Drizzle with olive oil and season with salt and black pepper.

[0551] 5. Providing recipe data to users:

[0552] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[0553] In this way, the system of the present invention provides users with the ability to easily recognize ingredients and create recipes, helping them make the most of the ingredients they have on hand. This allows users to make their daily cooking activities more efficient and creative. This system can reduce ingredient waste and enrich users' cooking repertoire.

[0554] The processing flow will be explained below.

[0555] Step 1:

[0556] A user takes a photo of an ingredient using a device such as a smartphone or tablet, and the device uses its camera function to acquire image data of the ingredient.

[0557] Step 2:

[0558] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[0559] Step 3:

[0560] The server temporarily stores the received image data, which is then prepared for analysis.

[0561] Step 4:

[0562] The server feeds the image data into an image recognition model, which uses deep learning algorithms to identify ingredients in the image.

[0563] Step 5:

[0564] The server analyzes the output of the image recognition model and recognizes the food. For example, if the model identifies a food as a "tomato," the server will recognize the food as a tomato.

[0565] Step 6:

[0566] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[0567] Step 7:

[0568] The device parses the received ingredient information and displays "Recognized ingredient: Tomato" to the user. Based on this information, the user prepares for the next operation.

[0569] Step 8:

[0570] The user uses the terminal interface to say "generate a recipe using tomatoes," which is treated as an appropriate command within the application.

[0571] Step 9:

[0572] The device sends a request to the server to "create a recipe using tomatoes." This request includes ingredient information (tomatoes).

[0573] Step 10:

[0574] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes."

[0575] Step 11:

[0576] The generative AI uses an internal natural language generation algorithm to generate tomato-based recipes, including details such as the title, ingredients, and cooking instructions.

[0577] Step 12:

[0578] The server formats the generated recipe into JSON format and sends it to the device.

[0579] Step 13:

[0580] The device parses the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in a layout that the user can view. For example, a detailed recipe for "Tomato and Mozzarella Salad" is displayed.

[0581] Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In today's world, users need to efficiently utilize the ingredients they use on a daily basis and quickly obtain a variety of recipes based on those ingredients. However, users often have to spend a lot of time and effort to find appropriate recipes for the ingredients they have on hand. Furthermore, existing recipe suggestion systems often struggle to flexibly accommodate the specific ingredients a user has and are unable to meet individual needs. Therefore, an effective means is needed to reduce ingredient waste and support users' cooking activities.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image and recognizing ingredients in the server, means for providing the recognized ingredient information to the user, means for generating a prompt sentence based on the provided ingredient information and generating a recipe using a generative AI model, and means for providing the generated recipe to the user, thereby enabling the user to quickly obtain a variety of specific recipes based on ingredients on hand.

[0587] "User" refers to an individual who utilizes the system to request ingredient recognition and recipe generation.

[0588] "Terminal" refers to a hardware device that a user uses to operate the system, and specifically includes a smartphone or tablet.

[0589] "Server" refers to a remote computer system that analyzes image data and generates recipes.

[0590] "Means for acquiring images" refers to the function that allows a user to take a photo of an ingredient using a terminal.

[0591] "Means for transmitting to a server" refers to the function of transmitting image data captured from a terminal to a server via the Internet.

[0592] "Means for analyzing images and recognizing ingredients" refers to the algorithms and processes for analyzing image data received by the server and identifying the corresponding ingredients.

[0593] "Means for providing ingredient information" refers to the function of the server returning the ingredient information identified by the server to the terminal and notifying the user.

[0594] "Means for generating a prompt sentence" refers to a function that generates an input sentence required for recipe generation based on the identified ingredient information.

[0595] "Means for generating a recipe using a generative AI model" refers to a function that uses a generative AI model (e.g., a natural language generation model) to generate a specific recipe based on a prompt sentence.

[0596] The "means for providing the generated recipe to the user" refers to a function for providing the generated recipe to the terminal and displaying it to the user.

[0597] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0598] First, this system uses a device (such as a smartphone or tablet) for users to take photos of ingredients. The image data of the ingredients taken by the user is sent from the device and stored on a server. The device then transmits this data via the HTTP protocol.

[0599] The server then uses a machine learning model (e.g., an image recognition model using deep learning) to analyze the received image data. This model often uses algorithms such as ResNet or YOLO. The server uses this model to analyze the image data and identify the ingredients. For example, the server may identify the ingredient in the image taken by the user as a "tomato."

[0600] The server sends the analysis results back to the device to provide to the user, and the device displays this information to the user, such as "Recognized ingredient: Tomato."

[0601] Next, the user makes a request through the device interface to generate a recipe based on specific ingredients, for example, "Generate a recipe using tomatoes." The device sends this request to the server, which generates a prompt, such as "Generate a simple and delicious recipe using tomatoes."

[0602] The server then uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. The model takes the prompt as input and generates a detailed recipe, such as a "Tomato and Mozzarella Salad" with specific, detailed instructions:

[0603] Tomato and mozzarella salad

[0604] material:

[0605] Tomatoes: 2

[0606] Mozzarella cheese: 100g

[0607] Basil leaves: appropriate amount

[0608] Olive oil: 2 tablespoons

[0609] Salt: 1 / 2 teaspoon

[0610] Black pepper: appropriate amount

[0611] Instructions:

[0612] 1. Slice the tomatoes.

[0613] 2. Slice the mozzarella cheese.

[0614] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0615] 4. Garnish with basil leaves.

[0616] 5. Drizzle with olive oil and season with salt and black pepper.

[0617] The server sends the generated recipe back to the device, and the device provides this recipe information to the user, who can check the recipe on the device screen and use it in their cooking activities.

[0618] In this way, the system of the present invention supports the user's cooking activities by recognizing ingredients on hand and providing specific and detailed recipes based on those ingredients, allowing the user to cook efficiently and creatively without wasting ingredients.

[0619] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0620] Step 1: Get an image of the ingredient

[0621] A user takes a photo of an ingredient using a device such as a smartphone or tablet. Specifically, the user opens a camera app and takes a clear image of the ingredient (e.g., a tomato). The input data is an image file of the tomato, and this image is saved on the device.

[0622] Step 2: Send image data to the server

[0623] The image data captured by the device is sent to a server via the Internet. Specifically, an HTTP POST request is used to send the image file to a specific API endpoint on the server. The input data is the saved image file of the tomato, and the output data is the image file passed to the server.

[0624] Step 3: Analyze the image and recognize the ingredients

[0625] The server analyzes the received image data using a deep learning model. Specifically, it runs machine learning algorithms such as ResNet and YOLO to recognize ingredients in the image. The input data is the image file of the tomato that was sent, and the output data is the ingredient information recognized as "tomato."

[0626] Step 4: Provide ingredients information to users

[0627] The server returns the recognition results to the device, which then displays this information to the user. Specifically, the server returns the information "Ingredient: Tomato" in JSON format, and the device analyzes this information and displays "Recognized ingredient: Tomato" on the screen. The input data is the recognized ingredient information, and the output data is the ingredient information displayed to the user.

[0628] Step 5: Request Recipe Generation

[0629] The user issues a command through the device interface to "create a recipe using tomatoes." Specifically, the user taps a button in the application to send the request. The input data is the user's command, and the output data is the creation request.

[0630] Step 6: Send the request to the server

[0631] The terminal sends a user request to the server. This request contains a specific instruction: "Generate a recipe using tomatoes." The input data is the generation request obtained from the user, and the output data is the request sent to the server.

[0632] Step 7: Generate the prompt

[0633] The server generates the prompt required to generate a recipe based on the provided ingredient information. For example, a prompt in the format "Please generate a simple and delicious recipe using tomatoes" is created. The input data is the ingredient information and the generation request, and the output data is the generated prompt.

[0634] Step 8: Generate recipes using generative AI models

[0635] The server uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. Specifically, the prompt is input into the model, and a detailed recipe (e.g., "Tomato and mozzarella salad") is generated. The input data is the prompt, and the output data is the generated recipe.

[0636] Step 9: Provide the generated recipe to the user

[0637] The server returns the generated recipe to the device, and the device provides this recipe information to the user. Specifically, the server returns detailed recipe information in JSON format, and the device parses this information and displays it on the screen. The input data is the generated recipe, and the output data is the recipe information displayed to the user.

[0638] (Application example 1)

[0639] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] Today's consumers have a growing need for easy access to recipe information that effectively utilizes ingredients purchased at supermarkets. However, conventional methods are limited in suggesting recipes based on specific ingredients, and lack means to provide information on other related ingredients available in the store and their locations. This makes it difficult for consumers to efficiently purchase the ingredients they need and smoothly carry out cooking activities.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0642] In this invention, the server includes means for acquiring images of ingredients photographed by a user, means for transmitting the acquired images to the server, means for analyzing the images to recognize ingredients, means for providing the user with the recognized ingredient information, means for generating a recipe based on the provided ingredient information, means for providing the generated recipe to the user, means for indicating the location of related ingredients based on ingredient information photographed in the store, and means for generating a recipe based on available ingredients in conjunction with store inventory information. This allows consumers to obtain location information for necessary ingredients based on information on ingredients they wish to purchase in the store, and easily obtain recipes that make the most of the ingredients they have purchased.

[0643] A "user" is a consumer who uses the system to take photos of ingredients and obtain recipe information.

[0644] "Ingredients" are food ingredients that are photographed by the user and are recognized by the system.

[0645] An "image" refers to visual information obtained by a user photographing an ingredient.

[0646] A "server" is a computer device that receives images sent by users and performs analysis and data processing.

[0647] "Means" refers to a method or apparatus for performing a particular function or process.

[0648] "Analysis" refers to the process in which the server analyzes the image data received and recognizes the ingredients.

[0649] "Recognition" is the process of identifying specific ingredients from the image analyzed by the server.

[0650] A "recipe" is a list of cooking instructions and necessary ingredients generated based on recognized ingredient information.

[0651] "Providing" refers to sending information from the server to the user and the user receiving that information.

[0652] A "store" refers to a physical sales location such as a supermarket where users purchase ingredients.

[0653] "Location of related ingredients" is location information of the sales area in the store that sells other ingredients related to the ingredient photographed by the user.

[0654] "Store inventory information" is data on the inventory status of ingredients currently being sold in the store.

[0655] "Integration" refers to the sharing of information and collaboration between different systems and data sources.

[0656] "Available ingredients" refers to food ingredients that are currently available in the store, i.e., available for purchase.

[0657] A "machine learning model" refers to an algorithm that automatically learns from large amounts of data and performs data analysis and predictions.

[0658] A "natural language generation model" is a model that generates linguistically natural sentences based on text data.

[0659] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0660] System configuration

[0661] 1. User Device:

[0662] A smartphone, tablet, etc. is a device that allows users to take photos of ingredients and request the system to create a recipe.

[0663] 2. Server:

[0664] A high-performance computer device, the server processes image data sent by users, recognizes ingredients, and generates recipes.

[0665] 3. Machine learning models:

[0666] Image recognition algorithms built using deep learning frameworks such as TensorFlow and PyTorch are used to identify ingredients from captured images.

[0667] 4. Natural Language Generation Models:

[0668] It uses generative AI (Natural Language Processing model) to generate recipes based on ingredients.

[0669] System Operation Overview

[0670] 1. Ingredient Recognition Process:

[0671] The user takes a photo of the food using their smartphone and sends the image data to the server.

[0672] The server inputs the received image data into an image recognition algorithm to recognize the ingredients.

[0673] The server returns the recognized ingredient information to the user terminal and displays it to the user.

[0674] 2. Recipe generation process:

[0675] The user requests recipe creation based on the provided ingredient information.

[0676] The server issues instructions to the generative AI model to generate the recipe.

[0677] The server transmits the generated recipe to the user terminal and displays it to the user.

[0678] 3. Providing store information:

[0679] Based on the food information photographed in the store, the location of related ingredients is displayed.

[0680] It works with store inventory information to generate recipes based on available ingredients.

[0681] Program processing flow

[0682] The server first receives image data sent from the user's device. Next, this image data is input into a machine learning model (using TensorFlow or PyTorch) to recognize ingredients. The recognition result (e.g., "tomato") is sent back to the user's device. The user then requests recipe generation based on the provided information. The server generates a recipe by inputting a prompt statement, such as "Please generate a recipe using tomatoes," into the generative AI model. The generated recipe is sent to the user's device and provided to the user. It can also be linked to in-store inventory information to show the user where related ingredients can be purchased.

[0683] Specific examples

[0684] A user takes a picture of a tomato with their smartphone at a supermarket and uses the app to obtain ingredient information. The server receives the image and uses a machine learning model to recognize it as a "tomato." When the user then requests, "Please generate a recipe using tomatoes," the server uses a generative AI model to generate a recipe, such as "Tomato and Mozzarella Salad," and provides the user with such a recipe. Additionally, by providing information on the location and inventory of mozzarella cheese in the store, the server can efficiently purchase the ingredients they need.

[0685] In this way, it is possible to support the user's cooking activities, reduce food waste, and enrich the user's cooking repertoire.

[0686] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0687] Step 1:

[0688] The user takes a photo of an ingredient with their smartphone. The input is image data acquired from the smartphone camera. The user takes a photo of an ingredient (e.g., a tomato) and inputs the image data into the application.

[0689] Step 2:

[0690] The device sends the captured image data to the server. The input is the image data stored on the smartphone, and the output is the image data sent to the server via the Internet. The device sends the image data to the server using the HTTP protocol.

[0691] Step 3:

[0692] The server analyzes the received image data and recognizes the ingredients. The input is the image data sent to the server, and the output is the recognized ingredient information (e.g., "tomato"). The server inputs the image data into a machine learning model built using TensorFlow or PyTorch to recognize the ingredients. Specifically, a deep learning algorithm analyzes pixel information in the image and identifies the ingredients.

[0693] Step 4:

[0694] The server returns the recognized ingredient information to the user's device. The input is the recognized ingredient information, and the output is the ingredient information sent to the user's device. The server again uses the HTTP protocol to send the recognition result to the device, and information such as "tomato" is displayed to the user.

[0695] Step 5:

[0696] The user requests a recipe to be generated based on the provided ingredient information. The input is the recognized ingredient information and the user's request, and the output is a recipe generation request. The user uses the app interface to make a request such as "Please generate a recipe using tomatoes."

[0697] Step 6:

[0698] The server instructs the generative AI model to generate a recipe. The input is the user's request and recognized ingredient information, and the output is the generated recipe. The server uses a natural language generation model (generative AI model) to generate a recipe based on the ingredients. Specifically, the prompt sentence "Please generate a recipe using tomatoes" is input into the model, and the generated text data is output as the recipe.

[0699] Step 7:

[0700] The server sends the generated recipe to the user's terminal. The input is the generated recipe data, and the output is the recipe information sent to the user's terminal. The server again uses the HTTP protocol to send the generated recipe to the user's terminal, allowing the user to view the recipe.

[0701] Step 8:

[0702] The server shows the location of related ingredients based on the ingredient information photographed in the store. The input is ingredient information photographed in the store, and the output is location information of related ingredients. The server connects to the store's inventory database to identify the location of available ingredients and provide it to the user.

[0703] Step 9:

[0704] The server works with store inventory information to generate recipes based on available ingredients. The input is store inventory information and recognized ingredient information, and the output is a recipe. The server generates recipes using a generative AI model while taking inventory information into account, and provides them to the user.

[0705] Through the above processing steps, the user can take a photo of the ingredients they want to purchase in the store and, based on that information, efficiently obtain the location of the ingredients and cooking recipes.

[0706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0707] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[0708] Ingredient recognition processing

[0709] 1. Acquire images of ingredients:

[0710] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[0711] 2. Send image data to the server:

[0712] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[0713] 3. Recognizing ingredients using image recognition models:

[0714] The server inputs the received image data into an image recognition algorithm (e.g., a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[0715] 4. Providing ingredients information to users:

[0716] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[0717] Emotion recognition processing

[0718] 1. Get user sentiment:

[0719] The user uses the device's camera to capture their facial expressions, and the device sends this image data to the emotion engine.

[0720] 2. Analyze emotion data:

[0721] The emotion engine in the device analyzes the image data and recognizes the user's emotion. For example, if the user is smiling, the emotion information of "happiness" is acquired.

[0722] 3. Send emotion information to the server:

[0723] The device sends the acquired emotion information to the server via a network, such as the HTTP protocol.

[0724] Handling recipe generation

[0725] 1. Request to generate a recipe:

[0726] The user uses the device interface to instruct "Create a recipe using tomatoes." At this time, the acquired emotional information is also sent to the server.

[0727] 2. Send the request to the server:

[0728] The terminal sends a request including ingredient information (tomato) and emotion information (e.g., happiness) to the server.

[0729] 3. Give instructions to the generative AI:

[0730] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[0731] 4. Generate the recipe:

[0732] Generative AI generates recipes based on instructions. For example, if the user's emotion is "happy," the generated recipe might include brightly colored ingredients and easy-to-prepare recipes. The generated recipe might include:

[0733] Tomato and mozzarella salad

[0734] material:

[0735] Tomatoes: 2

[0736] Mozzarella cheese: 100g

[0737] Basil leaves: appropriate amount

[0738] Olive oil: 2 tablespoons

[0739] Salt: 1 / 2 teaspoon

[0740] Black pepper: appropriate amount

[0741] Instructions:

[0742] 1. Slice the tomatoes.

[0743] 2. Slice the mozzarella cheese.

[0744] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0745] 4. Garnish with basil leaves.

[0746] 5. Drizzle with olive oil and season with salt and black pepper.

[0747] 5. Providing recipe data to users:

[0748] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[0749] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[0750] The processing flow will be explained below.

[0751] Ingredient recognition processing

[0752] Step 1:

[0753] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato in a refrigerator.

[0754] Step 2:

[0755] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[0756] Step 3:

[0757] The server temporarily stores the received image data, which is then prepared for analysis.

[0758] Step 4:

[0759] The server feeds the image data into an image recognition model (e.g., a deep learning algorithm) that has been trained to identify ingredients.

[0760] Step 5:

[0761] The server analyzes the output of the image recognition model and recognizes ingredients. For example, the model identifies "tomato" from an image.

[0762] Step 6:

[0763] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[0764] Step 7:

[0765] The device analyzes the received ingredient information and displays "Recognized ingredient: Tomato" to the user.

[0766] Emotion engine processing

[0767] Step 8:

[0768] The user uses the camera function of the device to capture a picture of their own facial expression. For example, the user uses the camera of a smartphone to take a picture of their own face.

[0769] Step 9:

[0770] The device sends the captured facial image data to the emotion engine, which is installed on the device.

[0771] Step 10:

[0772] The emotion engine analyzes the image data and recognizes the user's emotions. For example, it obtains emotional information such as "happiness" from the user's facial expression.

[0773] Step 11:

[0774] The device transmits the acquired emotional information to the server, where it is encoded and the necessary security measures are applied.

[0775] Handling recipe generation

[0776] Step 12:

[0777] The user uses the device interface to instruct "Create a recipe using tomatoes." This instruction also includes the user's emotional information.

[0778] Step 13:

[0779] The terminal sends a request including ingredient information (tomato) and emotion information (happiness) to the server.

[0780] Step 14:

[0781] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[0782] Step 15:

[0783] The generative AI uses an internal natural language generation algorithm to generate recipes using tomatoes. For example, if the user's emotion is "happy," the generated recipes will have brightly colored ingredients and be easy to prepare.

[0784] Step 16:

[0785] The server formats the generated recipe into JSON format and sends it to the device.

[0786] Step 17:

[0787] The device analyzes the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in an appropriate layout. For example, "Tomato and Mozzarella Salad" is displayed in detail.

[0788] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[0789] Example 2

[0790] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0791] Conventional cooking assistance systems provide a function to recognize ingredients available to the user and suggest recipes based on them. However, these systems cannot take into account personal circumstances such as the user's emotions and physical condition, making it difficult to suggest personalized meals. Therefore, there is a growing need for a system that provides more personalized recipes that are tailored to the user's emotions and circumstances.

[0792] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring the user's emotion, means for transmitting the acquired emotion information to the server, means for generating a recipe based on the provided ingredient information and emotion information, and means for providing the generated recipe to the user. This enables personalized recipe suggestions that take into account the user's emotion and situation in addition to ingredient information.

[0793] "Means for acquiring images" is a function that allows a user to take an image of an ingredient using the camera of the terminal and acquire that data.

[0794] A "server" is a computer system that receives and processes data sent from a terminal via a network.

[0795] The "analysis means" refers to an algorithm and process for analyzing the received image data of ingredients using a machine learning model and recognizing them as specific ingredients.

[0796] The "means for providing ingredient information" is a function that returns information about the recognized ingredients to the terminal and visually presents it to the user.

[0797] The "means for acquiring emotions" is a function for analyzing an image of a user's facial expression and recognizing the user's emotions.

[0798] The "means for transmitting emotion information to a server" is a communication function for transmitting acquired emotion data from the terminal to a server.

[0799] The "means for generating a recipe" is a process of creating a recipe using a generative AI model based on the acquired ingredient information and emotion information.

[0800] The "means for providing the generated recipe" is a function for transmitting the generated recipe from the server to the terminal and displaying it to the user.

[0801] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[0802] Ingredient recognition processing

[0803] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they have taken out of the refrigerator. This photo data is sent by the device to a server using the HTTP protocol. The server inputs the received image data into an image recognition model using deep learning (for example, TensorFlow or PyTorch) for analysis. The model identifies the ingredient from the image and recognizes it as a tomato. The server returns the recognized ingredient information to the device, and the user sees a message on their device screen saying "Recognized ingredient: Tomato."

[0804] Emotion recognition processing

[0805] The user takes a photo of their face using the device's camera function. The device then sends this image data to the built-in emotion engine. The emotion engine analyzes the facial expression and recognizes the user's emotion. For example, if the user is smiling, the emotion engine identifies the emotion information as "happiness." This emotion information is then sent by the device to the server. The HTTP protocol is used for transmission.

[0806] Handling recipe generation

[0807] The user uses the device's application interface to instruct "Generate a recipe using tomatoes." In response to this instruction, the acquired ingredient information (tomatoes) and emotion information (happiness) are sent together to the server. The device sends the data to the server via the HTTP protocol. The server then sends a prompt to a natural language generation model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotion."

[0808] Generative AI generates recipes based on prompts. For example, if the user's emotion is "happy," the generated recipes will feature brightly colored ingredients and easy-to-prepare recipes. The generated recipes might include:

[0809] Tomato and mozzarella salad

[0810] material:

[0811] Tomatoes: 2

[0812] Mozzarella cheese: 100g

[0813] Basil leaves: appropriate amount

[0814] Olive oil: 2 tablespoons

[0815] Salt: 1 / 2 teaspoon

[0816] Black pepper: appropriate amount

[0817] Instructions:

[0818] 1. Slice the tomatoes.

[0819] 2. Slice the mozzarella cheese.

[0820] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0821] 4. Garnish with basil leaves.

[0822] 5. Drizzle with olive oil and season with salt and black pepper.

[0823] The server returns the generated recipe data to the device. The device then displays the recipe to the user and provides detailed cooking instructions. Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it becomes possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[0824] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0825] Step 1:

[0826] A user takes a photo of an ingredient using a device such as a smartphone or tablet.

[0827] Input: Photo data of ingredients taken by the user

[0828] Specific operation: The user opens the device's camera app and takes a photo of an ingredient such as a tomato.

[0829] Output: Image data stored in the device's storage

[0830] Step 2:

[0831] The terminal transmits the captured image data to the server.

[0832] Input: Image data stored in the device's storage

[0833] Specific operation: The terminal sends image data to the server via an Internet connection using the HTTP protocol.

[0834] Output: Image data received by the server

[0835] Step 3:

[0836] The image data received by the server is input into a deep learning model to recognize ingredients.

[0837] Input: Image data received by the server

[0838] How it works: The server uses deep learning frameworks such as TensorFlow and PyTorch to apply image recognition algorithms and identify ingredients in the image as tomatoes.

[0839] Output: Recognized ingredient information (e.g., tomato)

[0840] Step 4:

[0841] The server returns the recognized ingredient information to the terminal.

[0842] Input: Recognized ingredient information

[0843] Specific operation: The server sends the recognized ingredient information (e.g., "tomato") to the terminal via the HTTP protocol.

[0844] Output: Ingredient information displayed on the device (e.g. "Recognized ingredient: Tomato")

[0845] Step 5:

[0846] The user takes a photo of their face using the camera function of the device.

[0847] Input: A photo of the user's face

[0848] Specific operation: The user opens the device's camera app and takes a photo of their face.

[0849] Output: Facial photo data stored in the device storage

[0850] Step 6:

[0851] The device sends the captured facial photo to the emotion engine for analysis.

[0852] Input: Facial photo data stored on the device

[0853] How it works: The device's built-in emotion engine analyzes the face photo and recognizes the user's emotion based on their facial expression. For example, if the user is smiling, the emotion information of "happiness" is obtained.

[0854] Output: Recognized emotion information (e.g., "happiness")

[0855] Step 7:

[0856] The terminal transmits the acquired emotion information to the server.

[0857] Input: Recognized emotion information

[0858] Specific operation: The device sends emotional information (e.g., "happiness") to the server via the HTTP protocol.

[0859] Output: Emotion information received by the server

[0860] Step 8:

[0861] A user requests recipe generation using the terminal's application interface.

[0862] Input: User's creation request (e.g. "Create a recipe using tomatoes")

[0863] Specific operation: The user enters the instruction "Create a recipe using tomatoes" on the device's application screen and presses the send button.

[0864] Output: The generated request sent to the server (e.g., containing the information "tomato" and "happiness")

[0865] Step 9:

[0866] The server generates a recipe based on a natural language generation model.

[0867] Input: The generation request received by the server (ingredients and emotion information)

[0868] Specific operation: The server sends a prompt to the generative AI model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotions," and instructs it to generate the recipe.

[0869] Output: The generated recipe

[0870] Step 10:

[0871] The server returns the generated recipe to the device.

[0872] Input: Generated recipe

[0873] Specific operation: The server sends the generated recipe to the terminal via the HTTP protocol.

[0874] Output: The generated recipe (e.g., a tomato and mozzarella salad recipe) displayed in the terminal.

[0875] The above is the specific flow of the system's program processing. Through this series of processes, users can easily obtain personalized recipes based on ingredients and emotions.

[0876] (Application example 2)

[0877] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0878] In many modern households, daily ingredient management and meal planning are major issues that require time and effort. Furthermore, personalized recipe suggestions tailored to the user's mood can be difficult to implement on one's own, and finding the right recipe and the necessary ingredients can be a challenge. Furthermore, cooking decisions can be difficult when ingredients are not on hand. There is a need to solve these challenges and make users' cooking experiences richer and more convenient.

[0879] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by the user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring and analyzing the user's emotion, means for generating a recipe based on the provided ingredient information and the acquired emotion information, means for providing the generated recipe to the user, and means for ordering ingredients based on the ingredient information. This allows the user to easily acquire an optimal recipe based on the ingredients they have on hand and their emotion, and also enables them to quickly order the necessary ingredients.

[0880] "Ingredient image" is image data of food that the user has taken a picture of.

[0881] A "server" is a computer system that sends and receives data over a network and provides processing and storage functions.

[0882] "Analyzing images" refers to identifying the contents of food ingredients from the photographed images and processing the data to identify the ingredients.

[0883] The "means for recognizing ingredients" refers to a method or device for identifying the type of ingredient using image analysis technology.

[0884] "Ingredient information" refers to data related to the recognized ingredient, specifically information such as the name and type of ingredient.

[0885] The "means for providing to the user" is a method or device for displaying the obtained information to the user.

[0886] "Acquiring emotions" refers to collecting data to identify the user's emotional state from facial expression data such as face and voice.

[0887] A "means for analyzing emotions" is a method or device for analyzing acquired emotion data to identify the user's emotional state.

[0888] The "means for generating a recipe" is a method or device for creating cooking instructions and ingredients based on the recognized ingredient information and emotion information.

[0889] A "means for ordering ingredients" is a system or device for ordering the ingredients you need online.

[0890] The system of this invention recognizes images of ingredients taken by users and acquires and analyzes the user's emotions to provide optimal recipes for individual users and enable them to order those ingredients. The detailed configuration and processing procedures of this system are described below.

[0891] Hardware and software used

[0892] In the present invention, the following hardware and software are used.

[0893] Hardware:

[0894] Smartphones with camera functions (e.g., iPhone 12, Samsung Galaxy series)

[0895] Data processing server (e.g. AWS EC2 instance)

[0896] software:

[0897] Image recognition model: A deep learning model for recognizing ingredients (e.g., using PyTorch)

[0898] Emotion Recognition Engine: A machine learning model to identify emotions from user facial expressions (e.g., using Huggingface Transformers)

[0899] Recipe generation engine: A natural language generation model (e.g., OpenAI GPT-3) to generate recipes.

[0900] Network protocol: Data communication uses HTTP and REST API

[0901] System processing overview

[0902] 1. How to get images of ingredients:

[0903] The user uses the smartphone camera to take a photo of an ingredient (e.g., a tomato) in their hand.

[0904] 2. How to send images to the server:

[0905] The captured image data is sent from the smartphone to a server via the Internet using the HTTP protocol.

[0906] 3. Methods for recognizing ingredients by analyzing images:

[0907] The server inputs the received image data into a deep learning-based image recognition model to identify the food item, for example, an image of a tomato.

[0908] 4. Means for providing recognized ingredient information to the user:

[0909] The server sends the recognized ingredient information (e.g., "tomato") back to the user's device, and the user's screen displays "Recognized ingredient: tomato."

[0910] 5. Means of acquiring and analyzing user emotions:

[0911] A user takes a picture of their face using a smartphone camera. The captured image data is sent to an emotion recognition engine for analysis. For example, if the user is smiling, emotion information of "happiness" is acquired.

[0912] 6. A method for generating recipes based on provided ingredient information and acquired emotion information:

[0913] The server creates a recipe generation prompt for the generative AI model based on the recognized ingredient information and emotion information. An example of a prompt sentence is as follows:

[0914] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[0915] A generative AI model (e.g., GPT-3) generates a recipe based on this prompt. For example, the generated recipe might look like this:

[0916] Tomato and mozzarella salad

[0917] material:

[0918] Tomatoes: 2

[0919] Mozzarella cheese: 100g

[0920] Basil leaves: appropriate amount

[0921] Olive oil: 2 tablespoons

[0922] Salt: 1 / 2 teaspoon

[0923] Black pepper: appropriate amount

[0924] Instructions:

[0925] 1. Slice the tomatoes.

[0926] 2. Slice the mozzarella cheese.

[0927] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[0928] 4. Garnish with basil leaves.

[0929] 5. Drizzle with olive oil and season with salt and black pepper.

[0930] 7. How to provide the generated recipe to the user:

[0931] The generated recipe is sent back from the server to the user's terminal and displayed so that the user can view it.

[0932] 8. How to order ingredients based on ingredient information:

[0933] Based on the information about the ingredients needed, the user orders ingredients through the food delivery service's API. The server sends this information to the food delivery system, which processes the order.

[0934] This allows users to easily obtain recipes based on the ingredients they have on hand and their current mood, and quickly order the ingredients they need.

[0935] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0936] Step 1:

[0937] The user uses the smartphone camera to take a photo of the ingredients they have on hand, and the captured image data is then stored on the device.

[0938] Input: Ingredient image

[0939] Output: Image data

[0940] Step 2:

[0941] The terminal sends the acquired image data to the server using the HTTP protocol.

[0942] Input: Image data

[0943] Output: Image data transferred to the server

[0944] Step 3:

[0945] The server inputs the received image data into a deep learning-based image recognition model, which analyzes the image and recognizes the ingredients.

[0946] Input: Image data

[0947] Output: Recognized ingredient information (e.g., "tomato")

[0948] Step 4:

[0949] The server returns the recognized ingredient information to the user's terminal, which displays this information on the user's screen.

[0950] Input: Recognized ingredient information

[0951] Output: The device screen will display "Recognized ingredient: Tomato"

[0952] Step 5:

[0953] The user takes a picture of their face using the smartphone camera and the image data is acquired by the terminal.

[0954] Input: An image of the user's face

[0955] Output: Face image data

[0956] Step 6:

[0957] The device sends the acquired facial image data to the server using the HTTP protocol.

[0958] Input: Facial image data

[0959] Output: Facial image data transferred to the server

[0960] Step 7:

[0961] The server inputs the received facial image data into an emotion recognition engine, which analyzes the image and identifies the user's emotion.

[0962] Input: Facial image data

[0963] Output: Recognized emotion information (e.g., "happiness")

[0964] Step 8:

[0965] The server creates a prompt for recipe generation based on the recognized ingredients and emotion information. An example prompt is as follows:

[0966] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[0967] Input: Recognized ingredient information, emotion information

[0968] Output: prompt statement

[0969] Step 9:

[0970] The server inputs prompts into the generative AI model to generate recipes, which include specific ingredients and steps.

[0971] Input: prompt statement

[0972] Output: The generated recipe (e.g. "Tomato and Mozzarella Salad")

[0973] Step 10:

[0974] The server returns the generated recipe to the user's terminal, which displays the recipe on the user's screen.

[0975] Input: Generated recipe

[0976] Output: The recipe is displayed on the terminal screen.

[0977] Step 11:

[0978] The server then sends an order for ingredients to the food delivery service based on the recognized ingredient information. The order calls an API using the HTTP protocol.

[0979] Input: Recognized ingredient information

[0980] Output: Place an order to a food delivery service

[0981] This allows users to get recipes based on the ingredients they have on hand and their current mood, and then quickly order the ingredients they need.

[0982] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0983] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0984] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0985] [Third embodiment]

[0986] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0987] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0988] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0989] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0990] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0991] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0992] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0993] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0994] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0995] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0996] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0997] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0998] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[0999] Ingredient recognition processing

[1000] 1. Acquire images of ingredients:

[1001] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[1002] 2. Send image data to the server:

[1003] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[1004] 3. Recognizing ingredients using image recognition models:

[1005] The server inputs the received image data into an image recognition algorithm (for example, a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[1006] 4. Providing ingredients information to users:

[1007] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[1008] Handling recipe generation

[1009] 1. Request to generate a recipe:

[1010] The user uses the device interface and instructs the device to "create a recipe using tomatoes."

[1011] 2. Send the request to the server:

[1012] The terminal sends a request including ingredient information (tomato) to the server.

[1013] 3. Give instructions to the generative AI:

[1014] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes."

[1015] 4. Generate the recipe:

[1016] Generative AI generates recipes based on instructions. For example, it generates the following recipes:

[1017] Tomato and mozzarella salad

[1018] material:

[1019] Tomatoes: 2

[1020] Mozzarella cheese: 100g

[1021] Basil leaves: appropriate amount

[1022] Olive oil: 2 tablespoons

[1023] Salt: 1 / 2 teaspoon

[1024] Black pepper: appropriate amount

[1025] Instructions:

[1026] 1. Slice the tomatoes.

[1027] 2. Slice the mozzarella cheese.

[1028] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1029] 4. Garnish with basil leaves.

[1030] 5. Drizzle with olive oil and season with salt and black pepper.

[1031] 5. Providing recipe data to users:

[1032] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[1033] In this way, the system of the present invention provides users with the ability to easily recognize ingredients and create recipes, helping them make the most of the ingredients they have on hand. This allows users to make their daily cooking activities more efficient and creative. This system can reduce ingredient waste and enrich users' cooking repertoire.

[1034] The processing flow will be explained below.

[1035] Step 1:

[1036] A user takes a photo of an ingredient using a device such as a smartphone or tablet, and the device uses its camera function to acquire image data of the ingredient.

[1037] Step 2:

[1038] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[1039] Step 3:

[1040] The server temporarily stores the received image data, which is then prepared for analysis.

[1041] Step 4:

[1042] The server feeds the image data into an image recognition model, which uses deep learning algorithms to identify ingredients in the image.

[1043] Step 5:

[1044] The server analyzes the output of the image recognition model and recognizes the food. For example, if the model identifies a food as a "tomato," the server will recognize the food as a tomato.

[1045] Step 6:

[1046] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[1047] Step 7:

[1048] The device parses the received ingredient information and displays "Recognized ingredient: Tomato" to the user. Based on this information, the user prepares for the next operation.

[1049] Step 8:

[1050] The user uses the terminal interface to say "generate a recipe using tomatoes," which is treated as an appropriate command within the application.

[1051] Step 9:

[1052] The device sends a request to the server to "create a recipe using tomatoes." This request includes ingredient information (tomatoes).

[1053] Step 10:

[1054] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes."

[1055] Step 11:

[1056] The generative AI uses an internal natural language generation algorithm to generate tomato-based recipes, including details such as the title, ingredients, and cooking instructions.

[1057] Step 12:

[1058] The server formats the generated recipe into JSON format and sends it to the device.

[1059] Step 13:

[1060] The device parses the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in a layout that the user can view. For example, a detailed recipe for "Tomato and Mozzarella Salad" is displayed.

[1061] Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients.

[1062] Example 1

[1063] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1064] In today's world, users need to efficiently utilize the ingredients they use on a daily basis and quickly obtain a variety of recipes based on those ingredients. However, users often have to spend a lot of time and effort to find appropriate recipes for the ingredients they have on hand. Furthermore, existing recipe suggestion systems often struggle to flexibly accommodate the specific ingredients a user has and are unable to meet individual needs. Therefore, an effective means is needed to reduce ingredient waste and support users' cooking activities.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1066] In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image and recognizing ingredients in the server, means for providing the recognized ingredient information to the user, means for generating a prompt sentence based on the provided ingredient information and generating a recipe using a generative AI model, and means for providing the generated recipe to the user, thereby enabling the user to quickly obtain a variety of specific recipes based on ingredients on hand.

[1067] "User" refers to an individual who utilizes the system to request ingredient recognition and recipe generation.

[1068] "Terminal" refers to a hardware device that a user uses to operate the system, and specifically includes a smartphone or tablet.

[1069] "Server" refers to a remote computer system that analyzes image data and generates recipes.

[1070] "Means for acquiring images" refers to the function that allows a user to take a photo of an ingredient using a terminal.

[1071] "Means for transmitting to a server" refers to the function of transmitting image data captured from a terminal to a server via the Internet.

[1072] "Means for analyzing images and recognizing ingredients" refers to the algorithms and processes for analyzing image data received by the server and identifying the corresponding ingredients.

[1073] "Means for providing ingredient information" refers to the function of the server returning the ingredient information identified by the server to the terminal and notifying the user.

[1074] "Means for generating a prompt sentence" refers to a function that generates an input sentence required for recipe generation based on the identified ingredient information.

[1075] "Means for generating a recipe using a generative AI model" refers to a function that uses a generative AI model (e.g., a natural language generation model) to generate a specific recipe based on a prompt sentence.

[1076] The "means for providing the generated recipe to the user" refers to a function for providing the generated recipe to the terminal and displaying it to the user.

[1077] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[1078] First, this system uses a device (such as a smartphone or tablet) for users to take photos of ingredients. The image data of the ingredients taken by the user is sent from the device and stored on a server. The device then transmits this data via the HTTP protocol.

[1079] The server then uses a machine learning model (e.g., an image recognition model using deep learning) to analyze the received image data. This model often uses algorithms such as ResNet or YOLO. The server uses this model to analyze the image data and identify the ingredients. For example, the server may identify the ingredient in the image taken by the user as a "tomato."

[1080] The server sends the analysis results back to the device to provide to the user, and the device displays this information to the user, such as "Recognized ingredient: Tomato."

[1081] Next, the user makes a request through the device interface to generate a recipe based on specific ingredients, for example, "Generate a recipe using tomatoes." The device sends this request to the server, which generates a prompt, such as "Generate a simple and delicious recipe using tomatoes."

[1082] The server then uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. The model takes the prompt as input and generates a detailed recipe, such as a "Tomato and Mozzarella Salad" with specific, detailed instructions:

[1083] Tomato and mozzarella salad

[1084] material:

[1085] Tomatoes: 2

[1086] Mozzarella cheese: 100g

[1087] Basil leaves: appropriate amount

[1088] Olive oil: 2 tablespoons

[1089] Salt: 1 / 2 teaspoon

[1090] Black pepper: appropriate amount

[1091] Instructions:

[1092] 1. Slice the tomatoes.

[1093] 2. Slice the mozzarella cheese.

[1094] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1095] 4. Garnish with basil leaves.

[1096] 5. Drizzle with olive oil and season with salt and black pepper.

[1097] The server sends the generated recipe back to the device, and the device provides this recipe information to the user, who can check the recipe on the device screen and use it in their cooking activities.

[1098] In this way, the system of the present invention supports the user's cooking activities by recognizing ingredients on hand and providing specific and detailed recipes based on those ingredients, allowing the user to cook efficiently and creatively without wasting ingredients.

[1099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1100] Step 1: Get an image of the ingredient

[1101] A user takes a photo of an ingredient using a device such as a smartphone or tablet. Specifically, the user opens a camera app and takes a clear image of the ingredient (e.g., a tomato). The input data is an image file of the tomato, and this image is saved on the device.

[1102] Step 2: Send image data to the server

[1103] The image data captured by the device is sent to a server via the Internet. Specifically, an HTTP POST request is used to send the image file to a specific API endpoint on the server. The input data is the saved image file of the tomato, and the output data is the image file passed to the server.

[1104] Step 3: Analyze the image and recognize the ingredients

[1105] The server analyzes the received image data using a deep learning model. Specifically, it runs machine learning algorithms such as ResNet and YOLO to recognize ingredients in the image. The input data is the image file of the tomato that was sent, and the output data is the ingredient information recognized as "tomato."

[1106] Step 4: Provide ingredients information to users

[1107] The server returns the recognition results to the device, which then displays this information to the user. Specifically, the server returns the information "Ingredient: Tomato" in JSON format, and the device analyzes this information and displays "Recognized ingredient: Tomato" on the screen. The input data is the recognized ingredient information, and the output data is the ingredient information displayed to the user.

[1108] Step 5: Request Recipe Generation

[1109] The user issues a command through the device interface to "create a recipe using tomatoes." Specifically, the user taps a button in the application to send the request. The input data is the user's command, and the output data is the creation request.

[1110] Step 6: Send the request to the server

[1111] The terminal sends a user request to the server. This request contains a specific instruction: "Generate a recipe using tomatoes." The input data is the generation request obtained from the user, and the output data is the request sent to the server.

[1112] Step 7: Generate the prompt

[1113] The server generates the prompt required to generate a recipe based on the provided ingredient information. For example, a prompt in the format "Please generate a simple and delicious recipe using tomatoes" is created. The input data is the ingredient information and the generation request, and the output data is the generated prompt.

[1114] Step 8: Generate recipes using generative AI models

[1115] The server uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. Specifically, the prompt is input into the model, and a detailed recipe (e.g., "Tomato and mozzarella salad") is generated. The input data is the prompt, and the output data is the generated recipe.

[1116] Step 9: Provide the generated recipe to the user

[1117] The server returns the generated recipe to the device, and the device provides this recipe information to the user. Specifically, the server returns detailed recipe information in JSON format, and the device parses this information and displays it on the screen. The input data is the generated recipe, and the output data is the recipe information displayed to the user.

[1118] (Application example 1)

[1119] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1120] Today's consumers have a growing need for easy access to recipe information that effectively utilizes ingredients purchased at supermarkets. However, conventional methods are limited in suggesting recipes based on specific ingredients, and lack means to provide information on other related ingredients available in the store and their locations. This makes it difficult for consumers to efficiently purchase the ingredients they need and smoothly carry out cooking activities.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1122] In this invention, the server includes means for acquiring images of ingredients photographed by a user, means for transmitting the acquired images to the server, means for analyzing the images to recognize ingredients, means for providing the user with the recognized ingredient information, means for generating a recipe based on the provided ingredient information, means for providing the generated recipe to the user, means for indicating the location of related ingredients based on ingredient information photographed in the store, and means for generating a recipe based on available ingredients in conjunction with store inventory information. This allows consumers to obtain location information for necessary ingredients based on information on ingredients they wish to purchase in the store, and easily obtain recipes that make the most of the ingredients they have purchased.

[1123] A "user" is a consumer who uses the system to take photos of ingredients and obtain recipe information.

[1124] "Ingredients" are food ingredients that are photographed by the user and are recognized by the system.

[1125] An "image" refers to visual information obtained by a user photographing an ingredient.

[1126] A "server" is a computer device that receives images sent by users and performs analysis and data processing.

[1127] "Means" refers to a method or apparatus for performing a particular function or process.

[1128] "Analysis" refers to the process in which the server analyzes the image data received and recognizes the ingredients.

[1129] "Recognition" is the process of identifying specific ingredients from the image analyzed by the server.

[1130] A "recipe" is a list of cooking instructions and necessary ingredients generated based on recognized ingredient information.

[1131] "Providing" refers to sending information from the server to the user and the user receiving that information.

[1132] A "store" refers to a physical sales location such as a supermarket where users purchase ingredients.

[1133] "Location of related ingredients" is location information of the sales area in the store that sells other ingredients related to the ingredient photographed by the user.

[1134] "Store inventory information" is data on the inventory status of ingredients currently being sold in the store.

[1135] "Integration" refers to the sharing of information and collaboration between different systems and data sources.

[1136] "Available ingredients" refers to food ingredients that are currently available in the store, i.e., available for purchase.

[1137] A "machine learning model" refers to an algorithm that automatically learns from large amounts of data and performs data analysis and predictions.

[1138] A "natural language generation model" is a model that generates linguistically natural sentences based on text data.

[1139] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[1140] System configuration

[1141] 1. User Device:

[1142] A smartphone, tablet, etc. is a device that allows users to take photos of ingredients and request the system to create a recipe.

[1143] 2. Server:

[1144] A high-performance computer device, the server processes image data sent by users, recognizes ingredients, and generates recipes.

[1145] 3. Machine learning models:

[1146] Image recognition algorithms built using deep learning frameworks such as TensorFlow and PyTorch are used to identify ingredients from captured images.

[1147] 4. Natural Language Generation Models:

[1148] It uses generative AI (Natural Language Processing model) to generate recipes based on ingredients.

[1149] System Operation Overview

[1150] 1. Ingredient Recognition Process:

[1151] The user takes a photo of the food using their smartphone and sends the image data to the server.

[1152] The server inputs the received image data into an image recognition algorithm to recognize the ingredients.

[1153] The server returns the recognized ingredient information to the user terminal and displays it to the user.

[1154] 2. Recipe generation process:

[1155] The user requests recipe creation based on the provided ingredient information.

[1156] The server issues instructions to the generative AI model to generate the recipe.

[1157] The server transmits the generated recipe to the user terminal and displays it to the user.

[1158] 3. Providing store information:

[1159] Based on the food information photographed in the store, the location of related ingredients is displayed.

[1160] It works with store inventory information to generate recipes based on available ingredients.

[1161] Program processing flow

[1162] The server first receives image data sent from the user's device. Next, this image data is input into a machine learning model (using TensorFlow or PyTorch) to recognize ingredients. The recognition result (e.g., "tomato") is sent back to the user's device. The user then requests recipe generation based on the provided information. The server generates a recipe by inputting a prompt statement, such as "Please generate a recipe using tomatoes," into the generative AI model. The generated recipe is sent to the user's device and provided to the user. It can also be linked to in-store inventory information to show the user where related ingredients can be purchased.

[1163] Specific examples

[1164] A user takes a picture of a tomato with their smartphone at a supermarket and uses the app to obtain ingredient information. The server receives the image and uses a machine learning model to recognize it as a "tomato." When the user then requests, "Please generate a recipe using tomatoes," the server uses a generative AI model to generate a recipe, such as "Tomato and Mozzarella Salad," and provides the user with such a recipe. Additionally, by providing information on the location and inventory of mozzarella cheese in the store, the server can efficiently purchase the ingredients they need.

[1165] In this way, it is possible to support the user's cooking activities, reduce food waste, and enrich the user's cooking repertoire.

[1166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1167] Step 1:

[1168] The user takes a photo of an ingredient with their smartphone. The input is image data acquired from the smartphone camera. The user takes a photo of an ingredient (e.g., a tomato) and inputs the image data into the application.

[1169] Step 2:

[1170] The device sends the captured image data to the server. The input is the image data stored on the smartphone, and the output is the image data sent to the server via the Internet. The device sends the image data to the server using the HTTP protocol.

[1171] Step 3:

[1172] The server analyzes the received image data and recognizes the ingredients. The input is the image data sent to the server, and the output is the recognized ingredient information (e.g., "tomato"). The server inputs the image data into a machine learning model built using TensorFlow or PyTorch to recognize the ingredients. Specifically, a deep learning algorithm analyzes pixel information in the image and identifies the ingredients.

[1173] Step 4:

[1174] The server returns the recognized ingredient information to the user's device. The input is the recognized ingredient information, and the output is the ingredient information sent to the user's device. The server again uses the HTTP protocol to send the recognition result to the device, and information such as "tomato" is displayed to the user.

[1175] Step 5:

[1176] The user requests a recipe to be generated based on the provided ingredient information. The input is the recognized ingredient information and the user's request, and the output is a recipe generation request. The user uses the app interface to make a request such as "Please generate a recipe using tomatoes."

[1177] Step 6:

[1178] The server instructs the generative AI model to generate a recipe. The input is the user's request and recognized ingredient information, and the output is the generated recipe. The server uses a natural language generation model (generative AI model) to generate a recipe based on the ingredients. Specifically, the prompt sentence "Please generate a recipe using tomatoes" is input into the model, and the generated text data is output as the recipe.

[1179] Step 7:

[1180] The server sends the generated recipe to the user's terminal. The input is the generated recipe data, and the output is the recipe information sent to the user's terminal. The server again uses the HTTP protocol to send the generated recipe to the user's terminal, allowing the user to view the recipe.

[1181] Step 8:

[1182] The server shows the location of related ingredients based on the ingredient information photographed in the store. The input is ingredient information photographed in the store, and the output is location information of related ingredients. The server connects to the store's inventory database to identify the location of available ingredients and provide it to the user.

[1183] Step 9:

[1184] The server works with store inventory information to generate recipes based on available ingredients. The input is store inventory information and recognized ingredient information, and the output is a recipe. The server generates recipes using a generative AI model while taking inventory information into account, and provides them to the user.

[1185] Through the above processing steps, the user can take a photo of the ingredients they want to purchase in the store and, based on that information, efficiently obtain the location of the ingredients and cooking recipes.

[1186] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1187] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[1188] Ingredient recognition processing

[1189] 1. Acquire images of ingredients:

[1190] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[1191] 2. Send image data to the server:

[1192] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[1193] 3. Recognizing ingredients using image recognition models:

[1194] The server inputs the received image data into an image recognition algorithm (e.g., a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[1195] 4. Providing ingredients information to users:

[1196] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[1197] Emotion recognition processing

[1198] 1. Get user sentiment:

[1199] The user uses the device's camera to capture their facial expressions, and the device sends this image data to the emotion engine.

[1200] 2. Analyze emotion data:

[1201] The emotion engine in the device analyzes the image data and recognizes the user's emotion. For example, if the user is smiling, the emotion information of "happiness" is acquired.

[1202] 3. Send emotion information to the server:

[1203] The device sends the acquired emotion information to the server via a network, such as the HTTP protocol.

[1204] Handling recipe generation

[1205] 1. Request to generate a recipe:

[1206] The user uses the device interface to instruct "Create a recipe using tomatoes." At this time, the acquired emotional information is also sent to the server.

[1207] 2. Send the request to the server:

[1208] The terminal sends a request including ingredient information (tomato) and emotion information (e.g., happiness) to the server.

[1209] 3. Give instructions to the generative AI:

[1210] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[1211] 4. Generate the recipe:

[1212] Generative AI generates recipes based on instructions. For example, if the user's emotion is "happy," the generated recipe might include brightly colored ingredients and easy-to-prepare recipes. The generated recipe might include:

[1213] Tomato and mozzarella salad

[1214] material:

[1215] Tomatoes: 2

[1216] Mozzarella cheese: 100g

[1217] Basil leaves: appropriate amount

[1218] Olive oil: 2 tablespoons

[1219] Salt: 1 / 2 teaspoon

[1220] Black pepper: appropriate amount

[1221] Instructions:

[1222] 1. Slice the tomatoes.

[1223] 2. Slice the mozzarella cheese.

[1224] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1225] 4. Garnish with basil leaves.

[1226] 5. Drizzle with olive oil and season with salt and black pepper.

[1227] 5. Providing recipe data to users:

[1228] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[1229] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[1230] The processing flow will be explained below.

[1231] Ingredient recognition processing

[1232] Step 1:

[1233] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato in a refrigerator.

[1234] Step 2:

[1235] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[1236] Step 3:

[1237] The server temporarily stores the received image data, which is then prepared for analysis.

[1238] Step 4:

[1239] The server feeds the image data into an image recognition model (e.g., a deep learning algorithm) that has been trained to identify ingredients.

[1240] Step 5:

[1241] The server analyzes the output of the image recognition model and recognizes ingredients. For example, the model identifies "tomato" from an image.

[1242] Step 6:

[1243] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[1244] Step 7:

[1245] The device analyzes the received ingredient information and displays "Recognized ingredient: Tomato" to the user.

[1246] Emotion engine processing

[1247] Step 8:

[1248] The user uses the camera function of the device to capture a picture of their own facial expression. For example, the user uses the camera of a smartphone to take a picture of their own face.

[1249] Step 9:

[1250] The device sends the captured facial image data to the emotion engine, which is installed on the device.

[1251] Step 10:

[1252] The emotion engine analyzes the image data and recognizes the user's emotions. For example, it obtains emotional information such as "happiness" from the user's facial expression.

[1253] Step 11:

[1254] The device transmits the acquired emotional information to the server, where it is encoded and the necessary security measures are applied.

[1255] Handling recipe generation

[1256] Step 12:

[1257] The user uses the device interface to instruct "Create a recipe using tomatoes." This instruction also includes the user's emotional information.

[1258] Step 13:

[1259] The terminal sends a request including ingredient information (tomato) and emotion information (happiness) to the server.

[1260] Step 14:

[1261] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[1262] Step 15:

[1263] The generative AI uses an internal natural language generation algorithm to generate recipes using tomatoes. For example, if the user's emotion is "happy," the generated recipes will have brightly colored ingredients and be easy to prepare.

[1264] Step 16:

[1265] The server formats the generated recipe into JSON format and sends it to the device.

[1266] Step 17:

[1267] The device analyzes the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in an appropriate layout. For example, "Tomato and Mozzarella Salad" is displayed in detail.

[1268] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[1269] Example 2

[1270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1271] Conventional cooking assistance systems provide a function to recognize ingredients available to the user and suggest recipes based on them. However, these systems cannot take into account personal circumstances such as the user's emotions and physical condition, making it difficult to suggest personalized meals. Therefore, there is a growing need for a system that provides more personalized recipes that are tailored to the user's emotions and circumstances.

[1272] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring the user's emotion, means for transmitting the acquired emotion information to the server, means for generating a recipe based on the provided ingredient information and emotion information, and means for providing the generated recipe to the user. This enables personalized recipe suggestions that take into account the user's emotion and situation in addition to ingredient information.

[1273] "Means for acquiring images" is a function that allows a user to take an image of an ingredient using the camera of the terminal and acquire that data.

[1274] A "server" is a computer system that receives and processes data sent from a terminal via a network.

[1275] The "analysis means" refers to an algorithm and process for analyzing the received image data of ingredients using a machine learning model and recognizing them as specific ingredients.

[1276] The "means for providing ingredient information" is a function that returns information about the recognized ingredients to the terminal and visually presents it to the user.

[1277] The "means for acquiring emotions" is a function for analyzing an image of a user's facial expression and recognizing the user's emotions.

[1278] The "means for transmitting emotion information to a server" is a communication function for transmitting acquired emotion data from the terminal to a server.

[1279] The "means for generating a recipe" is a process of creating a recipe using a generative AI model based on the acquired ingredient information and emotion information.

[1280] The "means for providing the generated recipe" is a function for transmitting the generated recipe from the server to the terminal and displaying it to the user.

[1281] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[1282] Ingredient recognition processing

[1283] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they have taken out of the refrigerator. This photo data is sent by the device to a server using the HTTP protocol. The server inputs the received image data into an image recognition model using deep learning (for example, TensorFlow or PyTorch) for analysis. The model identifies the ingredient from the image and recognizes it as a tomato. The server returns the recognized ingredient information to the device, and the user sees a message on their device screen saying "Recognized ingredient: Tomato."

[1284] Emotion recognition processing

[1285] The user takes a photo of their face using the device's camera function. The device then sends this image data to the built-in emotion engine. The emotion engine analyzes the facial expression and recognizes the user's emotion. For example, if the user is smiling, the emotion engine identifies the emotion information as "happiness." This emotion information is then sent by the device to the server. The HTTP protocol is used for transmission.

[1286] Handling recipe generation

[1287] The user uses the device's application interface to instruct "Generate a recipe using tomatoes." In response to this instruction, the acquired ingredient information (tomatoes) and emotion information (happiness) are sent together to the server. The device sends the data to the server via the HTTP protocol. The server then sends a prompt to a natural language generation model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotion."

[1288] Generative AI generates recipes based on prompts. For example, if the user's emotion is "happy," the generated recipes will feature brightly colored ingredients and easy-to-prepare recipes. The generated recipes might include:

[1289] Tomato and mozzarella salad

[1290] material:

[1291] Tomatoes: 2

[1292] Mozzarella cheese: 100g

[1293] Basil leaves: appropriate amount

[1294] Olive oil: 2 tablespoons

[1295] Salt: 1 / 2 teaspoon

[1296] Black pepper: appropriate amount

[1297] Instructions:

[1298] 1. Slice the tomatoes.

[1299] 2. Slice the mozzarella cheese.

[1300] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1301] 4. Garnish with basil leaves.

[1302] 5. Drizzle with olive oil and season with salt and black pepper.

[1303] The server returns the generated recipe data to the device. The device then displays the recipe to the user and provides detailed cooking instructions. Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it becomes possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[1304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1305] Step 1:

[1306] A user takes a photo of an ingredient using a device such as a smartphone or tablet.

[1307] Input: Photo data of ingredients taken by the user

[1308] Specific operation: The user opens the device's camera app and takes a photo of an ingredient such as a tomato.

[1309] Output: Image data stored in the device's storage

[1310] Step 2:

[1311] The terminal transmits the captured image data to the server.

[1312] Input: Image data stored in the device's storage

[1313] Specific operation: The terminal sends image data to the server via an Internet connection using the HTTP protocol.

[1314] Output: Image data received by the server

[1315] Step 3:

[1316] The image data received by the server is input into a deep learning model to recognize ingredients.

[1317] Input: Image data received by the server

[1318] How it works: The server uses deep learning frameworks such as TensorFlow and PyTorch to apply image recognition algorithms and identify ingredients in the image as tomatoes.

[1319] Output: Recognized ingredient information (e.g., tomato)

[1320] Step 4:

[1321] The server returns the recognized ingredient information to the terminal.

[1322] Input: Recognized ingredient information

[1323] Specific operation: The server sends the recognized ingredient information (e.g., "tomato") to the terminal via the HTTP protocol.

[1324] Output: Ingredient information displayed on the device (e.g. "Recognized ingredient: Tomato")

[1325] Step 5:

[1326] The user takes a photo of their face using the camera function of the device.

[1327] Input: A photo of the user's face

[1328] Specific operation: The user opens the device's camera app and takes a photo of their face.

[1329] Output: Facial photo data stored in the device storage

[1330] Step 6:

[1331] The device sends the captured facial photo to the emotion engine for analysis.

[1332] Input: Facial photo data stored on the device

[1333] How it works: The device's built-in emotion engine analyzes the face photo and recognizes the user's emotion based on their facial expression. For example, if the user is smiling, the emotion information of "happiness" is obtained.

[1334] Output: Recognized emotion information (e.g., "happiness")

[1335] Step 7:

[1336] The terminal transmits the acquired emotion information to the server.

[1337] Input: Recognized emotion information

[1338] Specific operation: The device sends emotional information (e.g., "happiness") to the server via the HTTP protocol.

[1339] Output: Emotion information received by the server

[1340] Step 8:

[1341] A user requests recipe generation using the terminal's application interface.

[1342] Input: User's creation request (e.g. "Create a recipe using tomatoes")

[1343] Specific operation: The user enters the instruction "Create a recipe using tomatoes" on the device's application screen and presses the send button.

[1344] Output: The generated request sent to the server (e.g., containing the information "tomato" and "happiness")

[1345] Step 9:

[1346] The server generates a recipe based on a natural language generation model.

[1347] Input: The generation request received by the server (ingredients and emotion information)

[1348] Specific operation: The server sends a prompt to the generative AI model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotions," and instructs it to generate the recipe.

[1349] Output: The generated recipe

[1350] Step 10:

[1351] The server returns the generated recipe to the device.

[1352] Input: Generated recipe

[1353] Specific operation: The server sends the generated recipe to the terminal via the HTTP protocol.

[1354] Output: The generated recipe (e.g., a tomato and mozzarella salad recipe) displayed in the terminal.

[1355] The above is the specific flow of the system's program processing. Through this series of processes, users can easily obtain personalized recipes based on ingredients and emotions.

[1356] (Application example 2)

[1357] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1358] In many modern households, daily ingredient management and meal planning are major issues that require time and effort. Furthermore, personalized recipe suggestions tailored to the user's mood can be difficult to implement on one's own, and finding the right recipe and the necessary ingredients can be a challenge. Furthermore, cooking decisions can be difficult when ingredients are not on hand. There is a need to solve these challenges and make users' cooking experiences richer and more convenient.

[1359] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by the user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring and analyzing the user's emotion, means for generating a recipe based on the provided ingredient information and the acquired emotion information, means for providing the generated recipe to the user, and means for ordering ingredients based on the ingredient information. This allows the user to easily acquire an optimal recipe based on the ingredients they have on hand and their emotion, and also enables them to quickly order the necessary ingredients.

[1360] "Ingredient image" is image data of food that the user has taken a picture of.

[1361] A "server" is a computer system that sends and receives data over a network and provides processing and storage functions.

[1362] "Analyzing images" refers to identifying the contents of food ingredients from the photographed images and processing the data to identify the ingredients.

[1363] The "means for recognizing ingredients" refers to a method or device for identifying the type of ingredient using image analysis technology.

[1364] "Ingredient information" refers to data related to the recognized ingredient, specifically information such as the name and type of ingredient.

[1365] The "means for providing to the user" is a method or device for displaying the obtained information to the user.

[1366] "Acquiring emotions" refers to collecting data to identify the user's emotional state from facial expression data such as face and voice.

[1367] A "means for analyzing emotions" is a method or device for analyzing acquired emotion data to identify the user's emotional state.

[1368] The "means for generating a recipe" is a method or device for creating cooking instructions and ingredients based on the recognized ingredient information and emotion information.

[1369] A "means for ordering ingredients" is a system or device for ordering the ingredients you need online.

[1370] The system of this invention recognizes images of ingredients taken by users and acquires and analyzes the user's emotions to provide optimal recipes for individual users and enable them to order those ingredients. The detailed configuration and processing procedures of this system are described below.

[1371] Hardware and software used

[1372] In the present invention, the following hardware and software are used.

[1373] Hardware:

[1374] Smartphones with camera functions (e.g., iPhone 12, Samsung Galaxy series)

[1375] Data processing server (e.g. AWS EC2 instance)

[1376] software:

[1377] Image recognition model: A deep learning model for recognizing ingredients (e.g., using PyTorch)

[1378] Emotion Recognition Engine: A machine learning model to identify emotions from user facial expressions (e.g., using Huggingface Transformers)

[1379] Recipe generation engine: A natural language generation model (e.g., OpenAI GPT-3) to generate recipes.

[1380] Network protocol: Data communication uses HTTP and REST API

[1381] System processing overview

[1382] 1. How to get images of ingredients:

[1383] The user uses the smartphone camera to take a photo of an ingredient (e.g., a tomato) in their hand.

[1384] 2. How to send images to the server:

[1385] The captured image data is sent from the smartphone to a server via the Internet using the HTTP protocol.

[1386] 3. Methods for recognizing ingredients by analyzing images:

[1387] The server inputs the received image data into a deep learning-based image recognition model to identify the food item, for example, an image of a tomato.

[1388] 4. Means for providing recognized ingredient information to the user:

[1389] The server sends the recognized ingredient information (e.g., "tomato") back to the user's device, and the user's screen displays "Recognized ingredient: tomato."

[1390] 5. Means of acquiring and analyzing user emotions:

[1391] A user takes a picture of their face using a smartphone camera. The captured image data is sent to an emotion recognition engine for analysis. For example, if the user is smiling, emotion information of "happiness" is acquired.

[1392] 6. A method for generating recipes based on provided ingredient information and acquired emotion information:

[1393] The server creates a recipe generation prompt for the generative AI model based on the recognized ingredient information and emotion information. An example of a prompt sentence is as follows:

[1394] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[1395] A generative AI model (e.g., GPT-3) generates a recipe based on this prompt. For example, the generated recipe might look like this:

[1396] Tomato and mozzarella salad

[1397] material:

[1398] Tomatoes: 2

[1399] Mozzarella cheese: 100g

[1400] Basil leaves: appropriate amount

[1401] Olive oil: 2 tablespoons

[1402] Salt: 1 / 2 teaspoon

[1403] Black pepper: appropriate amount

[1404] Instructions:

[1405] 1. Slice the tomatoes.

[1406] 2. Slice the mozzarella cheese.

[1407] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1408] 4. Garnish with basil leaves.

[1409] 5. Drizzle with olive oil and season with salt and black pepper.

[1410] 7. How to provide the generated recipe to the user:

[1411] The generated recipe is sent back from the server to the user's terminal and displayed so that the user can view it.

[1412] 8. How to order ingredients based on ingredient information:

[1413] Based on the information about the ingredients needed, the user orders ingredients through the food delivery service's API. The server sends this information to the food delivery system, which processes the order.

[1414] This allows users to easily obtain recipes based on the ingredients they have on hand and their current mood, and quickly order the ingredients they need.

[1415] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1416] Step 1:

[1417] The user uses the smartphone camera to take a photo of the ingredients they have on hand, and the captured image data is then stored on the device.

[1418] Input: Ingredient image

[1419] Output: Image data

[1420] Step 2:

[1421] The terminal sends the acquired image data to the server using the HTTP protocol.

[1422] Input: Image data

[1423] Output: Image data transferred to the server

[1424] Step 3:

[1425] The server inputs the received image data into a deep learning-based image recognition model, which analyzes the image and recognizes the ingredients.

[1426] Input: Image data

[1427] Output: Recognized ingredient information (e.g., "tomato")

[1428] Step 4:

[1429] The server returns the recognized ingredient information to the user's terminal, which displays this information on the user's screen.

[1430] Input: Recognized ingredient information

[1431] Output: The device screen will display "Recognized ingredient: Tomato"

[1432] Step 5:

[1433] The user takes a picture of their face using the smartphone camera and the image data is acquired by the terminal.

[1434] Input: An image of the user's face

[1435] Output: Face image data

[1436] Step 6:

[1437] The device sends the acquired facial image data to the server using the HTTP protocol.

[1438] Input: Facial image data

[1439] Output: Facial image data transferred to the server

[1440] Step 7:

[1441] The server inputs the received facial image data into an emotion recognition engine, which analyzes the image and identifies the user's emotion.

[1442] Input: Facial image data

[1443] Output: Recognized emotion information (e.g., "happiness")

[1444] Step 8:

[1445] The server creates a prompt for recipe generation based on the recognized ingredients and emotion information. An example prompt is as follows:

[1446] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[1447] Input: Recognized ingredient information, emotion information

[1448] Output: prompt statement

[1449] Step 9:

[1450] The server inputs prompts into the generative AI model to generate recipes, which include specific ingredients and steps.

[1451] Input: prompt statement

[1452] Output: The generated recipe (e.g. "Tomato and Mozzarella Salad")

[1453] Step 10:

[1454] The server returns the generated recipe to the user's terminal, which displays the recipe on the user's screen.

[1455] Input: Generated recipe

[1456] Output: The recipe is displayed on the terminal screen.

[1457] Step 11:

[1458] The server then sends an order for ingredients to the food delivery service based on the recognized ingredient information. The order calls an API using the HTTP protocol.

[1459] Input: Recognized ingredient information

[1460] Output: Place an order to a food delivery service

[1461] This allows users to get recipes based on the ingredients they have on hand and their current mood, and then quickly order the ingredients they need.

[1462] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1464] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1465] [Fourth embodiment]

[1466] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1467] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1468] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1469] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1470] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1471] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1472] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1473] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1474] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1475] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1476] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1477] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1478] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1479] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[1480] Ingredient recognition processing

[1481] 1. Acquire images of ingredients:

[1482] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[1483] 2. Send image data to the server:

[1484] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[1485] 3. Recognizing ingredients using image recognition models:

[1486] The server inputs the received image data into an image recognition algorithm (for example, a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[1487] 4. Providing ingredients information to users:

[1488] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[1489] Handling recipe generation

[1490] 1. Request to generate a recipe:

[1491] The user uses the device interface and instructs the device to "create a recipe using tomatoes."

[1492] 2. Send the request to the server:

[1493] The terminal sends a request including ingredient information (tomato) to the server.

[1494] 3. Give instructions to the generative AI:

[1495] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes."

[1496] 4. Generate the recipe:

[1497] Generative AI generates recipes based on instructions. For example, it generates the following recipes:

[1498] Tomato and mozzarella salad

[1499] material:

[1500] Tomatoes: 2

[1501] Mozzarella cheese: 100g

[1502] Basil leaves: appropriate amount

[1503] Olive oil: 2 tablespoons

[1504] Salt: 1 / 2 teaspoon

[1505] Black pepper: appropriate amount

[1506] Instructions:

[1507] 1. Slice the tomatoes.

[1508] 2. Slice the mozzarella cheese.

[1509] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1510] 4. Garnish with basil leaves.

[1511] 5. Drizzle with olive oil and season with salt and black pepper.

[1512] 5. Providing recipe data to users:

[1513] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[1514] In this way, the system of the present invention provides users with the ability to easily recognize ingredients and create recipes, helping them make the most of the ingredients they have on hand. This allows users to make their daily cooking activities more efficient and creative. This system can reduce ingredient waste and enrich users' cooking repertoire.

[1515] The processing flow will be explained below.

[1516] Step 1:

[1517] A user takes a photo of an ingredient using a device such as a smartphone or tablet, and the device uses its camera function to acquire image data of the ingredient.

[1518] Step 2:

[1519] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[1520] Step 3:

[1521] The server temporarily stores the received image data, which is then prepared for analysis.

[1522] Step 4:

[1523] The server feeds the image data into an image recognition model, which uses deep learning algorithms to identify ingredients in the image.

[1524] Step 5:

[1525] The server analyzes the output of the image recognition model and recognizes the food. For example, if the model identifies a food as a "tomato," the server will recognize the food as a tomato.

[1526] Step 6:

[1527] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[1528] Step 7:

[1529] The device parses the received ingredient information and displays "Recognized ingredient: Tomato" to the user. Based on this information, the user prepares for the next operation.

[1530] Step 8:

[1531] The user uses the terminal interface to say "generate a recipe using tomatoes," which is treated as an appropriate command within the application.

[1532] Step 9:

[1533] The device sends a request to the server to "create a recipe using tomatoes." This request includes ingredient information (tomatoes).

[1534] Step 10:

[1535] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes."

[1536] Step 11:

[1537] The generative AI uses an internal natural language generation algorithm to generate tomato-based recipes, including details such as the title, ingredients, and cooking instructions.

[1538] Step 12:

[1539] The server formats the generated recipe into JSON format and sends it to the device.

[1540] Step 13:

[1541] The device parses the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in a layout that the user can view. For example, a detailed recipe for "Tomato and Mozzarella Salad" is displayed.

[1542] Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients.

[1543] Example 1

[1544] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1545] In today's world, users need to efficiently utilize the ingredients they use on a daily basis and quickly obtain a variety of recipes based on those ingredients. However, users often have to spend a lot of time and effort to find appropriate recipes for the ingredients they have on hand. Furthermore, existing recipe suggestion systems often struggle to flexibly accommodate the specific ingredients a user has and are unable to meet individual needs. Therefore, an effective means is needed to reduce ingredient waste and support users' cooking activities.

[1546] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1547] In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image and recognizing ingredients in the server, means for providing the recognized ingredient information to the user, means for generating a prompt sentence based on the provided ingredient information and generating a recipe using a generative AI model, and means for providing the generated recipe to the user, thereby enabling the user to quickly obtain a variety of specific recipes based on ingredients on hand.

[1548] "User" refers to an individual who utilizes the system to request ingredient recognition and recipe generation.

[1549] "Terminal" refers to a hardware device that a user uses to operate the system, and specifically includes a smartphone or tablet.

[1550] "Server" refers to a remote computer system that analyzes image data and generates recipes.

[1551] "Means for acquiring images" refers to the function that allows a user to take a photo of an ingredient using a terminal.

[1552] "Means for transmitting to a server" refers to the function of transmitting image data captured from a terminal to a server via the Internet.

[1553] "Means for analyzing images and recognizing ingredients" refers to the algorithms and processes for analyzing image data received by the server and identifying the corresponding ingredients.

[1554] "Means for providing ingredient information" refers to the function of the server returning the ingredient information identified by the server to the terminal and notifying the user.

[1555] "Means for generating a prompt sentence" refers to a function that generates an input sentence required for recipe generation based on the identified ingredient information.

[1556] "Means for generating a recipe using a generative AI model" refers to a function that uses a generative AI model (e.g., a natural language generation model) to generate a specific recipe based on a prompt sentence.

[1557] The "means for providing the generated recipe to the user" refers to a function for providing the generated recipe to the terminal and displaying it to the user.

[1558] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[1559] First, this system uses a device (such as a smartphone or tablet) for users to take photos of ingredients. The image data of the ingredients taken by the user is sent from the device and stored on a server. The device then transmits this data via the HTTP protocol.

[1560] The server then uses a machine learning model (e.g., an image recognition model using deep learning) to analyze the received image data. This model often uses algorithms such as ResNet or YOLO. The server uses this model to analyze the image data and identify the ingredients. For example, the server may identify the ingredient in the image taken by the user as a "tomato."

[1561] The server sends the analysis results back to the device to provide to the user, and the device displays this information to the user, such as "Recognized ingredient: Tomato."

[1562] Next, the user makes a request through the device interface to generate a recipe based on specific ingredients, for example, "Generate a recipe using tomatoes." The device sends this request to the server, which generates a prompt, such as "Generate a simple and delicious recipe using tomatoes."

[1563] The server then uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. The model takes the prompt as input and generates a detailed recipe, such as a "Tomato and Mozzarella Salad" with specific, detailed instructions:

[1564] Tomato and mozzarella salad

[1565] material:

[1566] Tomatoes: 2

[1567] Mozzarella cheese: 100g

[1568] Basil leaves: appropriate amount

[1569] Olive oil: 2 tablespoons

[1570] Salt: 1 / 2 teaspoon

[1571] Black pepper: appropriate amount

[1572] Instructions:

[1573] 1. Slice the tomatoes.

[1574] 2. Slice the mozzarella cheese.

[1575] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1576] 4. Garnish with basil leaves.

[1577] 5. Drizzle with olive oil and season with salt and black pepper.

[1578] The server sends the generated recipe back to the device, and the device provides this recipe information to the user, who can check the recipe on the device screen and use it in their cooking activities.

[1579] In this way, the system of the present invention supports the user's cooking activities by recognizing ingredients on hand and providing specific and detailed recipes based on those ingredients, allowing the user to cook efficiently and creatively without wasting ingredients.

[1580] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1581] Step 1: Get an image of the ingredient

[1582] A user takes a photo of an ingredient using a device such as a smartphone or tablet. Specifically, the user opens a camera app and takes a clear image of the ingredient (e.g., a tomato). The input data is an image file of the tomato, and this image is saved on the device.

[1583] Step 2: Send image data to the server

[1584] The image data captured by the device is sent to a server via the Internet. Specifically, an HTTP POST request is used to send the image file to a specific API endpoint on the server. The input data is the saved image file of the tomato, and the output data is the image file passed to the server.

[1585] Step 3: Analyze the image and recognize the ingredients

[1586] The server analyzes the received image data using a deep learning model. Specifically, it runs machine learning algorithms such as ResNet and YOLO to recognize ingredients in the image. The input data is the image file of the tomato that was sent, and the output data is the ingredient information recognized as "tomato."

[1587] Step 4: Provide ingredients information to users

[1588] The server returns the recognition results to the device, which then displays this information to the user. Specifically, the server returns the information "Ingredient: Tomato" in JSON format, and the device analyzes this information and displays "Recognized ingredient: Tomato" on the screen. The input data is the recognized ingredient information, and the output data is the ingredient information displayed to the user.

[1589] Step 5: Request Recipe Generation

[1590] The user issues a command through the device interface to "create a recipe using tomatoes." Specifically, the user taps a button in the application to send the request. The input data is the user's command, and the output data is the creation request.

[1591] Step 6: Send the request to the server

[1592] The terminal sends a user request to the server. This request contains a specific instruction: "Generate a recipe using tomatoes." The input data is the generation request obtained from the user, and the output data is the request sent to the server.

[1593] Step 7: Generate the prompt

[1594] The server generates the prompt required to generate a recipe based on the provided ingredient information. For example, a prompt in the format "Please generate a simple and delicious recipe using tomatoes" is created. The input data is the ingredient information and the generation request, and the output data is the generated prompt.

[1595] Step 8: Generate recipes using generative AI models

[1596] The server uses a generative AI model (e.g., a natural language generation model such as GPT-4) to create a recipe based on the generated prompt. Specifically, the prompt is input into the model, and a detailed recipe (e.g., "Tomato and mozzarella salad") is generated. The input data is the prompt, and the output data is the generated recipe.

[1597] Step 9: Provide the generated recipe to the user

[1598] The server returns the generated recipe to the device, and the device provides this recipe information to the user. Specifically, the server returns detailed recipe information in JSON format, and the device parses this information and displays it on the screen. The input data is the generated recipe, and the output data is the recipe information displayed to the user.

[1599] (Application example 1)

[1600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1601] Today's consumers have a growing need for easy access to recipe information that effectively utilizes ingredients purchased at supermarkets. However, conventional methods are limited in suggesting recipes based on specific ingredients, and lack means to provide information on other related ingredients available in the store and their locations. This makes it difficult for consumers to efficiently purchase the ingredients they need and smoothly carry out cooking activities.

[1602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1603] In this invention, the server includes means for acquiring images of ingredients photographed by a user, means for transmitting the acquired images to the server, means for analyzing the images to recognize ingredients, means for providing the user with the recognized ingredient information, means for generating a recipe based on the provided ingredient information, means for providing the generated recipe to the user, means for indicating the location of related ingredients based on ingredient information photographed in the store, and means for generating a recipe based on available ingredients in conjunction with store inventory information. This allows consumers to obtain location information for necessary ingredients based on information on ingredients they wish to purchase in the store, and easily obtain recipes that make the most of the ingredients they have purchased.

[1604] A "user" is a consumer who uses the system to take photos of ingredients and obtain recipe information.

[1605] "Ingredients" are food ingredients that are photographed by the user and are recognized by the system.

[1606] An "image" refers to visual information obtained by a user photographing an ingredient.

[1607] A "server" is a computer device that receives images sent by users and performs analysis and data processing.

[1608] "Means" refers to a method or apparatus for performing a particular function or process.

[1609] "Analysis" refers to the process in which the server analyzes the image data received and recognizes the ingredients.

[1610] "Recognition" is the process of identifying specific ingredients from the image analyzed by the server.

[1611] A "recipe" is a list of cooking instructions and necessary ingredients generated based on recognized ingredient information.

[1612] "Providing" refers to sending information from the server to the user and the user receiving that information.

[1613] A "store" refers to a physical sales location such as a supermarket where users purchase ingredients.

[1614] "Location of related ingredients" is location information of the sales area in the store that sells other ingredients related to the ingredient photographed by the user.

[1615] "Store inventory information" is data on the inventory status of ingredients currently being sold in the store.

[1616] "Integration" refers to the sharing of information and collaboration between different systems and data sources.

[1617] "Available ingredients" refers to food ingredients that are currently available in the store, i.e., available for purchase.

[1618] A "machine learning model" refers to an algorithm that automatically learns from large amounts of data and performs data analysis and predictions.

[1619] A "natural language generation model" is a model that generates linguistically natural sentences based on text data.

[1620] The system of the present invention recognizes ingredients that a user uses on a daily basis and generates appropriate recipes based on those ingredients, thereby supporting the user's cooking activities. A specific embodiment of this system will be described.

[1621] System configuration

[1622] 1. User Device:

[1623] A smartphone, tablet, etc. is a device that allows users to take photos of ingredients and request the system to create a recipe.

[1624] 2. Server:

[1625] A high-performance computer device, the server processes image data sent by users, recognizes ingredients, and generates recipes.

[1626] 3. Machine learning models:

[1627] Image recognition algorithms built using deep learning frameworks such as TensorFlow and PyTorch are used to identify ingredients from captured images.

[1628] 4. Natural Language Generation Models:

[1629] It uses generative AI (Natural Language Processing model) to generate recipes based on ingredients.

[1630] System Operation Overview

[1631] 1. Ingredient Recognition Process:

[1632] The user takes a photo of the food using their smartphone and sends the image data to the server.

[1633] The server inputs the received image data into an image recognition algorithm to recognize the ingredients.

[1634] The server returns the recognized ingredient information to the user terminal and displays it to the user.

[1635] 2. Recipe generation process:

[1636] The user requests recipe creation based on the provided ingredient information.

[1637] The server issues instructions to the generative AI model to generate the recipe.

[1638] The server transmits the generated recipe to the user terminal and displays it to the user.

[1639] 3. Providing store information:

[1640] Based on the food information photographed in the store, the location of related ingredients is displayed.

[1641] It works with store inventory information to generate recipes based on available ingredients.

[1642] Program processing flow

[1643] The server first receives image data sent from the user's device. Next, this image data is input into a machine learning model (using TensorFlow or PyTorch) to recognize ingredients. The recognition result (e.g., "tomato") is sent back to the user's device. The user then requests recipe generation based on the provided information. The server generates a recipe by inputting a prompt statement, such as "Please generate a recipe using tomatoes," into the generative AI model. The generated recipe is sent to the user's device and provided to the user. It can also be linked to in-store inventory information to show the user where related ingredients can be purchased.

[1644] Specific examples

[1645] A user takes a picture of a tomato with their smartphone at a supermarket and uses the app to obtain ingredient information. The server receives the image and uses a machine learning model to recognize it as a "tomato." When the user then requests, "Please generate a recipe using tomatoes," the server uses a generative AI model to generate a recipe, such as "Tomato and Mozzarella Salad," and provides the user with such a recipe. Additionally, by providing information on the location and inventory of mozzarella cheese in the store, the server can efficiently purchase the ingredients they need.

[1646] In this way, it is possible to support the user's cooking activities, reduce food waste, and enrich the user's cooking repertoire.

[1647] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1648] Step 1:

[1649] The user takes a photo of an ingredient with their smartphone. The input is image data acquired from the smartphone camera. The user takes a photo of an ingredient (e.g., a tomato) and inputs the image data into the application.

[1650] Step 2:

[1651] The device sends the captured image data to the server. The input is the image data stored on the smartphone, and the output is the image data sent to the server via the Internet. The device sends the image data to the server using the HTTP protocol.

[1652] Step 3:

[1653] The server analyzes the received image data and recognizes the ingredients. The input is the image data sent to the server, and the output is the recognized ingredient information (e.g., "tomato"). The server inputs the image data into a machine learning model built using TensorFlow or PyTorch to recognize the ingredients. Specifically, a deep learning algorithm analyzes pixel information in the image and identifies the ingredients.

[1654] Step 4:

[1655] The server returns the recognized ingredient information to the user's device. The input is the recognized ingredient information, and the output is the ingredient information sent to the user's device. The server again uses the HTTP protocol to send the recognition result to the device, and information such as "tomato" is displayed to the user.

[1656] Step 5:

[1657] The user requests a recipe to be generated based on the provided ingredient information. The input is the recognized ingredient information and the user's request, and the output is a recipe generation request. The user uses the app interface to make a request such as "Please generate a recipe using tomatoes."

[1658] Step 6:

[1659] The server instructs the generative AI model to generate a recipe. The input is the user's request and recognized ingredient information, and the output is the generated recipe. The server uses a natural language generation model (generative AI model) to generate a recipe based on the ingredients. Specifically, the prompt sentence "Please generate a recipe using tomatoes" is input into the model, and the generated text data is output as the recipe.

[1660] Step 7:

[1661] The server sends the generated recipe to the user's terminal. The input is the generated recipe data, and the output is the recipe information sent to the user's terminal. The server again uses the HTTP protocol to send the generated recipe to the user's terminal, allowing the user to view the recipe.

[1662] Step 8:

[1663] The server shows the location of related ingredients based on the ingredient information photographed in the store. The input is ingredient information photographed in the store, and the output is location information of related ingredients. The server connects to the store's inventory database to identify the location of available ingredients and provide it to the user.

[1664] Step 9:

[1665] The server works with store inventory information to generate recipes based on available ingredients. The input is store inventory information and recognized ingredient information, and the output is a recipe. The server generates recipes using a generative AI model while taking inventory information into account, and provides them to the user.

[1666] Through the above processing steps, the user can take a photo of the ingredients they want to purchase in the store and, based on that information, efficiently obtain the location of the ingredients and cooking recipes.

[1667] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1668] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[1669] Ingredient recognition processing

[1670] 1. Acquire images of ingredients:

[1671] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they are holding.

[1672] 2. Send image data to the server:

[1673] The device sends the captured image data to the server via a network, such as via the HTTP protocol.

[1674] 3. Recognizing ingredients using image recognition models:

[1675] The server inputs the received image data into an image recognition algorithm (e.g., a model using deep learning) to identify the food item. In this case, it is recognized as a tomato.

[1676] 4. Providing ingredients information to users:

[1677] The server returns the recognized ingredient information to the terminal, which then displays this information to the user. Specifically, the user's screen displays "Recognized ingredient: Tomato."

[1678] Emotion recognition processing

[1679] 1. Get user sentiment:

[1680] The user uses the device's camera to capture their facial expressions, and the device sends this image data to the emotion engine.

[1681] 2. Analyze emotion data:

[1682] The emotion engine in the device analyzes the image data and recognizes the user's emotion. For example, if the user is smiling, the emotion information of "happiness" is acquired.

[1683] 3. Send emotion information to the server:

[1684] The device sends the acquired emotion information to the server via a network, such as the HTTP protocol.

[1685] Handling recipe generation

[1686] 1. Request to generate a recipe:

[1687] The user uses the device interface to instruct "Create a recipe using tomatoes." At this time, the acquired emotional information is also sent to the server.

[1688] 2. Send the request to the server:

[1689] The terminal sends a request including ingredient information (tomato) and emotion information (e.g., happiness) to the server.

[1690] 3. Give instructions to the generative AI:

[1691] The server instructs a generative AI (for example, a natural language generation model) to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[1692] 4. Generate the recipe:

[1693] Generative AI generates recipes based on instructions. For example, if the user's emotion is "happy," the generated recipe might include brightly colored ingredients and easy-to-prepare recipes. The generated recipe might include:

[1694] Tomato and mozzarella salad

[1695] material:

[1696] Tomatoes: 2

[1697] Mozzarella cheese: 100g

[1698] Basil leaves: appropriate amount

[1699] Olive oil: 2 tablespoons

[1700] Salt: 1 / 2 teaspoon

[1701] Black pepper: appropriate amount

[1702] Instructions:

[1703] 1. Slice the tomatoes.

[1704] 2. Slice the mozzarella cheese.

[1705] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1706] 4. Garnish with basil leaves.

[1707] 5. Drizzle with olive oil and season with salt and black pepper.

[1708] 5. Providing recipe data to users:

[1709] The server sends the generated recipe back to the device, which displays it to the user and provides detailed instructions for cooking.

[1710] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[1711] The processing flow will be explained below.

[1712] Ingredient recognition processing

[1713] Step 1:

[1714] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato in a refrigerator.

[1715] Step 2:

[1716] The image data captured by the device is sent to the server using the HTTP protocol, and the image data is compressed as necessary before being sent to the server.

[1717] Step 3:

[1718] The server temporarily stores the received image data, which is then prepared for analysis.

[1719] Step 4:

[1720] The server feeds the image data into an image recognition model (e.g., a deep learning algorithm) that has been trained to identify ingredients.

[1721] Step 5:

[1722] The server analyzes the output of the image recognition model and recognizes ingredients. For example, the model identifies "tomato" from an image.

[1723] Step 6:

[1724] The server formats the recognized ingredient information into JSON format and sends it to the terminal.

[1725] Step 7:

[1726] The device analyzes the received ingredient information and displays "Recognized ingredient: Tomato" to the user.

[1727] Emotion engine processing

[1728] Step 8:

[1729] The user uses the camera function of the device to capture a picture of their own facial expression. For example, the user uses the camera of a smartphone to take a picture of their own face.

[1730] Step 9:

[1731] The device sends the captured facial image data to the emotion engine, which is installed on the device.

[1732] Step 10:

[1733] The emotion engine analyzes the image data and recognizes the user's emotions. For example, it obtains emotional information such as "happiness" from the user's facial expression.

[1734] Step 11:

[1735] The device transmits the acquired emotional information to the server, where it is encoded and the necessary security measures are applied.

[1736] Handling recipe generation

[1737] Step 12:

[1738] The user uses the device interface to instruct "Create a recipe using tomatoes." This instruction also includes the user's emotional information.

[1739] Step 13:

[1740] The terminal sends a request including ingredient information (tomato) and emotion information (happiness) to the server.

[1741] Step 14:

[1742] The server receives the request and instructs the generative AI to "generate a recipe using tomatoes and make adjustments based on the user's emotions."

[1743] Step 15:

[1744] The generative AI uses an internal natural language generation algorithm to generate recipes using tomatoes. For example, if the user's emotion is "happy," the generated recipes will have brightly colored ingredients and be easy to prepare.

[1745] Step 16:

[1746] The server formats the generated recipe into JSON format and sends it to the device.

[1747] Step 17:

[1748] The device analyzes the received recipe data and displays it to the user. Specifically, it displays the recipe title, ingredients, and instructions in an appropriate layout. For example, "Tomato and Mozzarella Salad" is displayed in detail.

[1749] Through this series of steps, users can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it is possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[1750] Example 2

[1751] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1752] Conventional cooking assistance systems provide a function to recognize ingredients available to the user and suggest recipes based on them. However, these systems cannot take into account personal circumstances such as the user's emotions and physical condition, making it difficult to suggest personalized meals. Therefore, there is a growing need for a system that provides more personalized recipes that are tailored to the user's emotions and circumstances.

[1753] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by a user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring the user's emotion, means for transmitting the acquired emotion information to the server, means for generating a recipe based on the provided ingredient information and emotion information, and means for providing the generated recipe to the user. This enables personalized recipe suggestions that take into account the user's emotion and situation in addition to ingredient information.

[1754] "Means for acquiring images" is a function that allows a user to take an image of an ingredient using the camera of the terminal and acquire that data.

[1755] A "server" is a computer system that receives and processes data sent from a terminal via a network.

[1756] The "analysis means" refers to an algorithm and process for analyzing the received image data of ingredients using a machine learning model and recognizing them as specific ingredients.

[1757] The "means for providing ingredient information" is a function that returns information about the recognized ingredients to the terminal and visually presents it to the user.

[1758] The "means for acquiring emotions" is a function for analyzing an image of a user's facial expression and recognizing the user's emotions.

[1759] The "means for transmitting emotion information to a server" is a communication function for transmitting acquired emotion data from the terminal to a server.

[1760] The "means for generating a recipe" is a process of creating a recipe using a generative AI model based on the acquired ingredient information and emotion information.

[1761] The "means for providing the generated recipe" is a function for transmitting the generated recipe from the server to the terminal and displaying it to the user.

[1762] The system of the present invention supports users' cooking activities by recognizing ingredients that the user uses on a daily basis and generating appropriate recipes based on those ingredients. Furthermore, the system recognizes the user's emotions and adjusts recipes based on those emotions, allowing for more personalized cooking suggestions.

[1763] Ingredient recognition processing

[1764] A user takes a photo of an ingredient using a device such as a smartphone or tablet. For example, the user takes a photo of a tomato they have taken out of the refrigerator. This photo data is sent by the device to a server using the HTTP protocol. The server inputs the received image data into an image recognition model using deep learning (for example, TensorFlow or PyTorch) for analysis. The model identifies the ingredient from the image and recognizes it as a tomato. The server returns the recognized ingredient information to the device, and the user sees a message on their device screen saying "Recognized ingredient: Tomato."

[1765] Emotion recognition processing

[1766] The user takes a photo of their face using the device's camera function. The device then sends this image data to the built-in emotion engine. The emotion engine analyzes the facial expression and recognizes the user's emotion. For example, if the user is smiling, the emotion engine identifies the emotion information as "happiness." This emotion information is then sent by the device to the server. The HTTP protocol is used for transmission.

[1767] Handling recipe generation

[1768] The user uses the device's application interface to instruct "Generate a recipe using tomatoes." In response to this instruction, the acquired ingredient information (tomatoes) and emotion information (happiness) are sent together to the server. The device sends the data to the server via the HTTP protocol. The server then sends a prompt to a natural language generation model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotion."

[1769] Generative AI generates recipes based on prompts. For example, if the user's emotion is "happy," the generated recipes will feature brightly colored ingredients and easy-to-prepare recipes. The generated recipes might include:

[1770] Tomato and mozzarella salad

[1771] material:

[1772] Tomatoes: 2

[1773] Mozzarella cheese: 100g

[1774] Basil leaves: appropriate amount

[1775] Olive oil: 2 tablespoons

[1776] Salt: 1 / 2 teaspoon

[1777] Black pepper: appropriate amount

[1778] Instructions:

[1779] 1. Slice the tomatoes.

[1780] 2. Slice the mozzarella cheese.

[1781] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1782] 4. Garnish with basil leaves.

[1783] 5. Drizzle with olive oil and season with salt and black pepper.

[1784] The server returns the generated recipe data to the device. The device then displays the recipe to the user and provides detailed cooking instructions. Through this series of steps, the user can easily identify ingredients and obtain recipes based on those ingredients. Furthermore, by combining it with an emotion engine, it becomes possible to suggest personalized recipes that match the user's emotions. This makes the user's cooking experience richer and more satisfying.

[1785] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1786] Step 1:

[1787] A user takes a photo of an ingredient using a device such as a smartphone or tablet.

[1788] Input: Photo data of ingredients taken by the user

[1789] Specific operation: The user opens the device's camera app and takes a photo of an ingredient such as a tomato.

[1790] Output: Image data stored in the device's storage

[1791] Step 2:

[1792] The terminal transmits the captured image data to the server.

[1793] Input: Image data stored in the device's storage

[1794] Specific operation: The terminal sends image data to the server via an Internet connection using the HTTP protocol.

[1795] Output: Image data received by the server

[1796] Step 3:

[1797] The image data received by the server is input into a deep learning model to recognize ingredients.

[1798] Input: Image data received by the server

[1799] How it works: The server uses deep learning frameworks such as TensorFlow and PyTorch to apply image recognition algorithms and identify ingredients in the image as tomatoes.

[1800] Output: Recognized ingredient information (e.g., tomato)

[1801] Step 4:

[1802] The server returns the recognized ingredient information to the terminal.

[1803] Input: Recognized ingredient information

[1804] Specific operation: The server sends the recognized ingredient information (e.g., "tomato") to the terminal via the HTTP protocol.

[1805] Output: Ingredient information displayed on the device (e.g. "Recognized ingredient: Tomato")

[1806] Step 5:

[1807] The user takes a photo of their face using the camera function of the device.

[1808] Input: A photo of the user's face

[1809] Specific operation: The user opens the device's camera app and takes a photo of their face.

[1810] Output: Facial photo data stored in the device storage

[1811] Step 6:

[1812] The device sends the captured facial photo to the emotion engine for analysis.

[1813] Input: Facial photo data stored on the device

[1814] How it works: The device's built-in emotion engine analyzes the face photo and recognizes the user's emotion based on their facial expression. For example, if the user is smiling, the emotion information of "happiness" is obtained.

[1815] Output: Recognized emotion information (e.g., "happiness")

[1816] Step 7:

[1817] The terminal transmits the acquired emotion information to the server.

[1818] Input: Recognized emotion information

[1819] Specific operation: The device sends emotional information (e.g., "happiness") to the server via the HTTP protocol.

[1820] Output: Emotion information received by the server

[1821] Step 8:

[1822] A user requests recipe generation using the terminal's application interface.

[1823] Input: User's creation request (e.g. "Create a recipe using tomatoes")

[1824] Specific operation: The user enters the instruction "Create a recipe using tomatoes" on the device's application screen and presses the send button.

[1825] Output: The generated request sent to the server (e.g., containing the information "tomato" and "happiness")

[1826] Step 9:

[1827] The server generates a recipe based on a natural language generation model.

[1828] Input: The generation request received by the server (ingredients and emotion information)

[1829] Specific operation: The server sends a prompt to the generative AI model (e.g., GPT-3) saying, "Generate a recipe using tomatoes and adjust it based on the user's emotions," and instructs it to generate the recipe.

[1830] Output: The generated recipe

[1831] Step 10:

[1832] The server returns the generated recipe to the device.

[1833] Input: Generated recipe

[1834] Specific operation: The server sends the generated recipe to the terminal via the HTTP protocol.

[1835] Output: The generated recipe (e.g., a tomato and mozzarella salad recipe) displayed in the terminal.

[1836] The above is the specific flow of the system's program processing. Through this series of processes, users can easily obtain personalized recipes based on ingredients and emotions.

[1837] (Application example 2)

[1838] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1839] In many modern households, daily ingredient management and meal planning are major issues that require time and effort. Furthermore, personalized recipe suggestions tailored to the user's mood can be difficult to implement on one's own, and finding the right recipe and the necessary ingredients can be a challenge. Furthermore, cooking decisions can be difficult when ingredients are not on hand. There is a need to solve these challenges and make users' cooking experiences richer and more convenient.

[1840] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of ingredients photographed by the user, means for transmitting the acquired image to the server, means for analyzing the image to recognize ingredients, means for providing the recognized ingredient information to the user, means for acquiring and analyzing the user's emotion, means for generating a recipe based on the provided ingredient information and the acquired emotion information, means for providing the generated recipe to the user, and means for ordering ingredients based on the ingredient information. This allows the user to easily acquire an optimal recipe based on the ingredients they have on hand and their emotion, and also enables them to quickly order the necessary ingredients.

[1841] "Ingredient image" is image data of food that the user has taken a picture of.

[1842] A "server" is a computer system that sends and receives data over a network and provides processing and storage functions.

[1843] "Analyzing images" refers to identifying the contents of food ingredients from the photographed images and processing the data to identify the ingredients.

[1844] The "means for recognizing ingredients" refers to a method or device for identifying the type of ingredient using image analysis technology.

[1845] "Ingredient information" refers to data related to the recognized ingredient, specifically information such as the name and type of ingredient.

[1846] The "means for providing to the user" is a method or device for displaying the obtained information to the user.

[1847] "Acquiring emotions" refers to collecting data to identify the user's emotional state from facial expression data such as face and voice.

[1848] A "means for analyzing emotions" is a method or device for analyzing acquired emotion data to identify the user's emotional state.

[1849] The "means for generating a recipe" is a method or device for creating cooking instructions and ingredients based on the recognized ingredient information and emotion information.

[1850] A "means for ordering ingredients" is a system or device for ordering the ingredients you need online.

[1851] The system of this invention recognizes images of ingredients taken by users and acquires and analyzes the user's emotions to provide optimal recipes for individual users and enable them to order those ingredients. The detailed configuration and processing procedures of this system are described below.

[1852] Hardware and software used

[1853] In the present invention, the following hardware and software are used.

[1854] Hardware:

[1855] Smartphones with camera functions (e.g., iPhone 12, Samsung Galaxy series)

[1856] Data processing server (e.g. AWS EC2 instance)

[1857] software:

[1858] Image recognition model: A deep learning model for recognizing ingredients (e.g., using PyTorch)

[1859] Emotion Recognition Engine: A machine learning model to identify emotions from user facial expressions (e.g., using Huggingface Transformers)

[1860] Recipe generation engine: A natural language generation model (e.g., OpenAI GPT-3) to generate recipes.

[1861] Network protocol: Data communication uses HTTP and REST API

[1862] System processing overview

[1863] 1. How to get images of ingredients:

[1864] The user uses the smartphone camera to take a photo of an ingredient (e.g., a tomato) in their hand.

[1865] 2. How to send images to the server:

[1866] The captured image data is sent from the smartphone to a server via the Internet using the HTTP protocol.

[1867] 3. Methods for recognizing ingredients by analyzing images:

[1868] The server inputs the received image data into a deep learning-based image recognition model to identify the food item, for example, an image of a tomato.

[1869] 4. Means for providing recognized ingredient information to the user:

[1870] The server sends the recognized ingredient information (e.g., "tomato") back to the user's device, and the user's screen displays "Recognized ingredient: tomato."

[1871] 5. Means of acquiring and analyzing user emotions:

[1872] A user takes a picture of their face using a smartphone camera. The captured image data is sent to an emotion recognition engine for analysis. For example, if the user is smiling, emotion information of "happiness" is acquired.

[1873] 6. A method for generating recipes based on provided ingredient information and acquired emotion information:

[1874] The server creates a recipe generation prompt for the generative AI model based on the recognized ingredient information and emotion information. An example of a prompt sentence is as follows:

[1875] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[1876] A generative AI model (e.g., GPT-3) generates a recipe based on this prompt. For example, the generated recipe might look like this:

[1877] Tomato and mozzarella salad

[1878] material:

[1879] Tomatoes: 2

[1880] Mozzarella cheese: 100g

[1881] Basil leaves: appropriate amount

[1882] Olive oil: 2 tablespoons

[1883] Salt: 1 / 2 teaspoon

[1884] Black pepper: appropriate amount

[1885] Instructions:

[1886] 1. Slice the tomatoes.

[1887] 2. Slice the mozzarella cheese.

[1888] 3. Arrange tomatoes and mozzarella cheese alternately on a plate.

[1889] 4. Garnish with basil leaves.

[1890] 5. Drizzle with olive oil and season with salt and black pepper.

[1891] 7. How to provide the generated recipe to the user:

[1892] The generated recipe is sent back from the server to the user's terminal and displayed so that the user can view it.

[1893] 8. How to order ingredients based on ingredient information:

[1894] Based on the information about the ingredients needed, the user orders ingredients through the food delivery service's API. The server sends this information to the food delivery system, which processes the order.

[1895] This allows users to easily obtain recipes based on the ingredients they have on hand and their current mood, and quickly order the ingredients they need.

[1896] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1897] Step 1:

[1898] The user uses the smartphone camera to take a photo of the ingredients they have on hand, and the captured image data is then stored on the device.

[1899] Input: Ingredient image

[1900] Output: Image data

[1901] Step 2:

[1902] The terminal sends the acquired image data to the server using the HTTP protocol.

[1903] Input: Image data

[1904] Output: Image data transferred to the server

[1905] Step 3:

[1906] The server inputs the received image data into a deep learning-based image recognition model, which analyzes the image and recognizes the ingredients.

[1907] Input: Image data

[1908] Output: Recognized ingredient information (e.g., "tomato")

[1909] Step 4:

[1910] The server returns the recognized ingredient information to the user's terminal, which displays this information on the user's screen.

[1911] Input: Recognized ingredient information

[1912] Output: The device screen will display "Recognized ingredient: Tomato"

[1913] Step 5:

[1914] The user takes a picture of their face using the smartphone camera and the image data is acquired by the terminal.

[1915] Input: An image of the user's face

[1916] Output: Face image data

[1917] Step 6:

[1918] The device sends the acquired facial image data to the server using the HTTP protocol.

[1919] Input: Facial image data

[1920] Output: Facial image data transferred to the server

[1921] Step 7:

[1922] The server inputs the received facial image data into an emotion recognition engine, which analyzes the image and identifies the user's emotion.

[1923] Input: Facial image data

[1924] Output: Recognized emotion information (e.g., "happiness")

[1925] Step 8:

[1926] The server creates a prompt for recipe generation based on the recognized ingredients and emotion information. An example prompt is as follows:

[1927] Cooking ingredients: Tomato, Mozzarella. User emotion: Happiness. Generate a recipe based on this criteria.

[1928] Input: Recognized ingredient information, emotion information

[1929] Output: prompt statement

[1930] Step 9:

[1931] The server inputs prompts into the generative AI model to generate recipes, which include specific ingredients and steps.

[1932] Input: prompt statement

[1933] Output: The generated recipe (e.g. "Tomato and Mozzarella Salad")

[1934] Step 10:

[1935] The server returns the generated recipe to the user's terminal, which displays the recipe on the user's screen.

[1936] Input: Generated recipe

[1937] Output: The recipe is displayed on the terminal screen.

[1938] Step 11:

[1939] The server then sends an order for ingredients to the food delivery service based on the recognized ingredient information. The order calls an API using the HTTP protocol.

[1940] Input: Recognized ingredient information

[1941] Output: Place an order to a food delivery service

[1942] This allows users to get recipes based on the ingredients they have on hand and their current mood, and then quickly order the ingredients they need.

[1943] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1944] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1945] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1946] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1947] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1948] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1949] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1950] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1951] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1952] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1953] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1954] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1955] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1956] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1957] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1958] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1959] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1960] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1961] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1962] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1963] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1964] The following is further disclosed regarding the above embodiment.

[1965] (Claim 1)

[1966] A means for acquiring an image of an ingredient photographed by a user;

[1967] means for transmitting the acquired image to a server;

[1968] a means for analyzing the image and recognizing ingredients in the server;

[1969] a means for providing the recognized ingredient information to the user;

[1970] A means for generating a recipe based on the provided ingredient information;

[1971] means for providing the generated recipe to the user;

[1972] A system including:

[1973] (Claim 2)

[1974] 10. The system of claim 1, wherein the means for analyzing the image uses a machine learning model.

[1975] (Claim 3)

[1976] 10. The system of claim 1, wherein the means for generating the recipe uses a natural language generation model.

[1977] "Example 1"

[1978] (Claim 1)

[1979] A means for acquiring an image of an ingredient photographed by a user;

[1980] means for transmitting the acquired image to a server;

[1981] a means for analyzing the image and recognizing ingredients in the server;

[1982] a means for providing the recognized ingredient information to the user;

[1983] A means for generating a prompt sentence based on the provided ingredient information and generating a recipe using a generative AI model;

[1984] means for providing the generated recipe to the user;

[1985] A system including:

[1986] (Claim 2)

[1987] 10. The system of claim 1, wherein the means for analyzing the image uses a machine learning model.

[1988] (Claim 3)

[1989] 10. The system of claim 1, wherein the means for generating the prompt sentence and generating the recipe uses a natural language generation model.

[1990] "Application Example 1"

[1991] (Claim 1)

[1992] A means for acquiring an image of an ingredient photographed by a user;

[1993] means for transmitting the acquired image to a server;

[1994] a means for analyzing the image and recognizing ingredients in the server;

[1995] a means for providing the recognized ingredient information to the user;

[1996] A means for generating a recipe based on the provided ingredient information;

[1997] means for providing the generated recipe to the user;

[1998] a means for indicating the location of related ingredients based on ingredient information photographed in the store;

[1999] a means of linking store inventory information to generate recipes based on available ingredients;

[2000] A system including:

[2001] (Claim 2)

[2002] 10. The system of claim 1, wherein the means for analyzing the image uses a machine learning model.

[2003] (Claim 3)

[2004] 10. The system of claim 1, wherein the means for generating the recipe uses a natural language generation model.

[2005] "Example 2: Combining Emotion Engines"

[2006] (Claim 1)

[2007] A means for acquiring an image of an ingredient photographed by a user;

[2008] means for transmitting the acquired image to a server;

[2009] a means for analyzing the image and recognizing ingredients in the server;

[2010] a means for providing the recognized ingredient information to the user;

[2011] A means for acquiring a user's emotion;

[2012] means for transmitting the acquired emotion information to a server;

[2013] a means for generating a recipe based on the provided ingredient information and emotion information;

[2014] means for providing the generated recipe to the user;

[2015] A system including:

[2016] (Claim 2)

[2017] 10. The system of claim 1, wherein the means for analyzing the image uses a machine learning model.

[2018] (Claim 3)

[2019] 10. The system of claim 1, wherein the means for generating the recipe uses a natural language generation model.

[2020] "Application example 2 when combining emotion engines"

[2021] (Claim 1)

[2022] A means for acquiring an image of an ingredient photographed by a user;

[2023] means for transmitting the acquired image to a server;

[2024] a means for analyzing the image and recognizing ingredients in the server;

[2025] a means for providing the recognized ingredient information to the user;

[2026] means for acquiring and analyzing the user's emotions;

[2027] A means for generating a recipe based on the provided ingredient information and the acquired emotion information;

[2028] means for providing the generated recipe to the user;

[2029] a means for ordering ingredients based on the ingredient information;

[2030] A system including:

[2031] (Claim 2)

[2032] 10. The system of claim 1, wherein the means for analyzing the image uses a machine learning model.

[2033] (Claim 3)

[2034] 10. The system of claim 1, wherein the means for generating the recipe uses a generative AI model and prompt statements. [Explanation of symbols]

[2035] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring an image of an ingredient photographed by a user; means for transmitting the acquired image to a server; a means for analyzing the image and recognizing ingredients in the server; a means for providing the recognized ingredient information to the user; A means for generating a recipe based on the provided ingredient information; means for providing the generated recipe to the user; A system including:

2. The system of claim 1 , wherein the means for analyzing the image uses a machine learning model.

3. The system of claim 1 , wherein the means for generating the recipe uses a natural language generation model.

Images