Information processing system, control method, and program

Abstract

[Problem] To assist a creation of a user in cooking. [Solution] An information processing system provided with a control unit that performs: a process for generating cooking information using a cooking learning model on the basis of instruction information input by a user; a process for generating cooking processing information corresponding to cooking steps included in the cooking information, and integrating the cooking processing information to generate integrated cooking processing information; and a process for presenting the integrated cooking processing information being generated.

Description

Information processing system, control method, and program

[0001] The present disclosure relates to an information processing system, a control method, and a program.

[0002] In recent years, technologies have been disclosed that use cooking robots to automate various cooking processes. For example, Patent Literature 1 listed below discloses a technology that senses the biological reactions of a chef while he or she is preparing a dish, and incorporates the sensed biological data into recipe data to recreate a dish that suits the preferences of the person eating it.

[0003] JP 2022-63885 A

[0004] However, there has been insufficient support for users such as chefs and culinary experts when creating new dishes.

[0005] Therefore, the present disclosure proposes an information processing system, a control method, and a program that can assist users in creating dishes.

[0006] According to the present disclosure, an information processing system is provided that includes a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to the cooking steps included in the cooking information using a cooking processing learning model, integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

[0007] In addition, according to the present disclosure, a control method is provided, which includes a processor generating cooking information using a cooking learning model based on instruction information input by a user, generating cooking processing information corresponding to cooking steps included in the cooking information, integrating the cooking processing information to generate integrated cooking processing information, and presenting the generated integrated cooking processing information.

[0008] In addition, according to the present disclosure, a program is provided that causes a computer to function as a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to the cooking steps included in the cooking information and integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

[0009] 1 is a diagram illustrating an overall configuration of a cooking information generation system 1 according to an embodiment of the present disclosure. FIG. 1 is a block diagram illustrating an example of the configuration of an information processing device 10 according to this embodiment. FIG. 2 is a diagram for explaining details of the function of a draft generator 112 according to this embodiment. FIG. 3 is a diagram for explaining generation of cooking information according to this embodiment. FIG. 4 is a flowchart illustrating an example of the overall flow of the cooking information generation system 1 according to this embodiment. FIG. 5 is a flowchart illustrating an example of the flow of a basic information input process according to this embodiment. FIG. 6 is a diagram for explaining estimation of dish contents based on a user's cooking information according to this embodiment. FIG. 7 is a flowchart illustrating an example of the flow of a process for generating a recipe draft according to this embodiment. FIG. 8 is a diagram illustrating an example of a display screen on which a recipe draft is displayed according to this embodiment. FIG. 9 is a diagram illustrating a display screen for displaying details of a recipe draft. FIG. 10 is a flowchart illustrating an example of a process for generating recipe drafts with different levels of freshness according to this embodiment. FIG. 11 is a flowchart illustrating an example of a flow of an adjustment process according to this embodiment. FIG. 12 is a diagram illustrating an example of adjustment according to a user's cooking environment according to this embodiment. FIG. 13 is a flowchart illustrating an example of the flow of an element review process according to this embodiment. FIG. 14 is a flowchart illustrating the flow of a review process for element 1. FIG. 15 is a flowchart illustrating the flow of a review process for element 2. FIG. 16 is a flowchart illustrating the flow of a review process for element 3. FIG. 17 is a flowchart illustrating the flow of a review process for element 4. FIG. 18 is a flowchart illustrating an example of the flow of an integration process according to this embodiment. FIG. 19 is a diagram illustrating an example of presentation of integrated cooking and processing information according to this embodiment. Fig. 10 is a flowchart showing an example of the flow of a process for presenting integrated cooking and processing information according to the present embodiment. Fig. 11 is a diagram showing an example of a detailed display screen for a selected dish according to the present embodiment.

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] The description will be given in the following order: 1. Overview of the recipe information generation system according to an embodiment of the present disclosure 2. Configuration example of the information processing device 10 3. Operation process 3-1. Basic information input process 3-2. Recipe draft generation process 3-3. Adjustment process 3-4. Element review process 3-5. Integration process 3-6. Presentation process 4. Other 5. Supplementary information

[0012] <<1. Overview of a Cooking Information Generation System According to an Embodiment of the Present Disclosure>> An overview of a cooking information generation system according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of a cooking information generation system 1 according to an embodiment of the present disclosure.

[0013] As shown in FIG. 1, the cooking information generation system 1 according to this embodiment includes an information processing device 10, a camera 20, an ingredient sensor 22, a laser processing machine 24, a cooking robot 26, and a cutting machine 28.

[0014] The information processing device 10 can be connected to and communicate with the camera 20, food ingredient sensor 22, laser processing machine 24, cooking robot 26, and cutting machine 28 via network 30. Note that the camera 20, food ingredient sensor 22, laser processing machine 24, cooking robot 26, and cutting machine 28 are examples of a configuration used to consider the feasibility of cooking processing information corresponding to a cooking step, and the present embodiment is not limited to this.

[0015] The information processing device 10 is realized by a PC (personal computer), a tablet terminal, a smartphone, an HMD (head-mounted display), etc. The information processing device 10 creates a new dish in response to instructions (hereinafter also referred to as suggestions) from a user such as a chef, and presents the created dish to the user. The information processing device 10 can appropriately reflect the suggestions from the user in the creation of the dish during the process of creating the dish. In this embodiment, the information processing device 10 can create recipe data, more specifically, including cooking steps for the dish, as the creation of the dish.

[0016] While it has been common to generate new recipes using AI (Artificial Intelligence) based on learning models that have learned from existing recipes, cooking is not simply a matter of cutting and pasting multiple recipes. Creating a dish involves a variety of factors, including changes in ingredients that occur during the cooking process, safety, and design. Furthermore, simply generating new dish images using image generation AI does not lead to the suggestion of feasible recipes using actual ingredients.

[0017] Therefore, this disclosure proposes a system to support users in creating dishes. Specifically, an information processing device 10 generates a recipe (an example of cooking information) using AI, adjusts the recipe according to user feedback, and considers the feasibility of each cooking step included in the recipe (i.e., generates corresponding cooking processing information), and integrates each cooking processing information to generate integrated cooking processing information (a new recipe, also referred to as an integrated recipe), which is presented to the user. The cooking processing information is, for example, information on processing related to the appearance of ingredients (laser processing, cutting, heating, presentation, etc.).

[0018] When generating cooking and processing information, the information processing device 10 may use the results of simulations using a camera 20, food ingredient sensor 22, laser processing machine 24, cooking robot 26, cutting machine 28, etc., or the results of actual cooking.

[0019] For example, the camera 20 captures an image of the food ingredient F and outputs the captured image to the information processing device 10, the laser processing machine 24, the cooking robot 26, or the cutting machine 28. The food ingredient F may be placed by the cooking robot 26. The food ingredient sensor 22 senses various components (e.g., moisture, sugar content, etc.) or conditions (e.g., temperature) of the food ingredient F and outputs the sensing data to the information processing device 10, etc.

[0020] The laser processing machine 24 can draw letters, patterns, marks, etc. on the food material F by irradiating the food material F with a laser to scorch it, or can carve letters, patterns, marks, etc. into the food material F by using a laser to carve the food material F. The cooking robot 26 has one or more robotic arms and can grasp the food material F and place it in a predetermined position, add seasonings, pour sauces, and perform various cooking operations using various cooking utensils. The cutting machine 28 can cut the food material F using a CNC (Computer Numerical Control) milling machine, a laser, etc.

[0021] The laser processing machine 24, cooking robot 26, and cutting machine 28 are digital fabrication devices that can precisely perform cutting, engraving, heating, laminating, and various cooking processes based on digital data. The laser processing machine 24, cooking robot 26, and cutting machine 28 are examples of processing devices that perform various cooking processes, and the configuration of the cooking information generation system 1 is not limited to these.

[0022] The outline of the cooking information generation system 1 according to an embodiment of the present disclosure has been described above. Next, the information processing device 10 that generates cooking information according to this embodiment will be specifically described with reference to the drawings.

[0023] 2 is a block diagram showing an example of the configuration of the information processing device 10 according to this embodiment. As shown in FIG. 2, the information processing device 10 includes a control unit 110, a communication unit 120, an operation input unit 130, a display unit 140, a storage unit 150, and an audio input / output unit 160.

[0024] (Communication Unit 120) The communication unit 120 has a transmission unit that transmits data to an external device and a reception unit that receives data from the external device. For example, the communication unit 120 is communicatively connected to the camera 20 and receives captured images of the food ingredient F from the camera 20. The communication unit 120 is also communicatively connected to the food ingredient sensor 22 and receives sensing data of the food ingredient F from the food ingredient sensor 22. The communication unit 120 is also communicatively connected to various digital fabrication devices (the laser processing machine 24, the cooking robot 26, and the cutting machine 28) and transmits control signals for controlling the various cooking operations of the various digital fabrication devices.

[0025] The communication unit 120 may be communicatively connected to an external device or the Internet using, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution)), 5G (fifth generation mobile communication system)), etc.

[0026] (Operation Input Unit 130 and Display Unit 140) The operation input unit 130 accepts operation input by a user and outputs the input information to the control unit 110. The operation input unit 130 may be realized by a mouse, a keyboard, a switch, a button, or the like. The display unit 140 displays various operation screens and a predicted image of a processed result, which will be described later. The display unit 140 may be a display panel such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The operation input unit 130 and the display unit 140 may be provided integrally. The operation input unit 130 may be a touch sensor stacked on the display unit 140 (for example, a panel display).

[0027] (Control Unit 110) The control unit 110 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 10 in accordance with various programs. The control unit 110 is realized by electronic circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, etc. The control unit 110 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0028] The control unit 110 can also function as an agent unit 111 , a draft generation unit 112 , a display control unit 113 , an adjustment unit 114 , an element review unit 115 , an integration unit 116 , and a story generation unit 117 .

[0029] The agent unit 111 functions as an agent capable of interacting with the user and acquires appropriate information from the interaction with the user. Information input by the user to the information processing device 10 may be input by user operation from the operation input unit 130, or may be input through interaction with an agent presented by the agent unit 111. The interaction between the user and the agent may be carried out in text or audio.

[0030] More specifically, the agent unit 111 may be an AI agent that utilizes large language models (LLMs) or the like, which allows the user to input natural language.

[0031] The agent unit 111 acquires the user's suggestions (requests) regarding the dish to be created through a dialogue with the user. For example, the user may interact with the agent about the concept of a new dish they want to make, the ingredients they want to use, and so on. The agent unit 111 generates and outputs prompts corresponding to the user's suggestions based on the dialogue with the user. AI may be used to generate the prompts. More specifically, a learning model that has learned about prompt generation may be used. Hereinafter, the prompts corresponding to the user's suggestions output from the agent unit 111 are also referred to as suggestions.

[0032] The recipe plan generation unit 112 has a function of generating a recipe plan (an example of recipe information). The recipe plan generation unit 112 uses a learning model generated by machine learning to generate one or more recipe plans in response to user suggestions (prompts) input from the agent unit 111. The recipe plan includes at least recipe data for the dish. The recipe plan may also include dish images. The generation of a recipe plan will be described in detail below with reference to FIG. 3.

[0033] 3 is a diagram for explaining in detail the functions of the recipe plan generator 112 according to this embodiment. As shown in FIG. 3, the recipe plan generator 112 includes an ingredient pairing unit 510 and a recipe plan generator 520.

[0034] The agent unit 111 interacts with the user U, and inputs a prompt 41 generated from the user's suggestions 40 into the draft generation unit 112. The user's suggestions 40 include information about the concept of a new dish to be made and the ingredients to be used. The concept of a dish may include, for example, the impression, atmosphere, style, genre, theme, purpose, and key points of the dish. In addition, requests regarding the design of the dish and the ingredients to be used may also be input as concepts.

[0035] An example of a concept is shown in Table 1 below. The concept may be the overall image of the dish, or it may be related to the design or ingredients.

[0036]

[0037] The ingredient pairing unit 510 outputs combination ingredients that are suitable for use in combination with the ingredients (used ingredients) that the user wants to use. For example, the ingredient pairing unit 510 inputs data on the used ingredients into an ingredient pairing learning model and outputs ingredients that are well combined with the used ingredients or ingredients that are compatible at the molecular level. The ingredient pairing learning model is a learning model generated by learning ingredient combinations using machine learning.

[0038] The recipe draft generator 520 inputs the concept, ingredients used, and ingredient combinations into the cooking learning model as input data, and outputs one or more recipe drafts 42, which are recipe information proposals. The cooking learning model is a model that has learned about cooking in advance through machine learning. The cooking learning model can learn images (captured images) of various dishes and corresponding recipe data, etc. The recipe data to be learned includes the name of the dish, ingredients, cooking steps, cooking utensils used, cooking time, and tags (for example, attribute information such as recipe category, purpose of cooking, genre, suitable season, suitable occasion, target demographic, and other keywords).

[0039] Specific examples of recipe categories to be tagged are as follows: Examples of "suitable seasons" include spring, summer, fall, winter, New Year's, cherry blossom viewing, moon viewing, Christmas, etc. Examples of "suitable occasions" include entertaining, parties, snacks, late-night snacks, pairings with wine, anniversaries, etc. Examples of "target demographics" include athletes, children, adults, and seniors. Examples of "other keywords" include gorgeous, colorful, lively, nutritious, low-salt, easy to digest, healthy, relaxing, mood changer, easy, time-saving, nostalgic, Italian, Chinese, etc. The recipe draft output from the cooking learning model includes recipe data and corresponding dish images. The cooking learning model may include a recipe generation AI that generates recipe data and an image generation AI that generates dish images corresponding to the recipe data.

[0040] The cooking learning models used by the recipe draft generator 520 include a user cooking learning model 521 that has learned the user's cooking, and an other user cooking learning model 522 that has learned the cooking of other users (e.g., other chefs). There may be multiple other user cooking learning models 522. For example, there may be an other user cooking learning model that has learned the cooking of a French chef, or an other user cooking learning model that has learned the cooking of a Chinese chef. Furthermore, the other user cooking learning model 522 may be an other user cooking learning model that has learned the cooking of many chefs regardless of cooking genre.

[0041] The recipe draft generation unit 520 may output a recipe draft using the user's cooking learning model 521, or may output a recipe draft using the other user's cooking learning model 522, or may output a recipe draft using the user's cooking learning model 521 and the other user's cooking learning model 522. The user's cooking learning model 521 and the other user's cooking learning model 522 may be stored in the storage unit 150. For convenience of explanation, FIG. 3 illustrates the cooking learning models used by the recipe draft generation unit 520 as the configuration of the recipe draft generation unit 520, but it is assumed that the recipe draft generation unit 520 calls and uses each cooking learning model stored in the storage unit 150.

[0042] Furthermore, when generating the dish information, the recipe draft generation unit 520 may link with an external database as appropriate to acquire external information (for example, information about the dish, information about ingredients, information related to the input concept, various designs, etc.) Furthermore, the recipe draft generation unit 520 may use a design generation AI to acquire a design according to the concept, input the design data into the cooking learning model, and output a dish image decorated based on that design.

[0043] The display control unit 113 controls the display of images on the display unit 140. The display control unit 113 controls the display on the display unit 140 of various input screens such as a screen for prompting a dialogue with an AI agent, a presentation screen for a recipe draft generated by the draft generation unit 112, a presentation screen for presenting new recipe information (integrated cooking and processing information) generated by the integration unit 116 (described later), a presentation screen for presenting a development story for a new dish generated by the story generation unit 117 (described later), and the like.

[0044] The adjustment unit 114, the element review unit 115, and the integration unit 116 will be described below with reference to Fig. 4. Fig. 4 is a diagram for explaining the generation of cooking information according to this embodiment.

[0045] The adjustment unit 114 adjusts the recipe drafts 42 generated by the draft generation unit 112 and outputs the adjusted recipe drafts to the element review unit 115. The adjustment unit 114 may be input with one or more recipe drafts automatically selected from the recipe drafts 42 generated by the draft generation unit 112, or with one or more recipe drafts selected by the user. As a method of automatic selection, for example, the adjustment unit 114 may assign a score to each recipe draft based on a certain criterion (e.g., whether the recipe draft is close to the characteristics of the user's cooking) and select the recipe draft with the highest score. Furthermore, when the user makes a selection, the recipe drafts may be sorted based on scores according to various criteria, making it easier for the user to make a selection.

[0046] 4, a recipe draft 42A selected by the user is input to the adjustment unit 114. The recipe draft 42A includes recipe data 421A and a dish image 422A.

[0047] The adjustment unit 114 may receive new user suggestions (prompts based on the suggestions, also simply referred to as suggestions) obtained from the dialogue between the user and the agent from the agent unit 111. The adjustment unit 114 adjusts the recipe draft in accordance with the user suggestions input from the agent unit 111. In this case, the adjustment unit 114 may use a cooking learning model (user cooking learning model, other user cooking learning model), an ingredient pairing learning model, image generation AI, etc. The cooking learning model is stored in the memory unit 150, and the adjustment unit 114 can call up and use the cooking learning model from the memory unit 150 as needed. Furthermore, it is assumed that this cooking learning model is the same as the cooking learning model used in the recipe draft generation unit 520 of the draft generation unit 112.

[0048] The adjustment unit 114 also functions as an environment adjustment unit that adjusts the recipe plan according to the user's cooking environment information. The adjustment unit 114 may also acquire external information from an external database or the like as needed and use the external information to adjust the recipe plan.

[0049] Specifically, adjustments to the recipe draft by the adjustment unit 114 may include replacing ingredients used, obtaining new combinations of ingredients, changing the cooking utensils used, and modifying the dish image (appearance of the dish).

[0050] For example, when a user selects a recipe draft and converses with the agent saying, "This is good, but maybe fruit would be better as a topping," the agent unit 111 inputs information such as "change the topping to fruit" as a suggestion based on the conversation with the user, and the adjustment unit 114 adjusts the recipe data to change the topping ingredients to fruit.

[0051] The adjustment unit 114 may also change the cooking utensils in the recipe data to suit the user's cooking environment. For example, recipe data "steam potatoes in a pot for 10 minutes" may be modified to recipe data that uses a "microwave oven," a method commonly used in the user's cooking environment, specifically, recipe data "steam potatoes in a microwave oven at 600W for 3 minutes." When the cooking utensil is changed from a pot to a microwave oven, the adjustment unit 114 may generate appropriate setting information for the microwave oven (microwave output, time) based on external information (such as a correspondence table between boiling time and microwave heating time) obtained from an external database or the like.

[0052] In this way, the adjustment unit 114 adjusts the recipe plan to more closely match the user's intentions and environment.

[0053] The element review unit 115 has a function of dividing the recipe data into elements according to the cooking steps of the recipe data included in the recipe draft, in order to increase the feasibility of the recipe draft output from the adjustment unit 114. In other words, at the recipe draft stage, there is no certainty as to whether the dish can actually be made using this recipe, or whether it is realistically possible to create the appearance of the dish shown in the dish image. The recipe draft includes recipe data indicating the cooking steps and a dish image showing the appearance of the dish, but the appearance of the dish shown in the dish image generated by the image generation AI (e.g., presentation on a dish, decoration) may not be realistic.

[0054] Therefore, the element review unit 115 reviews the feasibility of each element, specifically, optimizes the control of the cooking robot to be used and generates variations (expands the elements), and generates cooking processing information for each element that is feasible, i.e., that will result in the successful cooking process. The element review involves actually cooking with a cooking robot and analyzing the results, or cooking through simulation. The element review unit 115 can generate one or more pieces of cooking processing information for each element.

[0055] If there is an element (cooking step) that does not produce a successful result, the element review unit 115 requests adjustment from the adjustment unit 114. In this case, the adjustment unit 114 adjusts the recipe data (for example, changes to the ingredients used, the cooking utensils used, or the appearance of the dish), and the element review unit 115 reviews the element again.

[0056] As shown in FIG. 4, the element reviewing section 115 divides the recipe data 421A' of the recipe plan adjusted by the adjusting section 114 into elements 1 to 3, . . . , and generates variations in the process of reviewing each element.

[0057] The integration unit 116 combines each successful element (cooking process information) to generate new cooking information (integrated cooking process information). As described above, control optimization and variation generation are performed for each element (cooking process) of the decomposed recipe data, and one or more successful derived elements can be generated for each element. The integration unit 116 can appropriately combine the successful derived elements for each element to generate multiple patterns of new recipe data as integrated cooking process information. In this case, the integration unit 116 can generate new recipe data that meets conditions such as the overall cooking time. The integration unit 116 also generates dish images corresponding to each new recipe data as integrated cooking process information.

[0058] As shown in FIG. 4, the integration unit 116 appropriately combines variations of each element to generate new dish information such as dish A'1 (integration 1) to dish A'3 (integration 3), . . .

[0059] The one or more pieces of integrated cooking and processing information generated by the integration unit 116 are output to the display control unit 113 and displayed on the display unit 140, thereby being presented to the user. When the integrated cooking and processing information is displayed, elements not used in the integration may also be presented as candidates in addition to the elements (cooking and processing information) used in the integration. The user may interact with the agent about the presented integrated cooking and processing information and further input suggestions. For example, a suggestion to change the elements used in the integration to other elements listed as candidates may be input. Furthermore, a change in ingredients or a change in the concept may also be input.

[0060] The agent unit 111 inputs the new indications to the adjustment unit 114, the element review unit 115, or the integration unit 116, and causes them to appropriately correct the integrated cooking and processing information.

[0061] It is also possible that the user's requests may change from the original content during the process of creating a new dish. In this embodiment, the user can input new suggestions as needed during the creation process, not only at the start of creating dish information, but also when selecting a recipe plan or when the integrated cooking and processing information is presented. This allows the system to cooperate with the user to propose a creative dish that is closer to the user's intentions, thereby supporting the user's dish creation.

[0062] The story generation unit 117 generates a development story for the original dish based on information stored in the storage unit 150 as appropriate during the creation process. The information stored in the storage unit 150 includes the user's dialogue with the agent (user's suggestions), the initially selected dish draft, the unselected dish drafts, and examples of success and failure in element review. The development story may be generated using text and images. The development story may also be a still image with text and images arranged therein, or a video showing the changes in the image of the dish from the original dish draft to the original dish. The generated development story is output to the display control unit 113 and displayed on the display unit 140 for presentation to the user. The development story may be used, for example, for advertising.

[0063] The functional configuration of the control unit 110 has been specifically described above. The control unit 110 is not limited to the functional configuration described above and has a tool linkage function. Specifically, the control unit 110 may link with various conventional tools as appropriate when generating cooking information. For example, the control unit 110 may link with 3D CAD when generating a design in the draft generation unit 112 or the element review unit 115. The control unit 110 may also link with tools that control digital fabrication equipment such as the laser processing machine 24, cooking robot 26, and cutting machine 28 to control these devices. The control unit 110 may also link with various AI tools, such as image generation AI.

[0064] (Storage Unit 150) The storage unit 150 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 110, and a RAM that temporarily stores parameters that change as needed.

[0065] For example, the storage unit 150 may store a cooking learning model and an ingredient pairing learning model used by the plan generation unit 112 and the adjustment unit 114. The storage unit 150 may also store basic information about the user (such as recipe data registered by the user and cooking environment information). The storage unit 150 may also store various information output in the process of creating a new dish, such as the dish plan generated by the plan generation unit 112, the dish plan adjusted by the adjustment unit 114, and the element review results by the element review unit 115.

[0066] (Audio Input / Output Unit 160) The audio input / output unit 160 has an audio input unit that collects audio and inputs the audio data to the control unit 110, and an audio output unit that outputs audio. The audio input unit is realized by, for example, a microphone and collects user audio. The audio output unit is realized by, for example, a speaker and outputs agent audio.

[0067] The configuration of the information processing device 10 has been specifically described above. Note that the configuration of the information processing device 10 according to the present disclosure is not limited to the example shown in FIG. 2 . For example, the information processing device 10 may be realized by a plurality of devices. Furthermore, at least some of the functions of the information processing device 10 may be realized by a server on the Internet. For example, each functional configuration of the control unit 110 may be provided in the server. In other words, each configuration of the information processing device 10 may be realized by an information processing system including a user terminal and a server.

[0068] <<3. Operational Processing>> Next, the operational processing of the cooking information generation system 1 according to this embodiment will be described with reference to FIG.

[0069] FIG. 5 is a flowchart showing an example of the overall flow of the cooking information generation system 1 according to this embodiment.

[0070] 5, the information processing device 10 first performs an input process for user basic information (step S103). The user basic information may be input by a user operation from the operation input unit 130, or may be input through a dialogue between the user and an agent presented by the agent unit 111.

[0071] Next, the information processing device 10 generates one or more recipe drafts using the draft generator 112 in response to the user's suggestions regarding the new original dish, which are acquired from the dialogue between the agent and the user (step S106).

[0072] Next, the information processing device 10 presents one or more recipe drafts to the user on the display unit 140 (step S109).

[0073] Next, the information processing device 10 accepts the user's selection of a recipe plan, and further acquires new suggestions from the user (step S112).

[0074] Next, the information processing device 10 causes the adjustment unit 114 to perform an adjustment process on the recipe draft (step S115).

[0075] Next, the information processing device 10 performs a process of dividing the recipe data included in the recipe draft into elements corresponding to the cooking steps, and examining the feasibility of each element (step S118) by the element review unit 115. Specifically, one or more pieces of cooking processing information are generated for each element.

[0076] Next, the information processing device 10 performs an integration process using the integration unit 116 to integrate each generated element (each cooking processing information) (step S121), and then displays the integrated cooking processing information (new cooking information) generated by the integration process on the display unit 140 and presents it to the user (step S124).

[0077] Next, if the user inputs new instructions about the presented integrated cooking and processing information (Yes in step S127), it is determined that correction is necessary, and the process returns to step S115 (adjustment process), step S118 (element review process), or step S121 (integration process) as appropriate, and correction is made. The control unit 110 appropriately determines at which stage to start correction depending on the content of the new instructions.

[0078] If there are no user-specified items (step S127 / No), the information processing device 10 performs a process of registering the presented integrated cooking and processing information (step S130). The information processing device 10 may register the integrated cooking and processing information selected by the user from the presented plurality of pieces of integrated cooking and processing information.

[0079] Note that the operational process shown in FIG. 5 is merely an example, and all steps shown in FIG. 5 do not necessarily have to be performed in the order shown in FIG. 5 . For example, the information processing device 10 may skip steps S109 and S112 and have the control unit 110 arbitrarily select a recipe draft. Furthermore, after the adjustment process shown in step S115, the information processing device 10 may present the adjusted recipe draft to the user, and perform the adjustment process again if new suggestions are input. Furthermore, after the element review process shown in S118, the information processing device 10 may present the review results to the user, and perform the element review process again if new suggestions are input. Furthermore, before step S121, the information processing device 10 may present multiple cooking processing information (or processing results) as candidates and accept the user's selection, and in step S121, integrate the results to include the cooking processing information selected in accordance with the user's selection.

[0080] Each process shown in FIG. 5 will be described in detail below.

[0081] 3-1. Basic Information Input Process FIG. 6 is a flowchart showing an example of the flow of the basic information input process according to this embodiment. When the cooking information generation system 1 according to this embodiment (an AI tool that assists users in creating new dishes) is launched, it first checks whether the user's basic information has already been entered. If the necessary basic information has not been entered, the system executes the basic information input process described below. This system may be executed, for example, by an application installed on an information processing device 10. The user's cooking environment may also include various cooking appliances controllable by the system (e.g., a laser processing machine 24, a cutting machine 28, a heating device (e.g., a pot, a microwave, a frying pan, etc.)), a camera 20, an ingredient sensor 22, a robot (e.g., a cooking robot 26), etc.

[0082] 6 , in the information processing device 10, the display control unit 113 displays an input prompting screen on the display unit 140 that prompts the user to input information about the dish (basic information), and accepts input of the basic information by the user (step S203). The basic information may be input through a dialogue with an agent presented by the agent unit 111. For example, the agent unit 111 controls the display unit 140 via the display control unit 113 to display the agent, and controls the start of a dialogue with the user via the voice input / output unit 160.

[0083] The necessary basic information of the user includes, for example, the user's favorite dishes, specialty dishes (French cuisine, etc.), past cooking information (recipe data, cooking images (captured images)), and cooking environment information (mainly used utensils, special cooking utensils, etc.). The recipes and cooking images included in the past cooking information may be tagged with recipe categories (such as the purpose of cooking, such as for Christmas, or information on the target demographic, such as for athletes). The past cooking information may be obtained, for example, from a recipe management system used by the user, or from a database specified by the user.

[0084] Next, the control unit 110 stores the input basic information of the user in the storage unit 150 (step S206).

[0085] Next, the control unit 110 starts the learning process of the user's dish information. First, the control unit 110 prepares data to be used for learning. Specifically, the control unit 110 estimates the contents of the dish (step S209) and the tableware used (step S212) based on the input dish image and recipe data.

[0086] FIG. 7 is a diagram illustrating the estimation of dish contents based on the user's dish information according to this embodiment. As shown in FIG. 7 , the input user's dish information 44 includes a dish image 441 and recipe data 442. The control unit 110 analyzes the dish image 411 to perform image recognition (recognizing the objects shown in the image) and estimates the dish contents by referring to the recipe data. Specifically, the control unit 110 associates each dish item shown in the dish image 411 with the recipe data. In the example shown in FIG. 7 , association information 45 with the recipe data may be acquired for each image-recognized dish item, such as "Recipe 2: Fried Shrimp," "Recipe 3: Meat," and "Recipe 4: Side Dish." The control unit 110 also estimates the tableware used (i.e., the tableware owned by the user) from the image-recognized dish portion, such as "round plate, medium size."

[0087] Next, the control unit 110 uses the user's cooking information (recipe data, dish images) and the estimated results of the cooking contents and tableware used obtained from the cooking information as input data (learning data) to learn about the user's cooking and generate a user cooking learning model 521 (step S215). Note that the input data is not limited to the above example. For example, recipe data may include information on ingredients and cooking utensils used, but if calorie information is not included, the control unit 110 may access an external ingredient database to obtain calorie information as external information and add it to the recipe data as input data. Furthermore, as described above, recipe data and dish images may also have recipe categories (purpose of cooking, genre, target demographic, etc.) added as tags. Therefore, such recipe categories are also input and used for learning. The learning method is not particularly limited, but machine learning such as reinforcement learning is considered. The generated user cooking learning model 521 is stored in the storage unit 150.

[0088] In this way, by preparing a learning model that has learned the user's cooking, the draft generation unit 112 can generate a recipe draft based on the user's cooking (for example, in line with the characteristics of the user's cooking) in accordance with the input data (user's suggestions (purpose of cooking, genre, ingredients, etc.)).

[0089] While the generation of the user cooking learning model 521 has been described above, in this embodiment, another user cooking learning model 522 may also be prepared in advance, generated by learning the cooking of many users (e.g., chefs). The learning data used in the other user cooking learning model 522 is not particularly limited, but may be obtained, for example, from a recipe management system (application) used by many chefs, or from a database storing recipe collections of chefs.

[0090] <3-2. Recipe Plan Generation Process> FIG. 8 is a flowchart showing an example of the flow of a recipe plan generation process according to this embodiment.

[0091] 8 , first, the agent unit 111 of the information processing device 10 inputs the user's suggestions, acquired through a dialogue with the user, to the plan generation unit 112 (step S303). The user's suggestions are items (requests) related to the dish the user wants to make, and in this embodiment, are assumed to be the concept and ingredients to be used. The agent unit 111 may input the suggestions (concept, ingredients to be used) obtained by analyzing the user's dialogue (content converted into text by voice recognition or text entered by the user) using natural language processing directly to the plan generation unit 112 as a prompt, or may input a prompt generated by supplementing information as appropriate.

[0092] Possible contents of the input data (prompt) input to the draft generation unit 112 include, for example, "purpose of cooking (Christmas dish, party dish, etc.), ingredients (potatoes, cheese, seafood, etc.), genre (appetizer, stew, Chinese dish, etc.), nutrition (low calorie, high calorie, etc.), target demographic (children, athletes, etc.), cooking equipment used (oven cooking, 3D printer, etc.), design (geometric design, gentle design, flashy design, etc.)" The agent unit 111 may supplement the details of the concept input by the user through dialogue, or may supplement the information by eliciting necessary information by asking a question such as "What kind of customer base do you think?" The agent unit 111 may also randomly supplement missing information.

[0093] Next, the draft generation unit 112 uses the ingredient pairing learning model in the ingredient pairing unit 510 to output ingredients (combined ingredients) that are suitable for combination with the ingredients specified by the user (included in the above-mentioned suggestions) (step S306).

[0094] Next, the recipe plan generator 112 uses the cooking learning model to generate a recipe plan based on the input recommendations and ingredient combinations (step S309) using the recipe plan generator 520. As described above, the cooking learning model includes the user's cooking learning model and other users' cooking learning models. The recipe plan generator 520 can generate recipe plans with different levels of freshness using the cooking learning model as appropriate. The recipe plan generation process will be described later with reference to FIG. 12.

[0095] Note that the recommendations may include multiple items, multiple contents per item, or multiple combination ingredients. The recipe plan generation unit 112 may generate multiple input data by randomly combining the recommendations and the information contained in the combination ingredients. This makes it possible to generate recipe plans corresponding to various combinations based on the user's recommendations, etc. The generated recipe plans are assumed to include recipe data and dish images. A user learning model that has learned the recipe data and dish images (captured images) may generate dish images corresponding to the recipe data. Furthermore, the dish images may be generated by an image generation AI based on the generated recipe data. For example, the recipe plan generation unit 112 may generate appropriate prompts based on the results obtained by text analysis of the recipe data, and input the prompts to the image generation AI to obtain dish images.

[0096] The generated recipe plan is displayed on the display unit 140 by the display control unit 113 and presented to the user (step S312).

[0097] 9 is a diagram showing an example of a display screen on which a recipe plan is displayed according to this embodiment. As shown in FIG. 9, a large number of recipe plans are displayed on a display screen 600. Here, thumbnails of dish images included in the recipe plan are displayed. When the user taps on any dish image, details of the recipe plan, i.e., the contents of the recipe, are displayed.

[0098] 10 and 11 are diagrams illustrating a display screen displaying details of a recipe plan. When a user selects a dish image from the recipe plan list screen as shown in Fig. 9, the screen switches to a recipe plan detail display screen 610a, as shown on the left side of Fig. 10. The detail display screen 610a displays recipe data 611a, a dish image 612a, and a suggested items display area 613a. The recipe data 611a includes ingredients (foodstuffs, seasonings), cooking steps, etc.

[0099] The user can check the presented recipe and food images, and can make further suggestions at this point to revise the presented dish plan based on the initial user's suggestions. An example of how to input suggestions is described below.

[0100] 10 right, if a user selects a fruit in a cooking image 612b by touch operation (e.g., by circling it) and says, "I want to make this melon into a spring-like fruit," the agent unit 111 recognizes this and displays "Make it into a spring-like fruit" as the recognized suggestion in the suggestion display area 613a. The agent unit 111 may also mark the part of the recipe that corresponds to the part selected by the user (e.g., "Decorate the cake with melon").

[0101] 11, if the user selects a plate in the cooking image 612c by touch operation and says, "I'd like to add more decorations around here to make it more colorful," the agent unit 111 will recognize this and display the recognized suggestion, "Add more decorations to make it more colorful," in the suggestion display area 613c. The agent unit 111 may also mark the part of the recipe (for example, "Place the cake on the plate") that corresponds to the part selected by the user.

[0102] Then, in the information processing device 10, the adjustment unit 114 adjusts the recipe draft in accordance with the newly input suggestions. Details of the adjustment will be described later with reference to FIG. 13. The right side of FIG. 11 shows a display screen displaying the recipe draft adjusted in accordance with the suggestions input by the user. The display screen 610d displays recipe data 611d that has been changed by the adjustment, a dish image 612d that has been newly generated in accordance with the adjustment, and an explanation area 613d that explains the adjustment content. The changed parts may be marked in the recipe data 611d.

[0103] (Generation of Recipe Plans with Different Levels of Newness) Next, the generation of recipe plans using the cooking learning model in the plan generation unit 112 will be described in more detail. As described above, the plan generation unit 112 can generate recipe plans by appropriately using the user's cooking learning model and the other user's cooking learning model. When the user's cooking learning model is used, it is expected that a recipe plan that is in line with (similar to) the characteristics of the user's cooking will be generated, and when the other user's cooking learning model is used, it is expected that a recipe plan that is different from (not similar to) the characteristics of the user's cooking will be generated, that is, at least more new to the user.

[0104] The recipe plan generator 112 may use a recipe learning model as appropriate to control the generation rate of recipe plans with different levels of novelty in accordance with the obtained level of novelty of the recipe desired by the user.

[0105] FIG. 12 is a flowchart showing an example of the flow of a process for generating recipe plans with different levels of freshness according to this embodiment.

[0106] 12, first, the plan generator 112 checks the newness level of the dish desired by the user (step S323). The newness level of the dish desired by the user may be acquired by the agent unit 111 from a conversation with the user, or may be input by the user during the initial basic information input process.

[0107] Next, the plan generator 112 determines the ratio of the number of recipe drafts A to D to be generated depending on the newness level (step S326). The recipe drafts A to D are drafts with different recipe newness levels, with the relationship being newness level a of recipe draft A < newness level b of recipe draft B < newness level c of recipe draft C < newness level d of recipe draft D. The plan generator 112 performs control so that the higher the newness level, the more recipe drafts with higher newness levels are generated.

[0108] Next, the plan generator 112 sets information (prompts) about the concept and ingredients (including combination ingredients) indicated by the user as input data for the cooking learning model (step S329).

[0109] Next, the recipe plan generator 112 generates a recipe plan A using the user's recipe model (step S332).

[0110] The recipe plan generator 112 also uses another user's recipe model to change the style (here, the appearance or visual appearance of the dish) of the recipe plan generated using the user's recipe model, generating a recipe plan B (step S335). In this specification, "changing the style of the dish" primarily refers to changing the appearance or visual appearance of the dish, such as the presentation, arrangement, and decoration of the dish. More specifically, the dish image included in the recipe plan is changed. Even with the same recipe, the impression of the dish can change if the presentation, decoration, tableware used, etc. are different. The chef's characteristics are also reflected in the appearance of such dishes, and by changing them using another user's recipe model, it becomes possible to create a new dish (that is less similar to the user's own dish).

[0111] The plan generator 112 also generates a recipe plan C by changing the style of the recipe plan generated using the other user's recipe model using the user's recipe model (step S338).

[0112] In addition, the recipe plan generator 112 generates a recipe plan D using other users' recipe models (step S341).

[0113] In the generation of the recipe plans A to D described above, a large number of recipe plans A to D are generated. The plan generator 112 may generate the recipe plans A to D according to the ratio determined in step S326.

[0114] The plan generator 112 then outputs the recipe plans A to D in accordance with the determined ratios (the number of recipe plans corresponding to the ratios) (step S344). Each recipe plan is displayed on the display unit 140 by the display controller 113.

[0115] The generation of a recipe plan according to a newness level has been described above. Note that the "newness level of the recipe desired by the user" is not necessarily required, and the control unit 110 may set the newness level arbitrarily.

[0116] <3-3. Adjustment Process> Next, the adjustment process according to this embodiment will be described in detail. In the adjustment process, adjustments are made to the recipe draft. In this embodiment, the user can input new suggestions after viewing the presented recipe draft. A user's ideas may change during the creative process. In this embodiment, collaboration between the user and AI can be realized by allowing the user to input new suggestions to a recipe draft generated using a cooking learning model based on the suggestions initially input by the user. The information processing device 10 regenerates (modifies) a recipe draft using the cooking learning model in response to the new suggestions. Note that by generating and presenting multiple recipe drafts based on the initial suggestions and allowing the user to select at least one from these, it is possible to narrow down the candidates to those that more closely match the user's intentions. This also reduces the amount of calculation required in subsequent processing.

[0117] FIG. 13 is a flowchart showing an example of the flow of the adjustment process according to this embodiment.

[0118] 13 , if the user provides new instructions for the recipe draft (Yes in step S403) and the instructions are changes to ingredients (Yes in step S406), the adjustment unit 114 uses the ingredient pairing learning model to output new ingredients suitable for pairing with the changed ingredients (step S409). If the change in ingredients is not a specific instruction (e.g., "I want to change to a spring-like fruit," "I want to use ingredients with a good texture," "I want to increase the variety of vegetables," etc.), the adjustment unit 114 may determine ingredients using a database or AI (e.g., a pre-trained learning model for determining ingredients) in response to an abstract instruction such as "a spring-like fruit."

[0119] Next, the adjustment unit 114 adjusts the recipe data in the recipe draft using the cooking learning model, with the ingredients used, the combined ingredients, and the user's new instructions as input data (step S412). Examples of instructions other than ingredient changes include instructions on the cooking process or cooking contents. Furthermore, the adjustment unit 114 adjusts the appearance of the dish using the cooking learning model in accordance with the user's new instructions (e.g., instructions on the layout, design, etc. of the dish) (step S415). Specifically, the adjustment unit 114 converts the dish image. The adjustment unit 114 may also convert the dish image to correspond to the revised recipe data.

[0120] Finally, the adjustment unit 114 adjusts the recipe data to accommodate the user's cooking environment (step S418). For example, the adjustment unit 114 assigns a cooking robot, a laser processing machine, or the like to the cooking steps of a recipe based on the information about the user's cooking environment input as basic information. FIG. 14 is a diagram showing an example of adjustment according to the user's cooking environment according to this embodiment. Here, the recipe for "Mushroom Farci" will be used for explanation, which was generated based on the user's input that "I want to make a beautiful stuffed dish using vegetables."

[0121] As shown in recipe data 620 in the upper row of Figure 14, the cooking steps of the recipe are "1. Remove the stems from the mushrooms, 2. Add a pattern to the surface of the mushrooms, 3. Stuff the mushrooms with a filling (cheese, nuts, etc.), 4. Arrange the food on a plate and garnish with leaves," and if the user's cooking environment is equipped with a cooking robot, a laser processing machine, etc., the adjustment unit 114 assigns the cooking robot, etc. to the cooking steps. Specifically, as shown in recipe data 621 in the lower row of Figure 14, the recipe data 620 is modified to "1. The robot cuts off the stems from the mushrooms, 2. The robot engraves a pattern on the surface of the mushrooms with a laser, 3. The robot stuffs the mushrooms with a filling (cheese, nuts, etc.), 4. The robot arranges the food on a plate and garnishes with leaves."

[0122] 3-4. Element Review Process Next, the element review process will be described. Fig. 15 is a flowchart showing an example of the flow of the element review process according to this embodiment. Here, the feasibility of cooking is reviewed for each element of the recipe draft generated by the draft generation unit 112 or the recipe draft adjusted by the adjustment unit 114.

[0123] As shown in FIG. 15, first, the element reviewing unit 115 of the information processing device 10 determines each cooking step of the recipe data included in the recipe draft as an element (step S503).

[0124] Next, the element review unit 115 reviews the feasibility of the cooking process for each element (step S506). In reviewing elements, the content is derived (specified) (for example, for "cut carrots into the shape of a flower," variations of the flower shape are generated), and then cooking simulations are performed or actual cooking is performed using a cooking robot or the like. Specific examples of such element reviews will be described later with reference to FIGS. 16 to 19.

[0125] Next, if there are any elements that have all failed, the element reviewing unit 115 requests the adjusting unit 114 to adjust the recipe data (step S512), and performs the element review process according to this embodiment again.

[0126] Next, a specific example of element consideration will be described. For example, in the case of a recipe for "mushroom farci" as shown in Fig. 14, recipe data 621 is divided into four elements (elements 1 to 4) according to the cooking process: "1. The robot cuts off the stems of the mushrooms," "2. The robot engraves a pattern on the surface of the mushrooms with a laser," "3. The robot stuffs the mushrooms with a filling (cheese, nuts, etc.)," ​​and "4. The robot arranges the food on a plate and garnishes with leaves." Consideration of these elements will be described in order below.

[0127] (Review of Element 1) The review of the cooking process of element 1, "The robot cuts off the stems of the mushrooms," will be described with reference to Fig. 16. Fig. 16 is a flowchart showing the flow of the review process of element 1.

[0128] 16, first, the element reviewing unit 115 checks whether a method for cutting off the stem of a mushroom by a robot has been learned (step S523). If the method has been learned (step S523 / Yes), the process proceeds to step S532.

[0129] If the model has not yet been learned (step S523 / No), the element review unit 115 inputs the 3D mushroom model into the simulator (step S526) and learns how the robotic hand (cooking robot 26) in the user's environment will remove the mushroom stalk (step S529). The 3D mushroom model may be acquired from an external source, or the mushroom may be captured in 3D by the camera 20. For example, the element review unit 115 can use a simulator corresponding to the robotic hand in the user's environment to learn how to remove the mushroom stalk through machine learning such as reinforcement learning, thereby more efficiently exploring how to remove the mushroom stalk (simulation optimization). The learning model generated here (an example of a cooking and processing learning model) can be used to re-simulate how to remove the mushroom stalk.

[0130] Next, the element review unit 115 acquires the cooking time required (step S532). For example, the element review unit 115 acquires the cooking time required for the successful axis selection method.

[0131] Next, if the simulation results show that all cooking process plans (cooking process information) for the cooking step of element 1, "removing mushroom stems," are unsuccessful despite being considered in the user's cooking environment (step S535 / Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S538). The adjustment unit 114 may make adjustments, such as changing the recipe to one that requires a human to remove mushroom stems, since this is difficult for the cooking robot 26 to do, changing the recipe to one that does not require removing the stems, or changing ingredients to ones that are easier to remove the stems from. Note that if any of the cooking process plans for the cooking step of element 1 are successful (step S535 / No), the process ends. In the integration process described below, the successful cooking process plans are used to integrate the elements.

[0132] (Study of Element 2) The study of the cooking step of element 2, "carving a pattern on the surface of the mushroom with a laser", will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the flow of the study process of element 2.

[0133] 17, first, the element reviewing unit 115 checks whether engraving of a mushroom with a laser cutter (laser processing machine 24) has been learned (step S543). If learning has been learned (step S543 / Yes), the process proceeds to step S567.

[0134] If the element has not been learned (step S543 / No), the element review unit 115 generates n pattern patterns (step S546). For example, the element review unit 115 generates pattern variations based on the food image output from the adjustment unit 114 using an algorithm such as style conversion.

[0135] Next, the element review unit 115 defines m sets of laser processing parameters (step S549). For example, the element review unit 115 sets the ranges of laser control power and speed, such as "power: minimum 40, maximum 60" and "speed: minimum 50, maximum 90," and defines m combinations of power and speed. The element review unit 115 may refer to processing examples of similar ingredients or may define parameters starting from low power parameters in consideration of safety.

[0136] Next, the element reviewing unit 115 selects a set of a pattern and processing parameters (step S552).

[0137] Next, the element reviewing unit 115 causes the robot (cooking robot 26) to set the mushroom (in a location where it can be processed by a laser cutter) (step S555), and controls the laser cutter to process (carve) it (step S558).

[0138] Next, the element reviewing unit 115 photographs the processed food material with the camera 20 and acquires an image (step S561).

[0139] The above processes shown in steps S552 to S561 are performed for all combinations of pattern patterns and processing parameters (step S562), thereby obtaining a large number of processing results (images of processed ingredients) such as those shown in Table 2 below.

[0140]

[0141] If all patterns have been examined (step S403 / Yes), the element review unit 115 learns the relationships between the pattern pattern, processing parameters, and processed images and creates a learning model for predicting the processing result from the pattern pattern and processing parameters (step S564). Even with the same pattern pattern, the processing result may be clear, crushed, or thin depending on the processing parameters. This is because the processing reaction may not be linear depending on the ingredients of the ingredients, and phenomena such as the ingredients swelling or crumbling may occur. In this embodiment, a learning model (an example of a cooking processing learning model) is used to predict the processing result from the pattern pattern and processing parameters, making it possible to consider numerous combinations of pattern patterns and processing parameters. The learning model may be generated by machine learning, such as reinforcement learning.

[0142] Next, the element review unit 115 evaluates each cooking processing plan (a combination of pattern pattern and processing parameters: an example of cooking processing information) based on the degree of match between the pattern pattern generated from the pattern in element 2 and the processing result (predicted by the learning model) (whether the pattern is distorted by the processing) (step S567).

[0143] In addition, the element reviewing unit 115 obtains the cooking time required (for each combination pattern) (step S570).

[0144] If all the evaluations performed in step S567 are low (step S573 / Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S576). The adjustment unit 114 changes, for example, the ingredients or the pattern. If there is a cooking / processing plan for the cooking step of element 2 that has a certain rating or higher (step S573 / No), the process ends. In the integration process described below, cooking / processing plans that have a certain rating or higher are used to integrate the elements.

[0145] (Consideration of Element 3) The consideration of the cooking process of element 3, "Robot stuffs mushrooms with fillings (cheese, nuts, etc.)", will be described with reference to Fig. 18. Fig. 18 is a flowchart showing the flow of the consideration process of element 3.

[0146] As shown in Fig. 18, first, if the user has specified a restriction on the filling to be stuffed into the mushrooms (step S583 / Yes), the element review unit 115 sets a restriction on the filling material (step S586). The restriction from the user may be extracted from a conversation between the agent and the user, such as "I want to make a dish for customers with allergies" or "I want to reduce the calories." Possible restrictions on the filling material include, for example, controlling the ingredients and the amount. If the user has not specified a restriction (step S583 / No), the process proceeds to step S589.

[0147] Next, the element reviewing unit 115 creates a plurality of patterns of ingredients to be mixed as filling materials (step S589).

[0148] Next, the element review unit 115 imports the 3D model of the mixed material into a simulator and uses the simulator to learn how the robot hand fills mushrooms with the filling material according to the mixing pattern (step S592). The generated learning model is an example of a cooking and processing learning model. The learning model can be generated by machine learning such as reinforcement learning.

[0149] Next, the element reviewing unit 115 acquires the cooking time required (step S595).

[0150] If all of the recipes are unsuccessful (step S598 / Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S601).The element review unit 115 can use the generated learning model to review the feasibility (success or failure) of each mixing pattern.If any of the cooking plans for the cooking step of element 3 is successful (step S598 / No), the process ends.In the integration process described below, the successful cooking plans are used to integrate the elements.

[0151] (Consideration of Element 4) The consideration of the cooking process of element 4, "The robot arranges the food on a plate and garnishes it with leaves," will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the flow of the consideration process of element 4.

[0152] 19, first, the element review unit 115 generates variations of food images in which food is served on tableware that the user normally uses (step S603). Information about the tableware that the user uses has already been acquired in the basic information input process.

[0153] Next, when decorating a plate with leaves or flowers, the element review unit 115 generates variations of 3D models of the decorations (step S606). For example, in the case of leaves, many different types of leaves are often combined for decoration, so 3D models of many different types of leaves are generated. Note that the generation of variations of the 3D models may be performed using a tool such as 3D CAD. Here, the case of decorating with leaves will be described below.

[0154] Next, the element review unit 115 generates a leaf arrangement pattern (step S609). The element review unit 115 generates an arrangement pattern by combining, for example, many types of leaves.

[0155] Next, the element review unit 115 uses a simulator to learn possible arrangements for the robot hand (cooking robot 26) (step S612). The generated learning model is an example of a cooking and processing learning model. The learning model can be generated by machine learning, such as reinforcement learning.

[0156] Next, the element review unit 115 deletes placement patterns that are difficult to realize (step S615). The element review unit 115 can use the generated learning model to review the feasibility (whether or not each placement pattern will be successful) of each placement pattern.

[0157] Next, the element review unit 115 generates food images by combining variations of the above food image of food served on tableware with leaf layouts of each learned arrangement pattern (specifically, arrangement patterns that are likely to be realized by the robot hand) (step S618).

[0158] In addition, the element reviewing unit 115 acquires the cooking time required (for each arrangement pattern) (step S621).

[0159] If all of the cooking plans are unsuccessful (i.e., there is no feasible arrangement pattern) (step S624 / Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S627). Note that if there is a successful cooking plan among the cooking process plans (arrangement pattern: an example of cooking process information) for element 4 (step S624 / No), the process ends. In the integration process described below, the successful cooking plan is used to integrate the elements.

[0160] The above describes specific examples of the consideration of each element. In this way, in the element consideration process, more specific variations of the cooking process (cooking processing information) are created based on each cooking process in the recipe data of the recipe draft, and a consideration is made as to whether the cooking process will be successful. Furthermore, in the learning of the cooking processing learning model used in the consideration of each element described above, the information processing device 10 can further improve the accuracy of generating the learning model by including failure cases in the learning data. The element consideration unit 115 may present multiple cooking processing information (or processing results) for each element as candidates to the user, allowing the user to select the cooking processing information to be included in the integration.

[0161] 20 is a flowchart showing an example of the flow of the integration process according to this embodiment. The integration unit 116 combines the elements reviewed by the element review unit 115 to generate integrated cooking and processing information.

[0162] 20 , if a user specifies a condition for integration (Yes in step S703), the integration unit 116 changes the default condition in response to the user's specification (step S706). For example, the user may specify a cooking time of 15 minutes or less and an evaluation rank of 3 or higher for element 2. Another example of the user's specification is the user's selection of cooking and processing information to be included in the integration.

[0163] On the other hand, if there is no instruction from the user regarding the integration conditions (step S703 / No), the integration unit 116 sets default conditions (step S709).

[0164] Next, the integration unit 116 generates combination patterns of the elements that satisfy the conditions (step S712).

[0165] Next, the integration unit 116 generates an integrated recipe and a corresponding dish image as integrated cooking and processing information based on each combination pattern (step S715).

[0166] Finally, the integration unit 116 adjusts the procedural efficiency and safety of the integrated recipe (step S718). For example, in the case of a recipe for "mushroom falci," if the ingredients of the filling need to be allowed to marinate in the refrigerator for a while, the integration unit 116 can shorten the overall cooking time by adjusting the process so that this step is performed first. Furthermore, in consideration of food safety, the integration unit 116 adds a step of temporarily storing ingredients that require temperature control in the refrigerator while the cooking robot performs time-consuming processing and arrangement. Knowledge of procedural efficiency and safety may be acquired through cooking training or by linking with an external database, etc.

[0167] 3-6. Presentation Process Next, a presentation process of the integrated cooking and processing information according to this embodiment will be described. The integrated recipe and the corresponding dish image integrated by the integration unit 116 are an example of integrated cooking and processing information.

[0168] The display control unit 113 may display each dish image as a thumbnail as an example of the presentation of the integrated cooking and processing information. Fig. 21 is a diagram showing an example of the presentation of the integrated cooking and processing information according to this embodiment. As shown in Fig. 21, the dish images are displayed as thumbnails on a presentation screen 640.

[0169] The presentation of the integrated cooking and processing information will be described in detail below. Fig. 22 is a flowchart showing an example of the flow of the process of presenting the integrated cooking and processing information according to this embodiment.

[0170] 22, if a user specifies a sauce condition (Yes in step S803), the display control unit 113 sorts the dish images according to the condition (step S806), such as by cooking time.

[0171] On the other hand, if the user has not specified any sauce conditions (step S803 / No), the display control unit 113 sorts the dish images, for example, in the order in which they were created (step S809).

[0172] Next, the display control unit 113 displays a list of food images (step S812), as shown in FIG.

[0173] Next, if a dish image is selected by the user (step S815 / Yes), the display control unit 113 displays details of the selected dish (step S818).

[0174] Next, when a component (ingredient) of a dish included in the dish image is pointed to on the details display screen of the selected dish (step S821 / Yes), the display control unit 113 displays details of that component (step S824).

[0175] 23 is a diagram showing an example of a detailed display screen for a selected dish according to this embodiment. As shown in Fig. 23, a detailed display screen 650 displays a dish image 651, element information display 652 for the selected dish, and a detailed display 653 for the cooking pattern.

[0176] In the element information display 652, not only adopted cooking processing patterns but also unadopted cooking processing patterns are displayed. For example, when "Cooking process 2" of element 2 is selected from among the elements of the dish "Mushroom Farci" selected in the dish image 651, blocks for "Pattern pattern" and "Processing parameter" are displayed. When "Pattern pattern" is selected, the detailed display 653 displays a large number of pattern patterns (examples of cooking processing information) generated by element review in the element review unit 115, including those that were not adopted in the dish, as shown in FIG.

[0177] Furthermore, when "processing parameters" is selected, the detailed display 653 displays a large number of processing parameters (an example of cooking processing information) generated by the element review in the element review unit 115, including parameters that were not adopted in the dish. At this time, an image after processing for each processing parameter of each pattern pattern (processing results predicted by the learning model) may also be displayed. The user can confirm what the processing results will be when which pattern pattern is processed using which processing parameters.

[0178] In this way, the user can check the pattern patterns derived but not adopted by the element review unit 115, the numerous processing parameters that were not adopted, and the processing results corresponding to the processing parameters. While checking these, the user can make more specific suggestions, such as, "The shape will be a little distorted, but I think this color (burnt color) would be good," thereby efficiently running the review cycle. That is, the user can select any of the many pieces of cooking and processing information presented and instruct the system to generate integrated cooking and processing information including the selected cooking and processing information. In this embodiment, an iterative process is assumed in which, each time a new suggestion is made by the user, corrections are made until a final proposal is reached.

[0179] Specifically, as shown in FIG. 22, if a user inputs a problem (Yes in step S827), the agent unit 111 acquires a new problem from the user through a dialogue between the agent and the user (step S830).

[0180] Next, the control unit 110 summarizes the user's suggestions and presents them to the user so that the user can confirm them, and starts correction (step S833). Specifically, as shown in Fig. 5, the adjustment process in the adjustment unit 114, the review process in the element review unit 115, and the integration process in the integration unit 116 are performed again as appropriate, and the corrected cooking and processing information is presented.

[0181] Examples of new issues and corrections made by users include the following:

[0182]

[0183] <<4. Others>> The various learning models used in the cooking information generation system 1 according to this embodiment may be realized by LLMs (large-scale language models). That is, the above-mentioned user cooking learning model 521, other users cooking learning model 522, ingredient pairing learning model, and cooking processing learning model may be configured to use LLMs as knowledge databases.

[0184] Furthermore, in the training of various learning models according to the present embodiment, the information processing device 10 may use failure cases for training, thereby improving the accuracy of generating the learning model.

[0185] The integration unit 116 can also set personalization conditions, which are conditions for changing the content of elements (cooking and processing information) for each individual receiving the dish, in accordance with input from the user during the integration process. For example, the integration unit 116 sets personalization conditions that determine the type, amount, arrangement, or processing state of ingredients in the specified elements in accordance with the preferences or calorie intake of each individual. The integration unit 116 generates an integrated recipe that satisfies the personalization conditions set by the user. Furthermore, during cooking, the display control unit 113 may rearrange the configuration of corresponding elements in accordance with the personalization conditions specified by the user and present the integrated recipe.

[0186] In the above-described embodiment, as shown in Figures 21 and 23, a food image is displayed on the screen as a method of presenting the generated cooking and processing information to the user, but this embodiment is not limited to this, and the information processing device 10 may actually cook a top-ranked dish using a cooking robot 26 or the like and present it to the user.

[0187] In this embodiment, a development story can also be generated by the story generation unit 117. During the process of creating a dish, various data is accumulated in the storage unit 150, such as dialogue between the agent and the user, adjustments made, and element considerations. When the dish is completed (when there are no new suggestions and it has been finalized), the story generation unit 117 picks out catchphrases and preferences that reflect the user's characteristics from the data accumulated in the storage unit 150, extracts unusual corrections made in response to the user's suggestions, re-learning, and episodes of failure and success (such as failed / successful cooking and processing methods), compiles a development story, and outputs it as text or images.

[0188] In this embodiment, the initial recipe draft is prepared taking into consideration user suggestions, but adjustments, element review, and integration are performed automatically and mechanically, and the result is presented to the user. Since cooking is often complex and requires multiple ingredients, the chef's sense is also important for creating delicious dishes. Therefore, manually collected data may be used as appropriate to create new dishes through collaboration between humans and AI tools.

[0189] For example, when the adjustment unit 114 adjusts the recipe for "mushroom falci" according to the user's cooking environment, if it determines that cooking step 3 (element 3): filling and cooking step 4 (element 4): plating are difficult for the cooking robot in the user's environment to handle, or if it determines that manual intervention is better, it may make the recipe incorporate manual intervention. The elements of the recipe when manual intervention is used are examined, for example, as follows.

[0190] When considering cooking processes (elements) that require manual labor, the information processing device 10 may select personnel by acquiring profiles (specialties, etc.) of assignable staff and schedule information (available time).

[0191] In considering element 3, the element review unit 115 generates n filling mixing patterns and has the assigned staff member create (cook) them. At this time, the cooking time can be obtained using a timer or the like. The element review unit 115 may also have the staff member think up filling mixing patterns. When cooking manually, cooking tends to proceed without recording the cooking process, so the agent unit 111 interacts with the staff member and hears from them about mixing points, recommended mixing patterns (rankings), etc., and stores the information. The element review unit 115 may also take images of the fillings cooked by the staff member and create a list in order of the staff member's ranking. The staff member may evaluate whether they can create the fillings according to each mixing pattern, as well as the appearance and taste of the created fillings, and input the results into the information processing device 10.

[0192] In considering element 4, the element review unit 115 creates n plating patterns and has the assigned staff create (cook) them. The element review unit 115 may also have the staff think about the plating patterns. For example, the staff may be asked to select the plate on which the food will be served. The acquisition of cooking time, interviews with the staff, and storage of the cooking process are the same as in the consideration of element 3 above. The staff may evaluate whether they can plate the food according to each plating pattern, and input the results into the information processing device 10.

[0193] The elements thus manually examined are also used in the integration unit 116 .

[0194] In addition, the cooking process of each staff member is also stored in the storage unit 150 and can be incorporated into the development story generated by the story generation unit 117, thereby visualizing the staff member's contribution.

[0195] <<5. Supplementary Information>> Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0196] The above-described embodiments may be used, for example, by chefs or culinary experts when creating new dishes.

[0197] It is also possible to create one or more computer programs for causing hardware such as a CPU, ROM, and RAM built into the information processing device 10 to perform the functions of the information processing device 10. A computer-readable storage medium storing the one or more computer programs is also provided.

[0198] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0199] The present technology can also be configured as follows: (1) An information processing system including a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to cooking processes included in the cooking information using a cooking processing learning model, integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information. (2) The information processing system described in (1), in which the control unit generates multiple pieces of cooking processing information for the cooking processes and determines feasible cooking processing information using the cooking processing learning model. (3) The information processing system described in (1) or (2), in which the cooking processing learning model is generated by reinforcement learning of simulation results based on the multiple pieces of cooking processing information generated for the cooking processes. (4) The information processing system described in (1) or (2), in which the cooking processing learning model is generated by reinforcement learning of cooking results of using a cooking robot to perform cooking processes indicated in the multiple pieces of cooking processing information generated for the cooking processes. (5) The information processing system according to any one of (1) to (4), wherein the control unit presents a plurality of pieces of cooking and processing information as candidates and generates the integrated cooking and processing information to include cooking and processing information selected by a user. (6) The information processing system according to any one of (1) to (5), wherein the instruction information includes a concept of a dish and information on ingredients desired to be used in the dish. (7) The information processing system according to (6), wherein the concept of the dish includes information on the purpose, impression, atmosphere, style, genre, theme, purpose, key points, target demographic, design, or layout of the dish. (8) The information processing system according to (6), wherein the control unit uses an ingredient pairing learning model based on the information on the ingredients included in the instruction information to generate combined ingredients suitable for combination with the ingredients.(9) The information processing system described in (8), wherein the control unit uses the concept, the ingredients, and the ingredient combinations as input data and generates the dish information using at least one of a first cooking learning model that has learned the user's dishes and a second cooking learning model that has learned the dishes of other users. (10) The information processing system described in (9), wherein the control unit is capable of performing the following processes: generating dish information in which the dish design style in the dish information generated using the first cooking learning model is changed using the second cooking learning model; and generating dish information in which the dish design style in the dish information generated using the second cooking learning model is changed using the first cooking learning model. (11) The information processing system described in (9) or (10), wherein the control unit determines the proportion of the number of dish information generated using each cooking learning model according to a set dish freshness level. (12) The information processing system according to any one of (1) to (11), wherein the control unit generates multiple pieces of the dish information, presents them to the user, and sends the dish information selected by the user to subsequent processing. (13) The information processing system according to any one of (1) to (12), wherein the control unit adjusts the dish information in response to user feedback before sending it to subsequent processing. (14) The information processing system according to any one of (1) to (13), wherein the control unit adjusts the dish information in response to the user's cooking environment before sending it to subsequent processing. (15) The information processing system according to any one of (1) to (14), wherein the control unit includes failure case data in learning data for the cooking learning model or the cooking processing learning model. (16) The information processing system according to any one of (1) to (15), wherein the cooking processing information is information on processing related to the appearance of ingredients. (17) The information processing system according to any one of (1) to (16), wherein the control unit sets personalization conditions, which are conditions for changing the cooking and processing information according to the preferences or calorie intake of an individual receiving the food, based on a user input, changes the cooking and processing information to satisfy the personalization conditions, and then generates the integrated cooking and processing information.(18) A control method including: a processor generating cooking information using a cooking learning model based on instruction information input by a user, generating cooking processing information corresponding to the cooking steps included in the cooking information, integrating the cooking processing information to generate integrated cooking processing information, and presenting the generated integrated cooking processing information. (19) A program causing a computer to function as a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user, generating cooking processing information corresponding to the cooking steps included in the cooking information, integrating the cooking processing information to generate integrated cooking processing information, and presenting the generated integrated cooking processing information.

[0200] REFERENCE SIGNS LIST 10 Information processing device 110 Control unit 111 Agent unit 112 Draft generation unit 113 Display control unit 114 Adjustment unit 115 Element review unit 116 Integration unit 117 Story generation unit 120 Communication unit 130 Operation input unit 140 Display unit 150 Storage unit 160 Voice input / output unit

Claims

1. An information processing system comprising a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to cooking steps included in the cooking information using a cooking processing learning model, integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

2. The information processing system according to claim 1, wherein the control unit generates a plurality of cooking processing information for the cooking process and determines feasible cooking processing information using the cooking processing learning model.

3. The information processing system according to claim 1, wherein the cooking and processing learning model is generated by reinforcement learning of simulation results based on multiple cooking and processing information generated for the cooking process.

4. The information processing system of claim 1, wherein the cooking processing learning model is generated by reinforcement learning of cooking results obtained by performing cooking processing indicated by multiple cooking processing information generated for the cooking process using a cooking robot.

5. The information processing system according to claim 1, wherein the control unit presents a plurality of cooking and processing information as candidates and generates the integrated cooking and processing information so as to include cooking and processing information selected by the user.

6. The information processing system according to claim 1, wherein the instruction information includes a concept for the dish and information on ingredients desired to be used in the dish.

7. The information processing system according to claim 6, wherein the concept of the dish includes information on the purpose, impression, atmosphere, style, genre, theme, objective, emphasis, target demographic, design, or layout of the dish.

8. The information processing system of claim 6, wherein the control unit uses an ingredient pairing learning model to generate a combination of ingredients suitable for combination with the ingredient based on the information about the ingredient included in the instruction information.

9. The information processing system of claim 8, wherein the control unit uses the concept, the ingredients, and the combination ingredients as input data and generates the cooking information using at least one of a first cooking learning model that has learned the user's cooking and a second cooking learning model that has learned the cooking of other users.

10. The information processing system of claim 9, wherein the control unit is capable of performing the following processes: generating cooking information in which the style of food design in cooking information generated using the first cooking learning model is changed using the second cooking learning model; and generating cooking information in which the style of food design in cooking information generated using the second cooking learning model is changed using the first cooking learning model.

11. The information processing system according to claim 9, wherein the control unit determines the ratio of the number of pieces of cooking information generated using each cooking learning model according to a set level of newness of the cooking.

12. The information processing system according to claim 1, wherein the control unit generates a plurality of pieces of recipe information, presents them to the user, and sends the recipe information selected by the user to subsequent processing.

13. The information processing system according to claim 1, wherein the control unit adjusts the cooking information in response to user feedback before sending it to subsequent processing.

14. The information processing system according to claim 1, wherein the control unit adjusts the cooking information according to the user's cooking environment before sending it to subsequent processing.

15. The information processing system according to claim 1, wherein the control unit includes failure case data in the learning data when learning the cooking learning model or the cooking processing learning model.

16. The information processing system according to claim 1, wherein the cooking and processing information is information about processing related to the appearance of ingredients.

17. The information processing system of claim 1, wherein the control unit sets personalization conditions, which are conditions for changing the cooking and processing information according to the preferences or calorie intake of the individual receiving the food, based on user input, changes the cooking and processing information to satisfy the personalization conditions, and then generates the integrated cooking and processing information.

18. A control method including: a processor generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to cooking steps included in the cooking information; integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

19. A program that causes a computer to function as a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to the cooking steps included in the cooking information; integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

Images