Information processing device, information processing method, program, and data structure of recipe data

Abstract

The present technology relates to an information processing device, an information processing method, a program, and a data structure of recipe data that make it possible to easily reproduce correct cooking work. An information processing device according to one aspect of the present technology acquires recipe data including at least reference work information indicating an operation in cooking work, and acquires sensor data obtained by measuring an operation during cooking performed by a cooking person. The information processing device also generates evaluation information pertaining to the operation performed by the cooking person on the basis of the reference work information and the sensor data, and presents the evaluation information. The present technology can be applied to a system that assists cooking in a kitchen.

Description

Information processing device, information processing method, program, and data structure of recipe data

[0001] In particular, the present technology relates to an information processing device, an information processing method, a program, and a data structure of recipe data that enable correct cooking operations to be easily reproduced.

[0002] Recipes that ordinary chefs refer to usually contain many ambiguous descriptions and expressions that assume tacit knowledge. As a result, the cooking process varies depending on the chef's skill and interpretation, as well as differences in cooking equipment and cooking environment, resulting in low reproducibility of the finished dish.

[0003] Patent Literature 1 describes a cooking assistance system that defines the target state of ingredients in each cooking step by parameters based on sensor data that can be measured during cooking, and enables the reproduction of a dish using recipe data that includes such parameters. During the reproduction cooking, the chef is guided through the work content of each step using images, audio, etc. By providing the chef with feedback on information obtained from sensor data measured during the reproduction cooking and controlling the heating of ingredients until they reach the state defined as the target state, it becomes possible to reproduce the same dish as the chef made.

[0004] International Publication No. 2023 / 182197

[0005] In the cooking support system, the chef who performs the re-creation of the dish follows the navigation to mix ingredients, etc., but there are some dishes whose re-creation level changes depending on the mixing method, etc.

[0006] It is difficult to convey the correct mixing method to the chefs who are recreating the recipe. Even the chefs who are recreating the recipe do not know how to mix the ingredients to achieve the desired result. These issues are thought to be due to the following reasons: - The mixing action cannot be recorded and expressed quantitatively. - The chef does not know the correct mixing method to achieve the desired result. - It is not possible to objectively (quantitatively) determine whether the ingredients have been mixed correctly during cooking.

[0007] The present technology has been developed in light of these circumstances, and aims to make it possible to easily reproduce correct cooking operations.

[0008] An information processing device according to a first aspect of the present technology includes a first acquisition unit that acquires recipe data including at least reference work information indicating actions in cooking work, a second acquisition unit that acquires sensor data measuring the actions of a chef while cooking, a control unit that generates evaluation information of the chef's actions based on the reference work information and the sensor data, and a presentation unit for the evaluation information.

[0009] The information processing device of the second aspect of the present technology includes an information processing unit that generates standard work information indicating the actions taken during cooking based on sensor data measured by the chef during cooking, and generates recipe data that includes at least the standard work information.

[0010] In a first aspect of the present technology, recipe data including at least reference task information indicating actions in a cooking task is acquired, sensor data measuring a chef's actions during cooking is acquired, and evaluation information of the chef's actions is generated based on the reference task information and the sensor data, and the evaluation information is presented.

[0011] In a second aspect of the present technology, reference work information indicating the actions in the cooking work is generated based on sensor data measuring the actions of a chef during cooking, and recipe data including at least the reference work information is generated.

[0012] 1 is a diagram showing the overall flow of the recording / reproducing process of a cooking process. A diagram showing an example of cooking based on recipe data. A diagram showing an example of the configuration of a cooking assistance system. A diagram showing an example of information included in recipe data. A diagram showing an example of reference stirring operation information. A block diagram showing an example of the functional configuration of a cooking assistance system. A block diagram showing an example of the functional configuration of a cooking assistance system. A flowchart explaining a series of processes for recording cooking. A flowchart explaining a series of processes for creating recipe data. A flowchart explaining a series of processes for reproducing cooking. A diagram showing an example of a navigation screen. A flowchart explaining stirring operation navigation processing. A diagram showing an example of a navigation screen for stirring operation. A diagram showing switching of the display of arrow images. A diagram showing an example of the staying rate for each area. A diagram showing an example of a presentation of a mixing range. A diagram showing an example of dividing a mixing range. A diagram showing another presentation example of navigation for stirring operation. A diagram showing an example of navigation. A diagram showing an example of the configuration of a cooking assistance system. A diagram showing an example of the configuration of a cooking assistance system. A block diagram showing an example of the configuration of a computer.

[0013] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the present technology 2. Overview of cooking assistance system 3. Overall flow of recording / reproducing process 4. System configuration 5. Details of recording / reproducing process 6. Example of navigation based on reference mixing task information 7. Modified example

[0014] <<Outline of the Present Technology>> In a cooking assistance system applying the present technology, whether or not a predetermined cooking action has been correctly reproduced is determined by comparing data on the reference action with data obtained by sensing the actions of the chef performing the replicated cooking. Such determination is made repeatedly during the replicated cooking and during practice prior to the replicated cooking.

[0015] Taking the action of mixing ingredients as an example, sensor data such as the speed, stroke (starting point and ending point), direction, strength, and the way the spatula is used for mixing are acquired using sensors installed in cooking equipment and utensils.Based on the data obtained by sensing the actions of the chef performing the recorded cooking, reference action data is generated and used when reproducing the cooking.

[0016] Before starting the cooking task, the chef is given guidance on what to do next. The information that guides the chef through the steps is presented in the following ways: - Using video and audio - Displaying the trajectory of the steps in real time

[0017] For example, thickening is done using water-soluble potato starch in Japanese and Chinese cuisine. When thickening with water-soluble potato starch, the degree of reproducibility generally depends on the ratio of potato starch to water and whether it is thoroughly mixed. The degree of reproducibility also depends on the timing of adding the water-soluble potato starch, temperature control, and rapid stirring. Therefore, in the cooking process of thickening with water-soluble potato starch, guidance is provided on how to prepare the water-soluble potato starch, as well as the appropriate timing for adding the water-soluble potato starch and the appropriate stirring method. This makes it possible to achieve the desired thickness.

[0018] During the cooking process, the chef receives the following feedback based on the comparison results using the sensor data: - A judgement result (pass / fail) on whether the cooking action is performed correctly - Identification of actions that require practice and suggestions for improvement

[0019] Focusing on the stirring operation, the process during the replicated cooking will follow the following flow. Details will be given later. 1. Output instructions to switch cooking processes (the cook adds ingredients according to the switching instructions) 2. Present navigation for the stirring operation (the cook begins stirring) 3. Sense the cook's movements as they stir 4. Match the sensor data of the cook's movements with reference stirring operation information, which serves as an evaluation index 5. Compare the reference stirring operation information with the cook's movement data to determine the difference 6. Present feedback according to the difference 7. Repeat steps 2 to 6 above 8. When the stirring operation is complete, present the cook with a pass / fail judgment result

[0020] Some cooking tasks, such as adding ingredients, stirring, thickening, etc., vary from person to person. Reproducing tasks that vary is difficult simply by controlling the cooking equipment so that the state of the ingredients reaches a target state.

[0021] By defining a reference action in advance, evaluating the state of the chef's actions based on sensor data acquired during the replicated cooking, and providing feedback based on the state of the chef's actions, the degree of reproducibility of the cooking work can be improved. By improving the degree of reproducibility of the cooking work, it is possible to improve the degree of reproducibility of the finished dish.

[0022] In the following, the case where the cooking task is a mixing task will be mainly described, but similar processing is performed for other tasks as well.

[0023] <<Outline of the Cooking Assistance System>> <Expected Usage Scenarios> This technology relates to a cooking assistance system that assists cooking in environments such as kitchens. This technology can be applied to cooking assistance systems that use heating cookers such as induction stoves.

[0024] A cooking assistance system using this technology is equipped with various sensor devices that measure the cooking state. Based on the measurement results from the sensor devices, time-series sensor data indicating the state of ingredients during the cooking process is recorded.

[0025] In addition, based on recipe data created using recorded data including sensor data, cooking support is provided to a chef who is trying to recreate a dish, so that the finished dish can be accurately reproduced.

[0026] The cooking process to replicate the final dish is carried out with some operations, such as adjusting the heat of the cooking appliance, automated. The chef, who is the user of the cooking support system, performs tasks such as adding ingredients and stirring by following the navigation provided by the UI.

[0027] In addition, "cooking" refers to the finished product that is produced through cooking. Cooking refers to the process of cooking food and the act (task) of cooking food.

[0028] <Cooking Process> FIG. 1 is a diagram showing the overall flow of the recording / reproducing process of a cooking process.

[0029] Details will be described later, but after "normal recipe creation", "record cooking" is performed as indicated by the tip of arrow A1 in FIG. 1, and sensor data indicating the measurement results in the record cooking is recorded.

[0030] As indicated by the tip of arrow A2, "recipe data creation" is performed based on the recorded data, including the recorded sensor data, to create recipe data. Recipe data is data that can be processed by a computer. Information from the previous normal recipe is also used to create the recipe data.

[0031] The recipe data describes the details of each cooking step until the dish is completed as follows: Normally, one dish is completed through multiple cooking steps as shown in FIG.

[0032] 1. The entire cooking process consists of multiple cooking steps that are arranged in a chronological order. 2. For each cooking step, a cooking task and a target state are defined, as shown in the speech bubbles in Figure 2. 3. Cooking tasks indicate operations performed on cooking equipment and ingredients. 4. The target state indicates the conditions for completing a cooking step and moving on to the next cooking step. I. The conditions for moving on to the next cooking step are defined using numerical indicators based on data that the cooking assistance system can measure. II. The timing at which the conditions for moving on to the next cooking step are reached is expressed so that it can be uniquely identified on time-series data that the cooking assistance system can detect. 5. The target state is defined using one of the TTW (Time / Temperature / Weight) values ​​that the cooking assistance system can measure. The TTW value is the value of the target parameter, which defines the target state. The target parameter includes Temperature and Weight, which are sensor data that the cooking assistance system can measure, as well as Time, which the cooking assistance system can measure. I. Time: Elapsed time from the start of the cooking step [sec] II. Temperature: Temperature of the object to be heated or the heating medium [°C] III. Weight: Weight of the object to be heated [g] 6. The initial state is defined for each ingredient used in the cooking process: I. Temperature of the ingredient when it is put into the cooking utensil (e.g., pot) [℃] II. Weight of the ingredient to be put into the cooking utensil (e.g., pot) [g]

[0033] In this way, in a cooking assistance system to which the present technology is applied, recipe data is created for all cooking steps that indicate the target state and initial state defined using indices that can be interpreted without being subject to personal judgment. As will be described later, various additional information other than the information indicating the target state and initial state is held in the recipe data.

[0034] Returning to the explanation of FIG. 1, as indicated by the tip of arrow A3, "reproduction cooking" is performed using the recipe data, and the dish is reproduced.

[0035] The replicated cooking is performed according to the following rules: 1. Cooking is performed at the start of each cooking process. 2. The next cooking process begins when the target state of each cooking process is reached. 3. The initial state is matched when each ingredient is added.

[0036] As shown by the arrows A4 and A5 in Figure 1, a reproducibility evaluation is performed using the dishes prepared by the recorded cooking and the dishes prepared by the replicated cooking. By performing the replicated cooking in this manner, variations in the cooking process due to differences in the chef's skills and interpretation, or differences in cooking utensils and cooking environment, can be reduced, making it possible to consistently prepare dishes with a high degree of reproducibility.

[0037] Hereinafter, the cook who performs the recording cooking will be referred to as the chef, and the cook who performs the re-creation cooking will be referred to as the user, as appropriate. If the chef is the first cook, the user will be the second cook. A cook who wants to recreate the cooking of a top chef becomes a user of the cooking assistance system and performs the re-creation cooking. When there is no need to distinguish between the cook who performs the recording cooking and the cook who performs the re-creation cooking, they will simply be referred to as the cook.

[0038] <<Overall Flow of Recording / Reproducing Process>> Each process in Fig. 1 will be described. The recording / reproducing process of a cooking process consists of the following processes: "Normal recipe creation," "Record cooking," "Recipe data creation," "Reproduced cooking," and "Reproduction evaluation."

[0039] 1. Regular recipe creation Regular recipe creation is a process in which a person creates a regular recipe, which is the recipe for the dish that is the target of the recording / reproduction process. A regular recipe containing content that can be understood by the chef who will be recording the cooking is created using any media or expression format.

[0040] 2. Record Cooking Record cooking is a process in which a chef cooks according to a normal recipe, and a recording system records data on the cooking process. The recording system is implemented by a cooking support system. Time series data on TTW values ​​measured by sensors is recorded as recorded data. The food produced by record cooking is called a "record sample."

[0041] 3. Recipe Data Creation Recipe data creation is a process in which the cooking assistance system creates recipe data to be used in the reproduction cooking based on recorded data. Recipe data creation is also performed based on data entered by a person as appropriate. The recipe data is recorded as digital data and used as input for the reproduction system during the reproduction cooking. The recipe data is also used to present human-understandable information as needed.

[0042] 4. Reproduction Cooking Reproduction cooking is a process in which the reproduction system controls cooking equipment according to recipe data and presents information about cooking tasks to the user based on the recipe data. The user performs cooking tasks in the reproduction system according to the presented information. The reproduction system is also realized by a cooking assistance system. The dish created by reproduction cooking is called a "reproduction sample."

[0043] 5. Reproducibility Evaluation Reproducibility evaluation is a process of comparing a recorded sample prepared by the recorded cooking with a reproduced sample prepared by the reproduced cooking, and evaluating the reproducibility of the reproduced sample.

[0044] <<System Configuration>> <Configuration Example of Cooking Assistance System> FIG. 3 is a diagram showing a configuration example of a cooking assistance system.

[0045] The cooking assistance system 1 shown in Fig. 3 as an information processing device is a type of so-called smart cooking appliance. The cooking assistance system 1 assists in cooking using cooking utensils such as pots and frying pans. Cooking utensils are appliances used by cooks, such as pots, frying pans, and spatulas. In addition to cooking utensils, cooking appliances also include electrically driven appliances such as induction stoves and microwave ovens.

[0046] The cooking assistance system 1 is configured by connecting an IH stove 11 with a display 21, a speaker 22, a camera 23, a microphone 24, lighting equipment 25, a cooking environment measuring device 26, and a router 27. Each device is connected to the IH stove 11 by wired communication or wireless communication such as wireless LAN.

[0047] The IH stove 11 is an electromagnetic cooker that heats cooking utensils 12 such as pots and frying pans. In Fig. 3, a pot is shown as the cooking utensil 12. A spatula used to stir the contents of the pot is also shown as cooking utensil 12A.

[0048] As shown in the balloon in FIG. 3, the IH stove 11 includes a processor 31, a temperature sensor 32, a force sensor 33, and a storage 34 built in.

[0049] The processor 31 is a computing unit that controls the heating of the cookware 12. The processor 31 also controls each device that constitutes the cooking assistance system 1 and communicates with external devices, a web server 42, etc. Communication with the web server 42 is performed via the router 27.

[0050] Furthermore, the processor 31 detects instructions and actions of the chef input via an input UI such as the microphone 24. The processor 31 also controls the shooting conditions of the camera 23 by adjusting the brightness of the lighting equipment 25, for example.

[0051] Temperature sensor 32 measures the temperature of the pot or the temperature of the food being cooked in the pot. In this case, the pot serves as the heating medium, and the food being cooked in the pot serves as the heated object. For example, a radiation thermometer (infrared sensor) that is highly responsive and capable of non-contact measurement is used as temperature sensor 32. Temperature sensor 32 may also be configured to measure both the temperature of the pot and the temperature of the food being cooked.

[0052] The force sensor 33 measures the force and weight applied to a cooking utensil such as a pot. For example, a piezoelectric sensor is used as the force sensor 33. The force sensor 33 is provided, for example, below the top plate of the IH stove 11.

[0053] The display 21 displays various screens under the control of the processor 31. For example, the display 21 is provided near the induction cooker 11. Various information is presented to the cook using the display on the display 21. The display 21 is configured with a video display device such as a liquid crystal monitor or a projector. A wearable display such as AR glasses may also be used as the display 21.

[0054] The speaker 22 outputs various sounds under the control of the processor 31. For example, sounds that serve as navigation for cooking operations are presented to the user using the speaker 22.

[0055] The camera 23 captures an image of the food being cooked in the pot. The camera 23 is installed, for example, above the induction cooker 11 with its angle of view directed toward the pot. The image captured by the camera 23 is transmitted to the processor 31 in real time. For example, a normal visible light camera is used as the camera 23. In addition to a visible light camera, a thermography camera, a hyperspectral camera, or a depth-sensing camera can also be installed as the camera 23. The camera 23 also captures an image of the cook who is cooking. The camera 23 is made up of multiple cameras, including a camera that captures the image inside the pot and a camera that captures the cook.

[0056] In addition to the microphone 24, a smartphone 41 is used as a device for inputting various pieces of information to the cooking assistance system 1 described above. Other mobile devices such as a tablet terminal may also be used for input. Information input by the chef using the smartphone 41 is acquired by, for example, the processor 31. The processor 31 controls each device as appropriate in response to the input by the chef.

[0057] The cooking environment measuring instrument 26 is a measuring instrument equipped with sensor devices such as a temperature sensor, a humidity sensor, an acoustic sensor, and an air pressure sensor. The cooking environment measuring instrument 26 measures the temperature, humidity, environmental sound, and air pressure of the cooking environment. Sensor data indicating the measurement results of the cooking environment measuring instrument 26 is also supplied to the processor 31 and used for various processes.

[0058] In this way, various sensor devices are provided in the cooking assistance system 1. In the cooking assistance system 1, the camera 23 captures an image of, for example, a chef mixing ingredients using a spatula (cooking utensil 12A).

[0059] In the cooking assistance system 1, the position and angle of the spatula are estimated based on the image captured by the camera 23. For example, an inference model generated by machine learning using a data set of various spatula shapes is prepared in the processor 31. The inference model, which receives the image captured by the camera 23 as input and outputs information on the position and angle of the spatula, is prepared in the processor 31 and is used to estimate the position and angle.

[0060] In the cooking assistance system 1, the trajectory and speed of the mixing motion are acquired based on the position and angle of the spatula. Furthermore, the start and end points of the mixing motion and the position in the depth direction are estimated based on the position and angle of the spatula. An AR marker or the like may be attached to the spatula, and the position and angle of the spatula may be measured by analyzing an image of the AR marker.

[0061] A depth sensor may be installed on the ceiling side of the cooking environment. Using the measurement results from the depth sensor, it is possible to obtain information on the height direction of cooking utensils such as spatulas, and based on the obtained information, it is possible to estimate the three-dimensional movement of the cooking utensils. By using the depth information in addition to the position information of the cooking utensils, it is possible to estimate whether the cook is mixing the surface part of the pot or the bottom part of the pot.

[0062] Furthermore, a force sensor may be provided in the IH stove 11 serving as a cooking device. By using a force sensor, it is possible to obtain data on the strength (intensity) of the chef's movements.

[0063] The chef's strength of force may also be measured using other sensors, such as a piezoelectric sensor, optical sensor, strain gauge, capacitance sensor, or weight sensor. The sensor used to measure the strength of force may be provided on a cooking utensil, such as a pot, spatula, or chopsticks, rather than on the IH stove 11. Data indicating the chef's strength of force measured using a force sensor or the like is managed in synchronization with images captured by the camera 23, for example. By synchronizing the strength of force information with the images captured by the camera 23, it is possible to improve the accuracy of the start and end points of the mixing action.

[0064] By providing an acceleration sensor, angular velocity sensor, position sensor, attitude sensor (IMU), etc. to the cooking utensil 12A such as a spatula, it becomes possible to estimate the mixing direction and speed in three dimensions.

[0065] <Information Included in Recipe Data> FIG. 4 is a diagram showing an example of information included in recipe data.

[0066] As shown in FIG. 4, the recipe data includes information on cooking conditions and information on cooking steps.

[0067] The cooking conditions are conditions that do not change throughout the cooking process. The information that constitutes the information about cooking conditions includes information about the cooking appliance, the cook, and the ingredients.

[0068] Information about cooking equipment includes stove specifications, pot specifications, individual characteristics of the pot, type / specification of other utensils, placement of the pot (when fixed), and information about the date, time, and location of cooking. For example, stove specifications represent the specifications of the induction stove 11 used for recording cooking. Information about the chef includes the attributes of the recording chef and information about the details of pre-training. Information about ingredients includes the genre and name of the dish, the finished amount, the type and amount of ingredients, and how the ingredients were cut (preparation method).

[0069] A cooking process is a state or task defined as a cooking process. Information about the cooking process is appropriately expressed as information on a time axis. Information about the cooking process includes information about the process configuration, initial state, target state, transient state, and cooking task.

[0070] The information about the process configuration includes information about the overall configuration of the cooking process. The information about the process configuration indicates, for example, the number of cooking steps, the order, the ingredients used in each cooking step, and the cooking utensils used. The information about the process configuration includes ingredient information that indicates the ingredients used in each cooking step.

[0071] The information about the initial state includes the initial temperature of the stove, the initial temperature of the pot, the initial temperature of the ingredients, and the weight of the ingredients added.

[0072] The information about the target state includes the target index value of each process and the target image of each process. For example, the information about the target index value of each process represents one of the TTW values ​​used to determine the end timing of the cooking process.

[0073] The information about the transient state includes a TTW profile showing the time series of TTW values, an image of the inside of the pot during cooking, the position / movement of the pot during cooking, and information about the ambient temperature / humidity.

[0074] The information about cooking tasks includes the stove's heat setting, the order in which ingredients are added (when multiple ingredients are added), how to add ingredients, how to stir ingredients, how to put on and take off the pot lid, and other information about the tasks. The information about cooking tasks also includes reference task information. The reference task information is information that serves as a standard for actions when working with ingredients. The reference task information represents the actions that should be aimed for when recreating cooking.

[0075] For example, the information about the mixing task includes reference mixing task information, which is reference task information for the mixing task. The reference mixing task information includes information indicating the feature amounts of the actions that define the reference mixing task.

[0076] 5 is a diagram showing an example of reference stirring work information. Information recorded in reference stirring work information If the cooking can be reproduced by mixing in the same way as the chef did during the recorded cooking, the chef's mixing is the reference stirring work. The reference stirring work is defined by characteristic quantities such as the speed, stroke, direction, and strength of the mixing action performed by the chef using a spatula.

[0077] 5, the reference mixing work information records information indicating the speed, stroke, direction, and strength of the movement at each time. Information indicating at least one of the speed, stroke, direction, and strength of the movement may be recorded in the reference mixing work information, rather than information indicating all of them. Information indicating other characteristic quantities may also be recorded in the reference mixing work information.

[0078] - Action conversion: Reproducing the chef's mixing technique during the recorded cooking may require skilled cooking skills. In such cases, the chef's mixing technique is converted into a simpler action, and information specifying the converted action is recorded in the reference mixing work information. For example, even if a mixing technique is simpler than the chef's mixing technique during the recorded cooking, it may be possible to reproduce the same thickness as the thickness during the recorded cooking. In this case, information indicating a simple action that can reproduce the same thickness is recorded in the reference mixing work information, rather than information indicating the chef's mixing technique as is.

[0079] For example, if a dish prepared using the chef's mixing method is compared with a dish prepared using a simple converted action and reproducibility is confirmed, the simple converted action is used as the action in the reference mixing task. Which action can be used as the simple converted action is determined, for example, during reproducibility evaluation using an evaluation method such as the three-point discrimination method.

[0080] This makes it possible for a user with inexperienced cooking skills to reproduce the actions of the chef during the recorded cooking. An inference model for action conversion used for converting such actions may be generated in advance by machine learning and prepared in the cooking assistance system 1. The inference model for action conversion is configured using a machine learning model such as a neural network that receives information representing the actions of the chef during the recorded cooking as input and outputs information representing simple actions.

[0081] Feature extraction: When sufficient sensor data is available for actions that meet and do not meet the pass criteria for a cooking process, the feature data for actions that meet the pass criteria can be extracted by analyzing the sensor data. Sensor data such as image data is analyzed, and data that is common to the sensor data for actions that meet the pass criteria is extracted. Which actions meet the pass criteria is also determined using an evaluation method such as three-point discrimination.

[0082] The actions defined by the extracted feature amounts are the actions required to reproduce the chef's cooking. Information indicating the extracted feature amounts is recorded in the reference mixing task information.

[0083] Others: A standard stirring operation for each cooking process is set based on the timing of adding ingredients, starting heating, changing the heating temperature, etc. A standard stirring operation defined by different feature quantities is set for each cooking process as appropriate. As feature quantities for defining the standard stirring operation, various feature quantities that can be compared with information obtained based on sensor data that can be measured during the stirring operation in the replicated cooking are used.

[0084] An operation that is always indicated by the same characteristic value at each time point in the cooking process may be set as the reference stirring operation, in which case, for example, the average stirring speed is used as the characteristic value of the reference stirring operation.

[0085] Furthermore, an action indicated by a characteristic amount that changes at a constant rate over time, such as gradually increasing the mixing speed, may be set as the reference mixing task.

[0086] A repetitive action, such as performing the same action multiple times in a predetermined direction at a constant speed, may be set as the reference mixing task. In this case, the average mixing speed, rhythm, and number of repetitions are used as the characteristic quantities of the reference mixing task.

[0087] In this way, data such as average values ​​indicating constant movements in the cooking process and time-series data indicating changes in movements in the cooking process are used as feature quantities to define the reference stirring task. The reference stirring task information shown in Fig. 5 is data composed of time-series data indicating changes in movements in the cooking process.

[0088] Recipe data including such reference task information is generated in the cooking assistance system 1. Furthermore, when reproducing a cooking recipe, various navigational instructions are presented to the user using the reference task information.

[0089] <Functional Configuration> FIGS. 6 and 7 are block diagrams showing an example of the functional configuration of the cooking assistance system 1. As shown in FIG.

[0090] As shown in Figures 6 and 7, the cooking assistance system 1 includes a recording processing unit 101, a recipe data creation unit 102, and a reproduction processing unit 103. At least a part of the components shown in Figures 6 and 7 is realized by the processor 31 executing a predetermined program. Details of the processing performed by each unit will be described later.

[0091] Regarding the recording processing unit 101: The recording processing unit 101 in Fig. 6 performs processing during recording cooking. A recording system is realized by the recording processing unit 101. The recording processing unit 101 is composed of a cooking environment measurement unit 111, a sensor data acquisition unit 112, a user input detection unit 113, and an information recording unit 114.

[0092] The cooking environment measurement unit 111 controls the cooking environment measuring device 26 to measure the environment in which the cooking to be recorded is performed, for example, at the start of the cooking to be recorded. The cooking environment information measured by the cooking environment measurement unit 111 is supplied to the information recording unit 114.

[0093] The sensor data acquisition unit 112 acquires sensor data used to generate recipe data, for example, after record cooking has started. For example, during cooking in record cooking, time series data is acquired, such as time measured by the timer inside the processor 31, temperature measured by the temperature sensor 32, and force and weight measured by the force sensor 33 during the cooking. Time series data of sensor data obtained by measuring the chef's movements is also acquired. The sensor data resulting from the measurement of the chef's movements includes image data such as RGB images and depth images, and data such as acceleration and angular velocity measured by sensors mounted on cooking utensils such as spatulas. The sensor data acquired by the sensor data acquisition unit 112 is supplied to the information recording unit 114.

[0094] The user input detection unit 113 detects inputs made by the cook using input devices such as the microphone 24 or operation buttons, and outputs information representing the content of the input to the information recording unit 114. For example, if the cook or an assistant presses an operation button at a timing that is important for creating recipe data, information representing this is output to the information recording unit 114, and a timestamp is recorded. The recorded timestamp is referenced in the recipe data creation process and is used for organizing cooking steps, etc. The timing of recording may be input not by operating an operation button, but by the cook's actions or statements during the recorded cooking.

[0095] The information recording unit 114 generates record data by recording, in a predetermined format, the cooking environment information supplied from the cooking environment measurement unit 111 and the sensor data supplied from the sensor data acquisition unit 112. When the recording cooking is completed, record data is generated that records the sensor data and other data acquired from the start to the end of the recording cooking. The record data generated by the information recording unit 114 is supplied to the recipe data creation unit 102 at the timing of recipe data creation, etc.

[0096] Recipe Data Creation Unit 102 The recipe data creation unit 102 is made up of a process organization unit 121 , a target state setting unit 122 , a cooking operation setting unit 123 , an information recording unit 124 , and a recipe data management unit 125 .

[0097] The process organization unit 121 organizes the processes by dividing the entire cooking process of the recorded cooking into multiple cooking processes. Information about each cooking process organized by the process organization unit 121 is supplied to the target state setting unit 122, the cooking task setting unit 123, and the information recording unit 124.

[0098] The target state setting unit 122 sets a target state for each cooking process set by the process organization unit 121. Information on the target states set by the target state setting unit 122 is supplied to the cooking task setting unit 123 and the information recording unit 124. The target state is set appropriately according to the cooking task set by the cooking task setting unit 123.

[0099] The cooking operation setting unit 123 sets the content of the cooking operation for each cooking process set by the process organization unit 121. For example, the cooking operation is set as an operation for bringing the state of the heated object, such as the ingredients used in each cooking process, into a state defined as a target state. The cooking operation setting unit 123 also sets multiple types of feature quantities that define the reference cooking operation for each cooking process. Information indicating the content of the cooking operation set by the cooking operation setting unit 123 is supplied to the information recording unit 124 and the target state setting unit 122.

[0100] The information recording unit 124 records the record data supplied from the recording processing unit 101 and the information supplied from the process organization unit 121, the target state setting unit 122, and the cooking task setting unit 123 in a predetermined format, thereby creating recipe data including each piece of information. The recipe data includes information on the cooking environment measured during the recorded cooking, time-series sensor data, information on each cooking step, the target state of the cooking step, and information indicating the content of the cooking task. In other words, the recipe data has a data structure in which information regarding each cooking step includes at least ingredient information indicating the ingredients used in the cooking step and reference task information generated based on sensor data measuring the actions of the chef as the first cook during cooking and indicating characteristic quantities of the actions performed in the cooking task using the ingredients. The recipe data created by the information recording unit 124 is provided to the recipe data management unit 125.

[0101] The recipe data management unit 125 manages the recipe data created by the information recording unit 124 by recording it in, for example, the storage 34. In the storage 34, recipe data for various dishes is recorded for each dish.

[0102] A DB (Data Base) of recipe data may be prepared in the Web server 42, and the recipe data may be managed in the Web server 42. In this case, the recipe data management unit 125 transmits the recipe data created by the information recording unit 124 to the Web server 42, and the Web server 42 manages the recipe data.

[0103] In this way, the recipe data creation unit 102 functions as an information processing unit that generates recipe data based on sensor data measured during recorded cooking. The recipe data includes at least reference task information including information indicating feature quantities used as evaluation indexes for the actions of a user reproducing the chef's cooking, as well as ingredient information. From the user's perspective, the reference task information is information generated based on sensor data measured during cooking by another chef.

[0104] 7 performs processing during the reproduction cooking. The reproduction system is realized by the reproduction processing unit 103. The reproduction processing unit 103 is composed of a recipe data acquisition unit 131, a cooking environment measurement unit 132, a sensor data acquisition unit 133, a user input detection unit 134, a control unit 135, and an information presentation unit 136.

[0105] The recipe data acquisition unit 131 acquires recipe data for the dish to be replicated from the storage 34 or the web server 42. The recipe data acquisition unit 131 functions as a first acquisition unit that acquires recipe data generated based on sensor data measured during the recorded cooking. The recipe data is acquired, for example, when the dish to be replicated is selected by the chef. The recipe data acquired by the recipe data acquisition unit 131 is supplied to the control unit 135.

[0106] The cooking environment measurement unit 132 controls the cooking environment measuring device 26 to measure the environment in which the reproduction cooking is performed, for example, at the start of the reproduction cooking. Information on the cooking environment measured by the cooking environment measurement unit 132 is supplied to the control unit 135.

[0107] The sensor data acquisition unit 133 acquires sensor data measured by each sensor, for example, after the start of a reproduction cooking. For example, during the reproduction cooking, time series data is acquired, such as the time measured by the timer inside the processor 31, the temperature measured by the temperature sensor 32, and the force and weight measured by the force sensor 33 during the cooking. Time series data of sensor data obtained by measuring the movements of the user performing the reproduction cooking is also acquired. The sensor data, which is the measurement result of the user's movements, includes image data such as RGB images and depth images, and data such as acceleration and angular velocity measured by sensors mounted on cooking utensils such as spatulas. The sensor data acquisition unit 133 functions as a second acquisition unit that acquires sensor data measured during cooking by the user reproducing the chef's cooking. The sensor data acquired by the sensor data acquisition unit 133 is supplied to the control unit 135.

[0108] The user input detection unit 134 detects inputs made by the chef via input devices such as the microphone 24 and operation buttons. For example, the chef inputs the timing of transitions between cooking steps. Information indicating the content of the input detected by the user input detection unit 134 is supplied to the control unit 135.

[0109] The control unit 135 performs various processes to support the reproduction of cooking based on the recipe data supplied from the recipe data acquisition unit 131. For example, the control unit 135 controls the IH stove 11 to control heating of the object to be heated. The heating control is performed in accordance with the cooking environment measured by the cooking environment measurement unit 132 and the sensor data acquired by the sensor data acquisition unit 133. The control unit 135 controls the presentation of information by the information presentation unit 136.

[0110] The control unit 135 also controls the progress of the reproduction cooking based on the recipe data. The progress of the reproduction cooking is triggered by appropriate input from the chef.

[0111] For example, when the target state is defined by the elapsed time from the start of the target cooking process (the cooking process currently being executed), the control unit 135 starts cooking the next cooking process when the elapsed time from the start of the cooking process reaches the time defined as the target state.

[0112] When the target state is defined by the temperature of the object to be heated or the heating medium, the control unit 135 starts cooking the next cooking process when the temperature measured by the temperature sensor 32 reaches the temperature defined as the target state.

[0113] When the target state is defined by the weight of the object to be heated, the control unit 135 starts cooking the next cooking process when the force and weight measured during the operation by the force sensor 33 reach the weight defined as the target state.

[0114] The information presentation unit 136 performs navigation to assist the chef in replicating the cooking based on the recipe data acquired by the recipe data acquisition unit 131. For example, the information presentation unit 136 displays a navigation screen including information indicating the content of the cooking work and information indicating the target state on the display 21. The presentation of various pieces of information to assist the chef in replicating the cooking is also performed according to the cooking environment measured by the cooking environment measurement unit 132 and the sensor data acquired by the sensor data acquisition unit 133.

[0115] Furthermore, the information presenting unit 136 displays information indicating the user's action status for each evaluation index on the navigation screen during the reproduction cooking. As will be described later, information indicating the status of each of the multiple evaluation indexes is evaluated based on the reference task information and presented to the user.

[0116] <<Details of Recording / Reproducing Process>> The details of the recording / reproducing process of the cooking process will be described.

[0117] The cooking process recording process and the reproduction process may be performed in the same device (cooking assistance system), or may be performed in different devices. Here, we will explain the case where the cooking process recording process and the reproduction process are performed in the same device, the cooking assistance system 1 shown in Figure 3.

[0118] <Record Cooking> A series of processes for record cooking will be described with reference to the flowchart in Figure 8. The record cooking process shown in Figure 8 is performed, for example, after creating a normal recipe. Note that if record cooking can be performed stably, creating a normal recipe can be omitted. For example, when a chef is preparing a dish that he or she is accustomed to making, the chef can perform record cooking stably, so a normal recipe is not necessary.

[0119] Step S1: Creating a new recipe The information recording unit 114 (FIG. 6) of the recording processing unit 101 creates new recipe data (file) for the dish to be recorded. When creating new recipe data, the information recording unit 114 records some of the information about the cooking conditions, for example, based on input by the chef. The chef will enter information about himself and about the ingredients using, for example, a screen displayed on the display 21.

[0120] Step S2: Cooking environment measurement The cooking environment measurement unit 111 measures the cooking environment and acquires information about the specifications and characteristics of the cooking appliances. For example, the cooking environment measurement unit 111 acquires information about the specifications of the IH stove 11, the pots that are the cooking utensils 12, and other appliances from a database provided in the storage 34 or the web server 42.

[0121] In the DB of the storage 34 and the web server 42, information on appliances such as induction cooktops that are compatible with the cooking assistance system 1 is provided by manufacturers and managed in association with model number information. The cooking environment measurement unit 111, for example, acquires the model number information of each appliance automatically or based on input by the chef via a UI such as a screen displayed on the display 21, and acquires information about the cooking appliances by referencing the DB based on the acquired model number information. The information acquired by the cooking environment measurement unit 111 is recorded by the information recording unit 114 as information about the cooking appliances. When measuring the cooking environment, information on the stove specifications, pot specifications, individual characteristics of the pots, types / specifications of other appliances, and the placement of the pots (when fixed) is recorded.

[0122] Step S3: Start of cooking When cooking starts, such as immediately before heating begins, the information recording unit 114 records information about the initial state of the cooking appliance based on the information measured by the sensor data acquisition unit 112. For example, information about the initial temperatures of the IH stove 11 and the pot is recorded.

[0123] The initial temperatures of the IH stove 11 and the pot are measured using the temperature sensor 32 installed in the cooking assistance system 1 and acquired by the sensor data acquisition unit 112. For example, the temperature of the cooking table in contact with the pot is recorded as the temperature of the IH stove 11. If measurable, the temperature inside the IH stove 11 is also recorded.

[0124] Step S4: During cooking During cooking, the information recording unit 114 chronologically records various information, such as sensor data measured by sensors installed in the cooking assistance system 1 and acquired by the sensor data acquisition unit 112. For example, information such as the initial temperature of the ingredients, the weight of the ingredients added, the TTW profile, images of the inside of the pot during cooking, the position / movement of the pot during cooking, the ambient temperature / humidity, and the stove heat setting is recorded. In addition, images of the chef's movements and data such as acceleration and angular velocity measured by sensors installed in cooking utensils such as spatulas are recorded.

[0125] Here, the recording of main information will be described.

[0126] Information about the heat setting of the IH stove 11 is acquired and recorded as cooking task information. In this case, the sensor data acquisition unit 112 detects and acquires the value of the heat setting set by the chef by operating an operation panel or the like provided as a UI of the cooking assistance system 1.

[0127] A TTW profile, which is time-series data of time / temperature / weight, is generated by the sensor data acquisition unit 112 and recorded as transient state information. The time constituting the TTW profile is measured by a real-time clock built into the processor 31. As the temperature constituting the TTW profile, for example, the temperature of the bottom of a pot is measured by a temperature sensor 32 built into the IH stove 11. As the weight constituting the TTW profile, for example, the weight of ingredients in the pot is measured by a force sensor 33 built into the IH stove 11.

[0128] The images of the inside of the pot are acquired by the sensor data acquisition unit 112 and recorded as information on the transient state. For example, images of the inside of the pot, generated based on images taken by the camera 23, are recorded in chronological order.

[0129] The initial temperature of the ingredients is acquired and recorded as initial state information. The initial temperature of the ingredients is the temperature of the ingredients at the time (just before) they are added to the pot. For example, the temperature of the ingredients is measured by a chef using a cooking thermometer.

[0130] The weight of the ingredients added is acquired and recorded as initial state information. Instead of the weight in a normal recipe, for example, the weight of the ingredients actually added to the pot and the timing of adding them are measured by the force sensor 33 and used for recording.

[0131] The above information is recorded repeatedly during cooking.

[0132] Step S5: Cooking End The information recording unit 114 records the time when cooking ends and stops recording the sensor data. The data in which each piece of information has been recorded as described above is supplied to the recipe data creation unit 102 as recorded data.

[0133] <Creating Recipe Data> A series of processes for creating recipe data will be described with reference to the flowchart of FIG.

[0134] Step S11: Process organization Process organization is the task of dividing the entire cooking process into multiple cooking processes. The process organization unit 121 of the recipe data creation unit 102 divides the entire cooking process from the start to the end of the recorded cooking into multiple sections based on, for example, an input by a chef, and sets each section as a section that constitutes one cooking process.

[0135] The process organization of recipe data defines the cooking process as a whole, i.e., what cooking task will be performed in the next cooking step when a certain target state is reached. In the process organization, the boundaries of each cooking step are set at points where timing is important, such as changing the heat or adding ingredients, and at points where it is important to judge the amount of water evaporation, temperature, etc.

[0136] The information on each cooking step set by the step organization unit 121 in this manner is recorded by the information recording unit 124 as step configuration information.

[0137] Step S12: Setting the target state for each process The target state setting unit 122 sets the target state for each cooking process using a quantitative index value that can be measured by the cooking assistance system 1. Specifically, the target state setting unit 122 selects one of the following values: time (time elapsed since the start of the cooking process), temperature (temperature of the bottom of the pot detected by the temperature sensor 32), or weight (weight of the ingredients in the pot measured by the force sensor 33), and sets it as the index value that defines the target state.

[0138] Information on the index values ​​defining the target state set by the target state setting unit 122 is recorded by the information recording unit 124 as information on the target state.

[0139] Step S13: Setting cooking tasks for each step The cooking task setting unit 123 sets the details of the cooking tasks to be performed in each cooking step, for example, based on input from the chef. The information indicating the details of the cooking tasks set by the cooking task setting unit 123 is recorded by the information recording unit 124 as information related to the cooking tasks. Information on the order in which ingredients are added, the method for adding ingredients, the method for stirring ingredients, putting on and taking off a pot lid, and other task details is recorded. The cooking task setting unit 123 generates reference task information by analyzing images of the chef's movements and data such as acceleration and angular velocity measured by sensors mounted on cooking utensils such as spatulas. The reference task information set by the cooking task setting unit 123 is recorded by the information recording unit 124 as information related to the cooking tasks.

[0140] Recipe data is created by recording information about the target state and the details of the cooking tasks for each cooking step set in the step organization (step S11). Step organization may be performed again after the target state is set, or after the cooking tasks are set.

[0141] <Reproduction Cooking> A series of processes for reproduction cooking will be described with reference to the flowchart in Fig. 10. In reproduction cooking, each cooking step is reproduced based on recipe data. The process flow for reproduction cooking is the same as the process flow for record cooking.

[0142] Step S21: Reading recipe data When the user specifies a dish to be reproduced, the recipe data acquisition unit 131 ( FIG. 7 ) of the reproduction processing unit 103 reads the recipe data. The dish to be reproduced is specified, for example, using a screen displayed on the display 21 by the information presentation unit 136.

[0143] The information presenting unit 136 displays mainly the contents of the cooking conditions on the display 21 from the information of the recipe data read by the recipe data acquiring unit 131. The user can check the contents displayed on the display 21 and confirm whether the cooking conditions important for executing the replicated cooking match the environmental conditions of the replicated cooking.

[0144] The user checks the cooking conditions and procures ingredients as specified in the recipe data. The user also prepares the ingredients as specified in the recipe data, so that the ingredients are in the same state as the initial state at the time of the recorded cooking. Information about the ingredients to be procured and how to prepare the ingredients is presented to the user, for example, on the screen of the display 21.

[0145] Before starting cooking, the user undergoes pre-training as specified in the recipe data. The pre-training is conducted to help the user acquire basic knowledge and techniques that they should master before attempting to recreate the dish. The content of the pre-training is presented to the user based on, for example, information about the chef included in the recipe data.

[0146] Step S22: Cooking environment measurement The cooking environment measurement unit 132 measures the cooking environment and acquires the specifications and characteristics of the cooking appliances used in the replicated cooking. The cooking environment measurement for the replicated cooking is performed in the same manner as the cooking environment measurement for the recorded cooking (step S2 in FIG. 8).

[0147] If there is a difference between the cooking environment of the reproduced cooking acquired by the cooking environment measurement unit 132 and the cooking environment of the recorded cooking represented by the information included in the recipe data, the information presentation unit 136 presents information indicating this difference to the user to warn them. The user can change cooking appliances in response to the difference in the cooking environment presented on the screen of the display 21, etc., to make the cooking environment of the reproduced cooking closer to the cooking environment of the recorded cooking.

[0148] Step S23: Start of cooking As with the start of cooking in recorded cooking (step S3 in FIG. 8), the sensor data acquisition unit 133 measures the temperatures of the IH stove 11 and the pot as the initial state of the cooking appliances, etc. If the measured temperatures are significantly different from the initial state temperatures recorded in the recipe data, the information presentation unit 136 suspends the start of heating and warns the user.

[0149] Step S24: During cooking The control unit 135 references the information about the cooking steps included in the recipe data and assists in reproducing the cooking steps. That is, the control unit 135 measures the state of the reproduced cooking in real time using a sensor and controls the cooking appliances to reproduce the same cooking steps as those described in the recipe data (time series of the TTW profile). The information presenting unit 136 presents information to assist the user's work as appropriate. As described above, the recipe data information includes information about the transient state of the cooking steps. The TTW profile is visualized based on the information about the transient state and presented to the user.

[0150] FIG. 11 is a diagram showing an example of a navigation screen displayed on the display 21. As shown in FIG.

[0151] A small rectangular area 151 in the upper left of the navigation screen in Fig. 11 displays a number indicating that the cooking step being performed is the 12th cooking step out of 22 cooking steps. A horizontally long area 152 to the right of area 151 displays text describing the cooking tasks. From the text displayed in area 152, the user can confirm that the task of mixing ingredients is the 12th cooking step.

[0152] Areas 153-1 to 153-3 are formed below the area 152. The areas 153-1 to 153-3 are display areas for weight, time, and temperature, respectively, which are set as index values ​​that define the target state.

[0153] In the example of FIG. 11, the difference in weight between the current weight and the weight set as the index value defining the target state is 5.9 g, and this is displayed in area 153-1. As water evaporates during heating, the difference in weight decreases over time. The user mixes the ingredients until the displayed difference in weight reaches 0 g. When the difference in weight reaches 0 g, it is determined that the target state has been reached, and the 13th cooking step begins.

[0154] When the cooking step currently being performed involves adding ingredients, information about the ingredients is displayed in area 154 at the bottom of the screen. Area 155 to the right of area 154 displays information indicating that the ingredients to be added in the thirteenth cooking step are crushed tomatoes and room temperature water.

[0155] By checking this navigation screen, the user can easily proceed through each cooking step of the recipe. The navigation for the mixing step will be described later.

[0156] In this way, measurable values ​​in the cooking assistance system 1 are uniquely set for all cooking processes as index values ​​that define the target state and are recorded as recipe data information, making it possible to stably achieve a high degree of reproducibility regardless of who is performing the reproducible cooking.

[0157] Step S25: End of Cooking When all cooking steps are completed, the replicated cooking is completed and the replicated sample is ready.

[0158] <Reproducibility Evaluation> The chef evaluates the reproducibility of the completed dish (reproduced sample) using sensory evaluation techniques and measuring equipment.

[0159] Through the above series of processes, users can consistently create dishes with a high degree of repeatability. In addition, the navigation screen displays information about each cooking step, making it easy for users to proceed with the repeat cooking.

[0160] <<Example of Navigation Based on Reference Stirring Operation Information>> <Stirring Operation Navigation Process> The stirring operation navigation process will be described with reference to the flowchart of Fig. 12. The process shown in Fig. 12 is performed as a process during cooking in step S24 of Fig. 10 when a cooking process in the replicated recipe involves stirring.

[0161] In step S101, the information presenter 136 outputs an instruction to input ingredients. The instruction to input ingredients is given using, for example, a navigation screen similar to the screen in FIG.

[0162] In step S102, the information presentation unit 136 uses video and audio to provide mixing guidance. Data for outputting the mixing guidance is included in the standard mixing task information. The user starts mixing the ingredients according to the display on the navigation screen. While the user is mixing the ingredients, sensor data, which is the measurement results of the user's actions mixing the ingredients, the state of the ingredients, etc., is acquired by the sensor data acquisition unit 133 and supplied to the control unit 135.

[0163] In step S103, the control unit 135 acquires user motion data, which serves as an evaluation index, based on the sensor data acquired by the sensor data acquisition unit 133. Here, as described above, motion data indicating the characteristic quantities of the speed, stroke, and direction of the mixing motion is acquired. Motion data indicating the characteristic quantities of other evaluation indexes, such as the amount of force and rhythm, may also be acquired.

[0164] The motion data indicating these feature quantities is acquired, for example, by analyzing images captured by the camera 23 or by analyzing sensor data measured by an acceleration sensor or angular velocity sensor mounted on the spatula. The motion data may be acquired using an inference model generated in advance by machine learning.

[0165] In step S104, the control unit 135 compares the user's motion data with the motion data included in the reference stirring task information, and calculates a score as evaluation information indicating the state of the user's motion. Here, the smaller the difference between the user's motion data and the motion data included in the reference stirring task information, the higher the score calculated. The comparison of the motion data is performed at a predetermined timing, such as one minute after the start of the stirring task.

[0166] In step S105, the information presenter 136 provides feedback according to the score, for example by displaying a score indicating the state of the user's action on the navigation screen.

[0167] In step S106, the information presenting unit 136 determines whether mixing has ended, and if it determines that mixing has not ended, repeats the processing from step S102 onwards. Based on the data included in the reference mixing work information, navigation of the mixing method for the next time is performed.

[0168] If it is determined in step S106 that the mixing has ended, the information presenting unit 136 displays the result of success or failure of the mixing work in step S107, and ends the process.

[0169] <Example of Navigation Screen> FIG. 13 is a diagram showing an example of a navigation screen for a mixing operation.

[0170] An action status display area 161, which is a display area for information indicating the user's action status, is formed above the navigation screen. A direction display area 162 and a video display area 163 are formed below the action status display area 161. The configuration of the navigation screen changes appropriately depending on the content of the task to be navigated.

[0171] The direction display area 162 is an area for displaying information indicating the mixing direction. As shown in Fig. 13 , the order of the mixing direction is indicated by arrow images 162A, which are multiple small arrow-shaped images, and the current mixing direction is indicated by one large arrow image below the arrow images 162A.

[0172] As shown in Fig. 14, the display of the arrow image 162A is switched to indicate the mixing direction at each timing. In the example of Fig. 14, the mixing direction at each timing is presented in the order of "bottom to top," "top to bottom," "bottom right to top left," etc. The user can confirm the transition of the mixing direction based on the display of the arrow image 162A and the like. Along with the display of the direction display area 162, navigation of the mixing direction is performed using audio such as "bottom to top" or "top to bottom" as appropriate.

[0173] The stirring direction may be fixed from bottom to top, and the stirring operation may be performed by rotating the pot. In this case, navigation is performed using images and audio to instruct the user to rotate the pot so that the handle is positioned at any desired position, such as "right" or "top left." If the stirring operation is performed by repeating a certain pattern of actions, the timing of stirring may be presented using a sound with a certain rhythm. By using a sound with a certain rhythm for navigation, the user can perform the stirring operation without looking at the screen.

[0174] The video display area 163 is a display area for a video that serves as a navigation for mixing. For example, a video showing the chef's actions taken during the recorded cooking is used for displaying the navigation. An image showing the mixing direction or an image showing the trajectory of the mixing action may be displayed superimposed on the video.

[0175] In this way, the user mixes ingredients according to the information displayed as navigation information in the direction display area 162 and the video display area 163. The navigation information is displayed based on the reference mixing task information. Navigation of not only the direction but also the speed and stroke of the mixing motion is performed as appropriate. Navigation of at least one of the speed, stroke, direction, and strength of the mixing motion is performed.

[0176] Even with such navigation, the accuracy of the work varies depending on the chef. Therefore, the state of the user's work is evaluated by comparing it with the action data of the standard mixing work. The evaluation result of the state of the user's work is fed back using the display in the action state display area 161.

[0177] 13, information indicating the speed, stroke, and direction as evaluation indices of the blending motion is displayed. For example, for the blending motion speed, the text "Good" and a score of 9 are displayed. At least one of the text and the score may be displayed.

[0178] Evaluation of the speed of mixing actions Speed ​​scores are calculated, for example, according to the difference in the speed of the user's actions relative to the average speed of the standard mixing task, as follows: A score for the speed of the user's actions is calculated with a maximum score of 10 points, with the greater the difference the greater the deduction of points. Difference of 1% or less: 10 points Difference of 1-3%: 9 points Difference of 3-5%: 8 points ... Difference of 50% or more: 1 point

[0179] A score may be calculated based on the time that the user's mixing speed is below the average speed of the standard mixing task, with points being deducted each time the speed is below the standard speed by a predetermined time, such as 3 seconds or more.

[0180] The threshold range for the speed of the standard mixing task may be set to, for example, 23 to 28 cm / s, and the speed score may be calculated according to the time that the speed of the user's mixing action is outside the threshold range. For example, the speed score may be calculated as follows according to the proportion of the time that is outside the threshold range to the total time of the mixing action: Less than 3%: 10 points out of 10; 3 to 5%: 9 points; 5 to 7%: 8 points; ... 50% or more: 1 point

[0181] Evaluation of Strokes of Mixing Actions The stroke score is calculated by dividing the area inside the pot, which is the mixing area, into eight areas, as shown in Figure 15, and based on the time spent in each of the eight areas (the time the spatula is moving). The circle shown in Figure 15 indicates the area inside the pot in a planar view. Each area is set by dividing the area inside the pot into eight equal parts, centered around the center point.

[0182] The numbers shown in each area in Figure 15 indicate the percentage of time (stay rate) the user is mixing. The larger the number, the longer the time spent mixing in that area. Depending on the difference between the chef's stay rate in each area indicated by the reference mixing task information and the stay rate in each area in the user's mixing action, the stroke score is calculated, for example, as follows: Difference in all areas is within 1%: 10 points maximum Areas with a difference of 1% or less are within 50% (within four areas), and the difference in other areas is within 3%: 9 points Difference in all areas is within 3%: 8 points ... Difference in all areas is 50% or more: 1 point

[0183] FIG. 16 is a diagram showing an example of a presentation of the range to be mixed.

[0184] Information indicating the dwell rate of each area may be presented using a circular image 162B as shown in FIG. 16 . For example, stroke navigation is performed by highlighting areas with low dwell rates or displaying information instructing the user to increase the number of blends. The colored areas in image 162B in FIG. 16 are areas with low dwell rates. The display of image 162B, which is navigation information, changes depending on the evaluation result of the user's action status for the stroke. This allows the user to modify their action so that the blending range is balanced.

[0185] FIG. 17 is a diagram showing an example of dividing the blending range.

[0186] As shown in A of Fig. 17, the mixing range may be divided into four areas, and the dwell rate of each area may be detected. Alternatively, as shown in B of Fig. 17, the mixing range may be divided into concentric circles, such as a central circular area and surrounding areas. In this way, the number of divisions and division pattern of the mixing range can be changed as desired. The number of divisions and division pattern of the mixing range may be changed depending on the dish to be reproduced.

[0187] - Evaluation of mixing direction The score for mixing direction is calculated based on the time that the direction shown using an arrow image, etc. matches the direction of the user's mixing action. The score for mixing direction is calculated so that the shorter the time that they match, the greater the deduction.

[0188] ・Example of feedback The scores for each evaluation index as described above are calculated and displayed in real time while the user is mixing. By being aware of the evaluation indexes with low scores and correcting their movements, the user can bring their movements closer to those in the standard mixing task.

[0189] For example, if the score for the speed of the mixing action is 8 to 10 points, the text "Feels good" is displayed as shown in Figure 13, and the state of the speed of the user's action is fed back. If the speed score is, for example, 7 points or less, it is determined whether the time spent faster or slower than the speed range set as the threshold is long, and the following feedback is given depending on the determination result: - If mixing is slow and the score is 7 to 6 points: "Please mix a little faster" - If mixing is slow and the score is 5 points or less: "Please mix more quickly" - If mixing is fast and the score is 7 to 6 points: "Please mix a little more slowly" - If mixing is fast and the score is 5 points or less: "Please mix more slowly"

[0190] Feedback is similarly provided for evaluation indices other than speed. The status of the movement for each evaluation indices is presented in real time, allowing the user to confirm specifically how to correct their movement. By correcting their movement in accordance with the feedback, the user can aim for a passing score for the entire mixing operation (pass as determined in step S107 of FIG. 12) even if their score during the mixing operation is low.

[0191] The speed, stroke, and direction may be fed back individually at different timings, or may be fed back simultaneously as shown in FIG.

[0192] A score for each evaluation index may be presented, or a total score may be presented. When the total score of each evaluation index is calculated as the score of the user's action, a weight may be assigned to the score of each evaluation index depending on the ingredients, the type of dish to be reproduced, etc. The importance of each evaluation index (speed, stroke, and direction) may differ in some cases, such as in some dishes where the mixing position is important and in other dishes where the mixing speed is important.

[0193] - Presentation of evaluation of overall mixing work Based on the scores for each evaluation index as described above, a score for the overall mixing work is calculated. If the maximum score is 10 points, the pass criteria are set such that 10 to 8 points is pass and 7 points or less is fail, and the pass / fail status of the overall mixing work is presented to the user according to the score. The evaluation information presented to the user includes not only the score for each evaluation index, but also the score for the overall mixing work, as appropriate. The evaluation information is generated based on the reference work information and data obtained by sensing the user's actions, and includes various pieces of information presented to the user.

[0194] If the mixing operation is successful, the user will proceed to the next operation. If the mixing operation is unsuccessful, the user may be instructed to, for example, restart the replicated cooking from the beginning or to perform an operation to recover from the mixing operation. The user may also be informed that the user needs to practice the mixing operation or be presented with an improvement plan on what operation to perform. In this way, different operations are performed as the next operation depending on whether the mixing operation is successful or unsuccessful.

[0195] <Modification of Cooking Task Navigation> In some cases, such as when a user has little proficiency with the cooking assistance system 1 or when the user has not seen many cooking tasks for the first time, it may be difficult to obtain a passing score for the cooking tasks using only the feedback provided during the replica cooking. It may be possible to practice using the cooking assistance system 1 before starting the replica cooking, and then proceed to the actual replica cooking after receiving a passing grade. A practice mode is provided as a separate mode from the production mode in which the actual replica cooking is performed.

[0196] The practice mode's cooking simulation is performed by, for example, presenting a video that guides the user through a difficult task, having the user perform the cooking task without using ingredients, and then calculating a score. Because the evaluation is based on the user's actions, it is possible to avoid failing the test and wasting ingredients.

[0197] Among users who perform cooking reproductions, some have inexperienced cooking skills and some have advanced cooking skills. For example, if standard operation information is prepared in a form tailored to those with inexperienced cooking skills, it may result in those with advanced cooking skills being forced to perform unnecessary work. For those with advanced cooking skills, navigation may be provided according to their level of proficiency. In this case, navigation data according to the level of proficiency in cooking skills is prepared and recorded as standard operation information. The level of proficiency of each user may be determined according to their score in the cooking reproduction practice mode, and navigation may be switched accordingly.

[0198] FIG. 18 is a diagram showing another example of the navigation presentation for the mixing operation.

[0199] 18, a projector 28 is installed near the ceiling of the environment where the replicated cooking will be performed, and the mixing action is navigated using information projected onto a pot by the projector 28. In this way, navigation can also be performed using projection mapping technology.

[0200] FIG. 19 is a diagram showing an example of navigation presented by projection mapping.

[0201] Fig. 19 shows how ingredients in a pot are mixed with a spatula. For example, the position and shape of the pot are recognized based on an image captured by camera 23, and guide image #1, which is an image of a curve, and guide image #2, which is an image of a circle, are projected as shown in Fig. 19A and B. For example, guide image #2 indicates the target position of the spatula, and guide image #1 indicates the trajectory to the target position.

[0202] 19A and 19B, the projection positions of guide image #1 and guide image #2 change over time. The user can easily reproduce the movements in the standard mixing task by moving the spatula to the position indicated by guide image #2, following the trajectory indicated by guide image #1.

[0203] In this way, navigation may be presented using a device other than the display 21. For example, navigation may be presented using AR glasses.

[0204] <<Modifications>> <System Configuration Example> FIG. 20 is a diagram showing a configuration example of a cooking assistance system.

[0205] Although the processes of recording cooking, creating recipe data, and reproducing cooking are each performed in the same cooking assistance system 1 having a recording processing unit 101, a recipe data creation unit 102, and a reproduction processing unit 103 as shown in A of Figure 20, each process may also be performed in a different information processing device.

[0206] 20B, the recording cooking and the reproducing cooking may be performed in the cooking assistance system 1, and the recipe data may be created in the Web server 42. In this case, the recording processing unit 101 and the reproducing processing unit 103 are realized in the cooking assistance system 1, and the recipe data creation unit 102 is realized in the Web server 42.

[0207] The recording data generated by the recording processing unit 101 of the cooking assistance system 1 is sent to the Web server 42 via a network such as the Internet, and is used to create recipe data by the recipe data creation unit 102. The recipe data created by the recipe data creation unit 102 of the Web server 42 is sent to the cooking assistance system 1 via the network, and is used by the reproduction processing unit 103 to reproduce the cooking.

[0208] 21, the recording cooking and the reproducing cooking may be performed in different cooking assistance systems 1. In the example shown in Fig. 21, the recording processing unit 101 is realized in the cooking assistance system 1-1, and the reproducing processing unit 103 is realized in the cooking assistance system 1-2. The cooking assistance systems 1-1 and 1-2 each have the same configuration as the cooking assistance system 1 described with reference to Fig. 3 etc.

[0209] The recording data generated by the recording processing unit 101 of the cooking assistance system 1-1 is transmitted to the Web server 42 via a network such as the Internet, and is used to create recipe data in the recipe data creation unit 102. The recipe data created by the recipe data creation unit 102 of the Web server 42 is transmitted to the cooking assistance system 1-2 via the network, and is used in the reproduction processing unit 103 to reproduce the cooking.

[0210] By providing a plurality of cooking assistance systems 1-2 that perform cooking reproduction, it becomes possible to reproduce a dish based on one recipe data in various locations.

[0211] In this way, it is possible to arbitrarily change which device is to perform at least one of the processes of cooking record, recipe data creation, and cooking reproduction.

[0212] - Computer Configuration Example The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed from a program recording medium into a computer built into dedicated hardware or a general-purpose personal computer.

[0213] Fig. 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. A computer having the configuration shown in Fig. 22 is provided in the induction cooker 11. External devices such as a web server 42 are also configured using a computer having the configuration shown in Fig. 22.

[0214] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0215] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0216] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0217] The program executed by the CPU 1001 is provided, for example, by being recorded on a removable medium 1011 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 1008.

[0218] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0219] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0220] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0221] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0222] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0223] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0224] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0225] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0226] (1) An information processing device comprising: a first acquisition unit that acquires recipe data including at least reference task information indicating actions in cooking work; a second acquisition unit that acquires sensor data measuring a cook's actions during cooking; a control unit that generates evaluation information of the cook's actions based on the reference task information and the sensor data; and a presentation unit that presents the evaluation information. (2) The information processing device described in (1), wherein the control unit generates multiple types of evaluation information. (3) The information processing device described in (1) or (2), wherein the control unit generates the evaluation information based on a comparison result between the actions indicated by the reference task information and the actions indicated by the sensor data. (4) The information processing device described in any of (1) to (3), wherein the presentation unit presents the evaluation information of the actions at each time to the cook performing the reproduction cooking. (5) The information processing device described in any of (1) to (4), wherein the presentation unit presents at least one of text and a score as the evaluation information. (6) The information processing device according to any one of (1) to (5), wherein the presentation unit presents navigation information for the action based on the reference task information. (7) The information processing device according to (6), wherein the presentation unit changes the navigation information according to the evaluation information. (8) The information processing device according to (6), wherein the cooking action is a mixing action, and the presentation unit presents the navigation information indicating at least one of the speed, stroke, direction, and strength of the mixing action. (9) The information processing device according to (8), wherein the presentation unit presents the evaluation information regarding at least one of the speed, stroke, direction, and strength of the mixing action. (10) The information processing device according to (9), wherein the presentation unit presents, as the navigation information for the stroke of the mixing action, information according to the time for each area of ​​the cooking utensil in which the cook mixed ingredients. (11) The information processing device according to any one of (1) to (10), wherein the reference task information is information generated based on sensor data measuring the actions of other cooks during cooking.(12) An information processing method in which an information processing device acquires recipe data including at least reference task information indicating movements in cooking work, acquires sensor data measuring the movements of a cook while cooking, generates evaluation information of the cook's movements based on the reference task information and the sensor data, and presents the evaluation information. (13) A program that causes a computer to execute processes in which: acquires recipe data including at least reference task information indicating movements in cooking work, acquires sensor data measuring the movements of a cook while cooking, generates evaluation information of the cook's movements based on the reference task information and the sensor data, and presents the evaluation information. (14) An information processing device comprising an information processing unit that generates reference task information indicating movements in cooking work based on sensor data measuring the movements of a cook while cooking, and generates recipe data including at least the reference task information. (15) The information processing device described in (14), in which the information processing unit generates the reference task information indicating multiple types of indicators used in generating evaluation information. (16) The information processing device according to (14) or (15), wherein the cooking task is stirring, and the information processing unit generates the reference task information indicating at least one of the speed, stroke, direction, and strength of the stirring motion. (17) An information processing method in which an information processing device generates reference task information indicating movements in cooking tasks based on sensor data measuring the movements of a cook while cooking, and generates recipe data including at least the reference task information. (18) A program causing a computer to execute processes in which the computer generates reference task information indicating movements in cooking tasks based on sensor data measuring the movements of a cook while cooking, and generates recipe data including at least the reference task information. (19) A data structure of recipe data in which reference task information indicating movements in cooking tasks, generated based on sensor data measuring the movements of a cook while cooking, is arranged as information related to cooking steps.(20) The reference work information is used by an information processing device that acquires sensor data measuring the actions of a second cook who reproduces the cooking of a first cook as the cook, to present information indicating the status of the actions of the second cook to the second cook. Data structure of recipe data described in (19).

[0227] 1 Cooking assistance system, 11 IH stove, 21 Display, 22 Speaker, 23 Camera, 24 Microphone, 25 Lighting equipment, 26 Cooking environment measuring device, 27 Router, 31 Processor, 32 Temperature sensor, 33 Force sensor, 34 Storage, 41 Smartphone, 42 Web server, 101 Recording processing unit, 102 Recipe data creation unit, 103 Reproduction processing unit, 111 Cooking environment measurement unit, 112 Sensor data acquisition unit, 113 User input detection unit, 114 Information recording unit, 121 Process organization unit, 122 Target state setting unit, 123 Cooking work setting unit, 124 Information recording unit, 125 Recipe data management unit, 131 Recipe data acquisition unit, 132 Cooking environment measurement unit, 133 Sensor data acquisition unit 134 User input detection unit, 135 Control unit, 136 Information presentation unit

Claims

1. An information processing device comprising: a first acquisition unit that acquires recipe data including at least reference task information indicating actions in cooking work; a second acquisition unit that acquires sensor data measuring the actions of a chef while cooking; a control unit that generates evaluation information of the chef's actions based on the reference task information and the sensor data; and a presentation unit that presents the evaluation information.

2. The information processing device according to claim 1, wherein the control unit generates a plurality of types of the evaluation information.

3. The information processing device according to claim 1, wherein the control unit generates the evaluation information based on a comparison result between the movement indicated by the reference task information and the movement indicated by the sensor data.

4. The information processing device according to claim 1, wherein the presentation unit presents the evaluation information of the actions at each time to the chef who is performing the re-creation cooking.

5. The information processing device according to claim 1, wherein the presentation unit presents at least one of text and a score as the evaluation information.

6. The information processing device according to claim 1, wherein the presentation unit presents navigation information for an action based on the reference task information.

7. The information processing device according to claim 6, wherein the presentation unit changes the navigation information in accordance with the evaluation information.

8. The information processing device according to claim 6, wherein the cooking task is a mixing task, and the presentation unit presents the navigation information indicating at least one of the speed, stroke, direction, and strength of the mixing motion.

9. The information processing device according to claim 8, wherein the presentation unit presents the evaluation information regarding at least one of the speed, stroke, direction, and strength of the mixing motion.

10. The information processing device according to claim 9, wherein the presentation unit presents information according to the time spent by the cook on each area of ​​the cooking utensil in which the cook mixed the ingredients as the navigation information for the stroke of the mixing action.

11. The information processing device according to claim 1, wherein the reference work information is information generated based on sensor data that measures the actions of other chefs while they are cooking.

12. An information processing method in which an information processing device acquires recipe data including at least reference task information indicating actions in cooking work, acquires sensor data measuring the actions of a chef while cooking, generates evaluation information of the chef's actions based on the reference task information and the sensor data, and presents the evaluation information.

13. A program that causes a computer to execute the following processes: acquire recipe data including at least reference task information indicating actions in cooking; acquire sensor data measuring the actions of a chef while cooking; generate evaluation information of the chef's actions based on the reference task information and the sensor data; and present the evaluation information.

14. An information processing device comprising an information processing unit that generates standard task information indicating actions during cooking based on sensor data measured by a chef during cooking, and generates recipe data including at least the standard task information.

15. The information processing device according to claim 14, wherein the information processing unit generates the reference work information indicating a plurality of types of indices used in generating evaluation information.

16. The information processing device according to claim 14, wherein the cooking task is a mixing task, and the information processing unit generates the reference task information indicating at least one of the speed, stroke, direction, and strength of the mixing motion.

17. An information processing method in which an information processing device generates reference task information indicating actions taken during cooking based on sensor data measuring the actions of a chef while cooking, and generates recipe data that includes at least the reference task information.

18. A program that causes a computer to execute the following process: generate standard operation information indicating characteristic quantities of actions in cooking work based on sensor data that measures the actions of a chef while cooking; and generate recipe data that includes at least the standard operation information.

19. A data structure of recipe data in which standard work information indicating the actions taken during cooking work, generated based on sensor data measuring the actions of a chef while cooking, is arranged as information about the cooking process.

20. The data structure of recipe data described in claim 19, wherein the reference work information is used by an information processing device that acquires sensor data measuring the actions of a second cook who reproduces the cooking of the first cook as the cook, to present information indicating the status of the actions of the second cook to the second cook.

Images