Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to easily confirm the content of the next cooking step and the time until the next cooking step. An information processing device according to one aspect of the present technology controls a display of a cooking navigation screen in which information about the current cooking step is fixedly displayed, and information about the next cooking step is moved as time passes and displayed. For example, the information about the next cooking step is moved so as to approach the information about the current cooking step and displayed. The present technology can be applied to a system that assists cooking in a kitchen.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable easy confirmation of the content of the next cooking step and the time until the next cooking step.

[0002] Recipes that ordinary chefs refer to usually contain many ambiguous descriptions and expressions that assume tacit knowledge. As a result, the cooking process varies depending on the chef's skill and interpretation, as well as differences in cooking equipment and cooking environment, resulting in low reproducibility of the finished dish.

[0003] Patent Literature 1 describes a cooking assistance system that defines the target state of ingredients in each cooking step by parameters based on sensor data that can be measured during cooking, and enables the reproduction of a dish using recipe data that includes such parameters. During the reproduction cooking, the chef is guided through the work content of each step using images, audio, etc. By providing the chef with feedback on information obtained from sensor data measured during the reproduction cooking and controlling the heating of ingredients until they reach the state defined as the target state, it becomes possible to reproduce the same dish as the chef made.

[0004] International Publication No. 2023 / 182197

[0005] In the cooking simulation of the cooking assistance system described in Patent Document 1, the more accurately the chef performs the tasks in response to instructions via navigation using images, audio, etc., i.e., the more faithfully the chef's tasks are replicated, the higher the degree of replication of the finished dish. Therefore, navigation using a user interface that is easy for the chef to understand is important.

[0006] The present technology has been made in view of such circumstances, and makes it possible to easily check the content of the next cooking step and the time remaining until the next cooking step.

[0007] An information processing device according to one aspect of the present technology includes a presentation control unit that controls the display of a cooking navigation screen on which information about the current cooking step is displayed fixedly and information about the next cooking step is displayed moving over time.

[0008] In one aspect of the present technology, the display of a cooking navigation screen is controlled so that information about the current cooking step is displayed in a fixed manner and information about the next cooking step is displayed in a moving manner over time.

[0009] 1 is a diagram showing the overall flow of the recording / reproducing process of a cooking process. A diagram showing an example of cooking based on a navigation dataset. A diagram showing an example of the configuration of a cooking assistance system. A diagram showing an example of information included in the navigation dataset. A diagram showing example of reference stirring task information. A block diagram showing an example of the functional configuration of a cooking assistance system. A block diagram showing an example of the functional configuration of a cooking assistance system. A flowchart explaining a series of processes for recording cooking. A flowchart explaining a series of processes for creating a navigation dataset. A flowchart explaining a series of processes for reproducing cooking. A diagram showing an example of a menu selection screen. A diagram showing an example of a cooking process confirmation screen. A diagram showing an example of a cooking navigation screen. A diagram showing an enlarged display of the time-series data area. A diagram showing an example of a cooking navigation screen after a predetermined time has elapsed. A diagram showing an example of a cooking navigation screen after a predetermined time has elapsed. A diagram showing transitions of the cooking navigation screen. A diagram showing an example of the display of a switching button. A diagram showing an example of the display of supplementary information. A diagram showing an example of the display of supplementary information. A diagram showing an example of the display of supplementary information. A diagram showing an example of the display of a countdown. A diagram showing an example of the display of a countdown. A diagram showing an example of a score display screen. A flowchart explaining cooking navigation processing. A diagram showing another example of the display of a cooking navigation screen. A diagram showing another example of the display of a cooking navigation screen. Fig. 1 is a diagram showing another example of the display of the cooking navigation screen. Fig. 2 is a diagram showing another example of the display of the cooking navigation screen. Fig. 3 is a diagram showing an example of the configuration of a cooking assistance system. Fig. 4 is a diagram showing an example of the configuration of a cooking assistance system. Fig. 5 is a block diagram showing an example of the configuration of a computer.

[0010] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the present technology 2. Overview of cooking assistance system 3. Overall flow of recording / reproducing process 4. System configuration 5. Details of recording / reproducing process 6. Specific examples of screen display 7. Modified examples

[0011] <<Outline of the Present Technology>> In a cooking reproduction using a cooking assistance system incorporating the present technology, the chef is guided through a screen containing information about the cooking process. The navigation is presented as follows.

[0012] Information about the current cooking step is displayed fixedly, and information about the next cooking step is displayed gradually over time. The cook can make the necessary preparations for the next cooking step while cooking according to the instructions presented as information about the current cooking step. The cook can also intuitively check the timing until the start of the next cooking step.

[0013] The cooking process is switched when the sensor data obtained as time-series data during the replicated cooking meets pre-set conditions. Since the chef does not need to be aware of when to switch cooking processes, he or she can concentrate on the cooking task at hand.

[0014] When the sensor data obtained as time-series data during the replicated cooking meets pre-set conditions, supplementary information is presented, allowing the chef to carry out the supplementary cooking tasks at the appropriate time.

[0015] The sensor data obtained during the recorded cooking and the sensor data obtained during the reproduced cooking are displayed as time-series data based on the same standard in a form that allows for comparison using graphs, etc. Based on the difference between the sensor data during the recorded cooking and the sensor data during the reproduced cooking, it becomes possible to make the chef performing the reproduced cooking aware of any adjustments to the cooking work.

[0016] <<Outline of the Cooking Assistance System>> <Expected Usage Scenarios> This technology is related to a cooking assistance system that assists cooking in environments such as kitchens. This technology can be applied to cooking assistance systems that use heating cookers such as induction stoves.

[0017] A cooking assistance system using this technology is equipped with various sensor devices that measure the cooking state. Based on the measurement results from the sensor devices, time-series sensor data indicating the state of ingredients during the cooking process is recorded.

[0018] In addition, the system provides cooking assistance to a chef who is trying to recreate a dish based on a navigation dataset created using recorded data including sensor data. The cooking assistance is provided to enable accurate reproduction of the finished dish.

[0019] The cooking process to replicate the final dish is carried out with some operations, such as adjusting the heat of the cooking appliance, automated. The chef, who is the user of the cooking support system, performs tasks such as adding ingredients and stirring by following the navigation provided by the UI.

[0020] In addition, "cooking" refers to the finished product that is produced through cooking. Cooking refers to the process of cooking food and the act (task) of cooking food.

[0021] <Cooking Process> FIG. 1 is a diagram showing the overall flow of the recording / reproducing process of a cooking process.

[0022] As will be described in detail later, a "record cooking" is performed as shown in FIG. 1, and sensor data indicating the measurement results in the record cooking is recorded.

[0023] As indicated by the arrow A1, a "navigation data set creation" is performed based on the recorded data including the recorded sensor data, and a navigation data set is created. The navigation data set is data that can be processed by a computer.

[0024] The navigation dataset describes the details of each cooking step until the dish is completed as follows: Normally, one dish is completed through multiple cooking steps as shown in Figure 2.

[0025] 1. The entire cooking process consists of multiple cooking steps that are arranged in a chronological order. 2. For each cooking step, a cooking task and a target state are defined, as shown in the speech bubbles in Figure 2. 3. Cooking tasks indicate operations performed on cooking equipment and ingredients. 4. The target state indicates the conditions for completing a cooking step and moving on to the next cooking step. I. The conditions for moving on to the next cooking step are defined using numerical indicators based on data that the cooking assistance system can measure. II. The timing at which the conditions for moving on to the next cooking step are reached is expressed so that it can be uniquely identified on time-series data that the cooking assistance system can detect. 5. The target state is defined using one of the TTW (Time / Temperature / Weight) values ​​that the cooking assistance system can measure. The TTW value is the value of the target parameter, which defines the target state. The target parameter includes Temperature and Weight, which are sensor data that the cooking assistance system can measure, as well as Time, which the cooking assistance system can measure. I. Time: Elapsed time from the start of the cooking step [sec] II. Temperature: Temperature of the object to be heated or the heating medium [°C] III. Weight: Weight of the object to be heated [g] 6. The initial state is defined for each ingredient used in the cooking process: I. Temperature of the ingredient when it is put into the cooking utensil (e.g., pot) [℃] II. Weight of the ingredient to be put into the cooking utensil (e.g., pot) [g]

[0026] In this way, in a cooking assistance system to which the present technology is applied, a navigation dataset is created for all cooking steps that indicates the goal state and initial state defined using indices that can be interpreted without being subject to personal judgment. As will be described later, the navigation dataset includes various information other than the information indicating the goal state and initial state.

[0027] Returning to the explanation of FIG. 1, as indicated by the tip of arrow A2, "reproduction cooking" is performed using the navigation data set, and the dish is reproduced.

[0028] The replicated cooking is performed according to the following rules: 1. Cooking is performed at the start of each cooking process. 2. The next cooking process begins when the target state of each cooking process is reached. 3. The initial state is matched when each ingredient is added.

[0029] By carrying out the above-described process, the same cooking as the recorded cooking can be reproduced. This reduces variations in the cooking process due to differences in the chef's skills and interpretation, as well as differences in cooking utensils and cooking environment, making it possible to consistently produce highly reproducible dishes.

[0030] Hereinafter, the cook who performs the recording cooking will be referred to as the chef, and the cook who performs the re-creation cooking will be referred to as the user, as appropriate. A cook who wants to recreate a dish made by a top chef will become a user of the cooking assistance system and perform the re-creation cooking. When there is no need to distinguish between the cook who performs the recording cooking and the cook who performs the re-creation cooking, they will simply be referred to as the cook.

[0031] <<Overall Flow of Recording / Reproducing Process>> Each process in Fig. 1 will be described. The recording / reproducing process of the cooking process consists of the processes of "recording cooking," "creating a data set for navigation," and "reproducing cooking."

[0032] 1. Record Cooking Record cooking is a process in which a chef cooks food and a recording system records data on the cooking process. The recording system is implemented by a cooking support system. Time series data of TTW values ​​measured by sensors is recorded as recorded data.

[0033] 2. Navigation Data Set Creation The navigation data set creation is a process in which the cooking assistance system creates a navigation data set to be used for the re-creation of a recipe based on recorded data. The navigation data set is also created based on data entered by a person as appropriate. The navigation data set is recorded as digital data and used as input for the re-creation system during the re-creation. The navigation data set is also used to present human-understandable information as needed.

[0034] 3. Reproduced Cooking Reproduced cooking is a process in which the reproduction system controls cooking appliances according to the navigation dataset and presents information about cooking tasks to the user based on the navigation dataset. The user performs cooking tasks in the reproduction system according to the presented information. The reproduction system is also realized by a cooking assistance system.

[0035] <<System Configuration>> <Configuration Example of Cooking Assistance System> FIG. 3 is a diagram showing a configuration example of a cooking assistance system.

[0036] The cooking assistance system 1 shown in Fig. 3 as an information processing device is a type of so-called smart cooking appliance. The cooking assistance system 1 assists in cooking using cooking utensils such as pots and frying pans. Cooking utensils are appliances used by cooks, such as pots, frying pans, and spatulas. Cooking appliances also include electrically driven appliances such as induction stoves and microwave ovens.

[0037] The cooking assistance system 1 is configured by connecting an IH stove 11 with a display 21, a speaker 22, a camera 23, a microphone 24, lighting equipment 25, a cooking environment measuring device 26, and a router 27. Each device is connected to the IH stove 11 by wired communication or wireless communication such as wireless LAN.

[0038] The IH stove 11 is an electromagnetic cooker that heats cooking utensils 12 such as pots and frying pans. In Fig. 3, a pot is shown as the cooking utensil 12. A spatula used to stir the contents of the pot is also shown as cooking utensil 12A.

[0039] As shown in the balloon in FIG. 3, the IH stove 11 is equipped with a processor 31, a temperature sensor 32, a force sensor 33, a storage 34, and a human sensor 35.

[0040] The processor 31 is a computing unit that controls the heating of the cookware 12. The processor 31 also controls each device that constitutes the cooking assistance system 1 and communicates with external devices, a web server 42, etc. Communication with the web server 42 is performed via the router 27. The processor 31 executes predetermined programs and controls the overall operation of the cooking assistance system 1.

[0041] Furthermore, the processor 31 detects instructions and actions of the chef input via an input UI such as the microphone 24. The processor 31 also controls the shooting conditions of the camera 23 by adjusting the brightness of the lighting device 25, for example.

[0042] Temperature sensor 32 measures the temperature of the pot or the temperature of the food being cooked in the pot. In this case, the pot serves as the heating medium, and the food being cooked in the pot serves as the heated object. For example, a radiation thermometer (infrared sensor) that is highly responsive and capable of non-contact measurement is used as temperature sensor 32. Temperature sensor 32 may also be configured to measure both the temperature of the pot and the temperature of the food being cooked.

[0043] The force sensor 33 measures the force and weight applied to a cooking utensil such as a pot. For example, a piezoelectric sensor is used as the force sensor 33. The force sensor 33 is provided, for example, below the top plate of the IH stove 11.

[0044] The storage 34 stores various data such as programs executed by the processor 31 and data sets for navigation.

[0045] The human presence sensor 35 is a sensor that uses infrared rays, etc. The human presence sensor 35 detects a cook near the IH stove 11.

[0046] The display 21 displays various screens under the control of the processor 31. For example, the display 21 is provided near the induction cooker 11. Various information is presented to the cook using the display on the display 21. The display 21 is configured by a display device such as an LCD monitor or a projector. A wearable display such as AR glasses may also be used as the display 21.

[0047] The speaker 22 outputs various sounds under the control of the processor 31. For example, sounds that provide guidance for cooking operations are presented to the user using the speaker 22.

[0048] Camera 23 captures an image of the food being cooked in the pot. Camera 23 is installed, for example, above IH stove 11 with its angle of view facing the pot. The image captured by camera 23 is transmitted to processor 31 in real time. For example, a normal visible light camera is used as camera 23. In addition to a visible light camera, a thermography camera, a hyperspectral camera, or a depth-sensing camera may also be installed as camera 23. Camera 23 also captures an image of the cook performing the cooking work. Camera 23 is made up of multiple cameras, including a camera that captures the state inside the pot and a camera that captures the cook.

[0049] In addition to the microphone 24, a smartphone 41 is used as a device for inputting various pieces of information to the cooking assistance system 1 described above. Other mobile devices such as a tablet terminal may also be used for input. Information input by the chef using the smartphone 41 is acquired, for example, by the processor 31. The processor 31 controls each device as appropriate in response to the input by the chef. A touch panel provided on the display surface of the display 21 is also used as an input device.

[0050] The cooking environment measuring instrument 26 is a measuring instrument equipped with sensor devices such as a temperature sensor, a humidity sensor, an acoustic sensor, and an air pressure sensor. The cooking environment measuring instrument 26 measures the temperature, humidity, environmental sound, and air pressure of the cooking environment. Sensor data indicating the measurement results of the cooking environment measuring instrument 26 is also supplied to the processor 31 and used for various processes.

[0051] In this way, various sensor devices are provided in the cooking assistance system 1. In the cooking assistance system 1, the camera 23 captures an image of, for example, a chef mixing ingredients using a spatula (cooking utensil 12A).

[0052] In the cooking assistance system 1, the position and angle of the spatula are estimated based on the image captured by the camera 23. For example, an inference model generated by machine learning using a data set of various spatula shapes is prepared in the processor 31. The inference model, which receives the image captured by the camera 23 as input and outputs information on the position and angle of the spatula, is prepared in the processor 31 and is used to estimate the position and angle.

[0053] In the cooking assistance system 1, the trajectory and speed of the mixing motion are acquired based on the position and angle of the spatula. Furthermore, the start and end points of the mixing motion and the position in the depth direction are estimated based on the position and angle of the spatula. An AR marker or the like may be attached to the spatula, and the position and angle of the spatula may be measured by analyzing an image of the AR marker.

[0054] A depth sensor may be installed on the ceiling side of the cooking environment. Using the measurement results from the depth sensor, it is possible to obtain information on the height direction of cooking utensils such as spatulas, and based on the obtained information, it is possible to estimate the three-dimensional movement of the cooking utensils. By using the depth information in addition to the position information of the cooking utensils, it is possible to estimate whether the cook is mixing the surface part of the pot or the bottom part of the pot.

[0055] Furthermore, by using the force sensor 33, it is possible to obtain data on the strength (intensity) of the chef's movements.

[0056] The chef's strength of force may also be measured using other sensors, such as a piezoelectric sensor, optical sensor, strain gauge, capacitance sensor, or weight sensor. The sensor used to measure the strength of force may be provided on a cooking utensil, such as a pot, spatula, or chopsticks, rather than on the IH stove 11. Data indicating the chef's strength of force measured using a force sensor or the like is managed in synchronization with images captured by the camera 23, for example. By synchronizing the strength of force information with the images captured by the camera 23, it is possible to improve the accuracy of the start and end points of the mixing action.

[0057] By providing an acceleration sensor, angular velocity sensor, position sensor, attitude sensor (IMU), etc. to the cooking utensil 12A such as a spatula, it becomes possible to estimate the mixing direction and speed in three dimensions.

[0058] <Information Included in Navigation Data Set> FIG. 4 is a diagram showing an example of information included in the navigation data set.

[0059] As shown in FIG. 4, the navigation data set includes information on cooking conditions and information on cooking steps.

[0060] The cooking conditions are conditions that do not change throughout the cooking process. The information about the cooking conditions includes at least one of information about the cooking appliance, the cook, and the ingredients.

[0061] Cooking equipment information includes stove specifications, pot specifications, individual characteristics of the pot, type / specification of other appliances, placement of the pot (when fixed), and information on the date, time, and location of cooking. For example, stove specifications represent the specifications of the induction stove 11 used for recording cooking. Chef information includes the attributes of the recording chef and information on the details of pre-training. Ingredient information includes the genre and name of the dish, the finished amount, the type and amount of ingredients, and how the ingredients were cut (preparation method).

[0062] A cooking process is a state or task defined as a cooking process. Information about the cooking process is appropriately expressed as information on a time axis. The information about the cooking process includes at least one of the process configuration, initial state, target state, transient state, and cooking task.

[0063] The process configuration information includes information about the overall configuration of the cooking process. The process configuration information indicates, for example, the number of cooking steps, the order, the ingredients used in each cooking step, and the cooking utensils used. The process configuration information includes ingredient information that indicates the ingredients used in each cooking step.

[0064] The initial state information includes the initial temperature of the stove, the initial temperature of the pot, the initial temperature of the ingredients, and the weight of the ingredients added.

[0065] The information on the target state includes a target index value for each process and information on a target image for each process. The target index value for each process is, for example, one of the TTW values ​​used to determine the end timing of a cooking process. The target state may also be set using other target index values ​​that can be detected by a sensor, such as the number of times the ingredients are stirred or turned over.

[0066] Transient state information includes a TTW profile showing the time series of TTW values, an image of the inside of the pot during cooking, the position / movement of the pot during cooking, and information on the ambient temperature / humidity.

[0067] Cooking task information includes stove heat settings, the order in which ingredients are added (when multiple ingredients are added), how ingredients are added, how ingredients are stirred, whether the pot lid is placed on or removed, and other task details. Also included as cooking task information is reference task information, which serves as a reference for actions in cooking tasks. The reference task information represents the actions to be aimed for when recreating cooking.

[0068] For example, the information on the mixing task includes reference mixing task information, which is reference task information for the mixing task. The reference mixing task information includes information indicating the feature amounts of the actions that define the reference mixing task.

[0069] FIG. 5 is a diagram showing an example of reference mixing work information.

[0070] If the cooking process can be reproduced by mixing in the same way as the chef did during the recorded cooking, then the chef's mixing is the reference mixing task. The reference mixing task is defined by the characteristics of the speed, stroke, direction, and force of the chef's mixing motion with a spatula.

[0071] 5, the reference mixing work information records information indicating the speed, stroke, direction, and strength of the movement at each time. Information indicating at least one of the speed, stroke, direction, and strength of the movement may be recorded in the reference mixing work information, rather than information indicating all of them. Information indicating other characteristic quantities may also be recorded in the reference mixing work information.

[0072] In this way, data such as average values ​​indicating constant movements in the cooking process and time-series data indicating changes in movements in the cooking process are used as feature quantities to define the reference stirring task. The reference stirring task information shown in Fig. 5 is data composed of time-series data indicating changes in movements in the cooking process.

[0073] <Functional Configuration> FIGS. 6 and 7 are block diagrams showing an example of the functional configuration of the cooking assistance system 1. As shown in FIG.

[0074] 6 and 7, the cooking assistance system 1 includes a recording processing unit 101, a navigation data set creation unit 102, and a reproduction processing unit 103. At least a part of the components shown in Fig. 6 and 7 is realized by the processor 31 executing a predetermined program.

[0075] Regarding the recording processing unit 101: The recording processing unit 101 in Fig. 6 performs processing during recording cooking. A recording system is realized by the recording processing unit 101. The recording processing unit 101 is composed of a cooking environment measurement unit 111, a sensor data acquisition unit 112, a user input detection unit 113, and an information recording unit 114.

[0076] The cooking environment measurement unit 111 controls the cooking environment measuring device 26 to measure the environment in which the cooking to be recorded is performed, for example, at the start of the cooking to be recorded. The cooking environment information measured by the cooking environment measurement unit 111 is supplied to the information recording unit 114.

[0077] The sensor data acquisition unit 112 acquires sensor data used to generate a navigation data set, for example, after recording cooking has started. For example, during cooking in recording cooking, time series data is acquired, such as time measured by the timer inside the processor 31, temperature measured by the temperature sensor 32, and force and weight measured by the force sensor 33 during the cooking. Time series data of sensor data obtained by measuring the chef's movements is also acquired. The sensor data resulting from the measurement of the chef's movements includes image data such as RGB images and depth images, and data such as acceleration and angular velocity measured by sensors mounted on cooking utensils such as spatulas. The sensor data acquired by the sensor data acquisition unit 112 is supplied to the information recording unit 114.

[0078] The user input detection unit 113 detects input by the chef using input devices such as the microphone 24 or operation buttons, and outputs information representing the content of the input to the information recording unit 114. For example, if the chef or an assistant staff member presses an operation button at a timing that is important for creating a navigation data set, information representing this is output to the information recording unit 114, and a timestamp is recorded. The recorded timestamp is referenced in the navigation data set creation process and is used for organizing cooking steps, etc. The timing of recording may be input not by operating an operation button, but by the chef's actions or statements during the recorded cooking.

[0079] The information recording unit 114 generates recording data by recording, in a predetermined format, the cooking environment information supplied from the cooking environment measurement unit 111 and the sensor data supplied from the sensor data acquisition unit 112. When the recording cooking is completed, recording data is generated that records the sensor data and other data acquired from the start to the end of the recording cooking. The recording data generated by the information recording unit 114 is supplied to the navigation dataset creation unit 102 at a timing such as when a navigation dataset is created.

[0080] Navigation Data Set Creation Unit 102 The navigation data set creation unit 102 is made up of a process organization unit 121 , a target state setting unit 122 , a cooking task setting unit 123 , an information recording unit 124 , and a navigation data set management unit 125 .

[0081] The process organization unit 121 organizes the processes by dividing the entire cooking process of the recorded cooking into multiple cooking processes. Information about each cooking process organized by the process organization unit 121 is supplied to the target state setting unit 122, the cooking task setting unit 123, and the information recording unit 124.

[0082] The target state setting unit 122 sets a target state for each cooking process set by the process organization unit 121. Information on the target states set by the target state setting unit 122 is supplied to the cooking task setting unit 123 and the information recording unit 124. The target state is set appropriately according to the cooking task set by the cooking task setting unit 123.

[0083] The cooking operation setting unit 123 sets the content of the cooking operation for each cooking process set by the process organization unit 121. For example, the cooking operation is set as an operation for bringing the state of the heated object, such as the ingredients used in each cooking process, into a state defined as a target state. The cooking operation setting unit 123 also sets multiple types of feature quantities that define the reference cooking operation for each cooking process. Information indicating the content of the cooking operation set by the cooking operation setting unit 123 is supplied to the information recording unit 124 and the target state setting unit 122.

[0084] The information recording unit 124 records the record data supplied from the recording processing unit 101 and the information supplied from the process organization unit 121, target state setting unit 122, and cooking task setting unit 123 in a predetermined format, thereby creating a navigation dataset including each piece of information. The navigation dataset includes information on the cooking environment measured during the recorded cooking, time-series sensor data, information on each cooking step, the target state of the cooking step, and information indicating the content of the cooking task. The navigation dataset created by the information recording unit 124 is provided to the navigation dataset management unit 125.

[0085] The navigation dataset management unit 125 manages the navigation dataset created by the information recording unit 124 by recording it, for example, in the storage 34. In the storage 34, navigation datasets for various dishes are recorded for each dish.

[0086] A database (DB) for the navigation datasets may be prepared in the Web server 42, and the navigation datasets may be managed by the Web server 42. In this case, the navigation dataset management unit 125 transmits the navigation datasets created by the information recording unit 124 to the Web server 42, and the Web server 42 manages the navigation datasets.

[0087] 7 performs processing during the reproduction cooking. The reproduction system is realized by the reproduction processing unit 103. The reproduction processing unit 103 is composed of a navigation data set acquisition unit 131, a cooking environment measurement unit 132, a sensor data acquisition unit 133, a user input detection unit 134, a control unit 135, and an information presentation unit 136.

[0088] The navigation dataset acquisition unit 131 acquires the navigation dataset for the dish to be recreated from the storage 34 or the web server 42. The navigation dataset is acquired, for example, when the dish to be recreated is selected by the chef. The navigation dataset acquired by the navigation dataset acquisition unit 131 is supplied to the control unit 135.

[0089] The cooking environment measurement unit 132 controls the cooking environment measuring device 26 to measure the environment in which the reproduction cooking is performed, for example, at the start of the reproduction cooking. Information on the cooking environment measured by the cooking environment measurement unit 132 is supplied to the control unit 135.

[0090] The sensor data acquisition unit 133 acquires sensor data measured by each sensor, for example, after the reproduction cooking has started. For example, during the reproduction cooking, time series data is acquired, such as the time measured by the timer inside the processor 31, the temperature measured by the temperature sensor 32, and the force and weight measured by the force sensor 33 during the cooking. Time series data of sensor data obtained by measuring the movements of the user performing the reproduction cooking is also acquired. The sensor data, which is the measurement result of the user's movements, includes image data such as RGB images and depth images, and data such as acceleration and angular velocity measured by sensors mounted on cooking utensils such as spatulas. The sensor data acquired by the sensor data acquisition unit 133 is supplied to the control unit 135.

[0091] The user input detection unit 134 detects input by the chef via input devices such as the microphone 24 and operation buttons. Information indicating the content of the input detected by the user input detection unit 134 is supplied to the control unit 135.

[0092] The control unit 135 performs various processes to assist in the reproduction cooking based on the navigation data set supplied from the navigation data set acquisition unit 131. For example, the control unit 135 controls the IH stove 11 to control heating of the heating object. The heating control is performed in accordance with the cooking environment measured by the cooking environment measurement unit 132 and the sensor data acquired by the sensor data acquisition unit 133. The control unit 135 controls the presentation of information by the information presentation unit 136.

[0093] The control unit 135 also controls the progress of the reproduction recipe based on the navigation data set. The progress of the reproduction recipe is triggered by appropriate input from the chef.

[0094] For example, when the target state is defined by the elapsed time from the start of the target cooking process (the cooking process currently being executed), the control unit 135 starts the next cooking process when the elapsed time from the start of the cooking process reaches the time defined as the target state.

[0095] When the target state is defined by the temperature of the object to be heated or the heating medium, the control unit 135 starts the next cooking process when the temperature measured by the temperature sensor 32 reaches the temperature defined as the target state.

[0096] When the target state is defined by the weight of the object to be heated, the control unit 135 starts the next cooking process when the force and weight measured during the operation by the force sensor 33 reach the weight defined as the target state.

[0097] The information presentation unit 136 performs navigation to assist the chef in replicating the cooking based on the navigation data set. For example, the information presentation unit 136 displays a cooking navigation screen including information indicating the contents of the cooking work on the display 21. The presentation of various information to assist the chef in replicating the cooking is also performed according to the cooking environment measured by the cooking environment measurement unit 132 and the sensor data acquired by the sensor data acquisition unit 133.

[0098] <<Details of Recording / Reproducing Process>> The details of the recording / reproducing process of the cooking process will be described.

[0099] The cooking process recording process and the reproduction process may be performed in the same device (cooking assistance system), or may be performed in different devices. Here, we will explain the case where the cooking process recording process and the reproduction process are performed in the same device, the cooking assistance system 1 shown in Figure 3.

[0100] <Record Cooking> A series of processes for record cooking will be described with reference to the flowchart of FIG.

[0101] Step S1: Creating a new recipe The information recording unit 114 (FIG. 6) of the recording processing unit 101 creates a new navigation dataset (file) for the dish to be recorded. When creating a new navigation dataset, the information recording unit 114 records some of the information about the cooking conditions, for example, based on input by the chef. The chef will enter information about himself and the ingredients using, for example, a screen displayed on the display 21.

[0102] Step S2: Cooking environment measurement The cooking environment measurement unit 111 measures the cooking environment and acquires information on the specifications and characteristics of the cooking appliances. For example, the cooking environment measurement unit 111 acquires information on the specifications of the IH stove 11, the pots that are the cooking utensils 12, and other appliances from a database provided in the storage 34 or the web server 42.

[0103] In the DB of the storage 34 and the web server 42, information on appliances such as induction cooktops that are compatible with the cooking assistance system 1 is provided by manufacturers and managed in association with model number information. The cooking environment measurement unit 111, for example, acquires the model number information of each appliance automatically or based on input by the chef via a UI such as a screen displayed on the display 21, and acquires information on the cooking appliances by referencing the DB based on the acquired model number information. The information acquired by the cooking environment measurement unit 111 is recorded by the information recording unit 114 as information on the cooking appliances. When measuring the cooking environment, information on the stove specifications, pot specifications, individual characteristics of the pots, types / specifications of other appliances, and the placement of the pots (when fixed) is recorded.

[0104] Step S3: Start of cooking When cooking starts, such as immediately before heating starts, the information recording unit 114 records information about the initial state of the cooking appliance based on the information measured by the sensor data acquisition unit 112. For example, information about the initial temperatures of the IH stove 11 and the pot is recorded.

[0105] The initial temperatures of the IH stove 11 and the pot are measured using the temperature sensor 32 installed in the cooking assistance system 1 and acquired by the sensor data acquisition unit 112. For example, the temperature of the cooking table in contact with the pot is recorded as the temperature of the IH stove 11. If measurable, the temperature inside the IH stove 11 is also recorded.

[0106] Step S4: During cooking During cooking, the information recording unit 114 chronologically records various information, such as sensor data measured by sensors installed in the cooking assistance system 1 and acquired by the sensor data acquisition unit 112. For example, information such as the initial temperature of the ingredients, the weight of the ingredients added, the TTW profile, images of the inside of the pot during cooking, the position / movement of the pot during cooking, the ambient temperature / humidity, and the stove heat setting is recorded. In addition, images of the chef's movements and data such as acceleration and angular velocity measured by sensors installed in cooking utensils such as spatulas are recorded.

[0107] Here, the recording of main information will be described.

[0108] Information about the heat setting of the IH stove 11 is acquired and recorded as cooking task information. In this case, the sensor data acquisition unit 112 detects and acquires the value of the heat setting set by the chef by operating an operation panel or the like provided as a UI of the cooking assistance system 1.

[0109] A TTW profile, which is time-series data of time / temperature / weight, is generated by the sensor data acquisition unit 112 and recorded as transient state information. The time constituting the TTW profile is measured by a real-time clock built into the processor 31. As the temperature constituting the TTW profile, for example, the temperature of the bottom of a pot is measured by a temperature sensor 32 built into the IH stove 11. As the weight constituting the TTW profile, for example, the weight of ingredients in the pot is measured by a force sensor 33 built into the IH stove 11.

[0110] The images of the inside of the pot are acquired by the sensor data acquisition unit 112 and recorded as information on the transient state. For example, images of the inside of the pot, generated based on images taken by the camera 23, are recorded in chronological order.

[0111] The initial temperature of the ingredients is acquired and recorded as initial state information. The initial temperature of the ingredients is the temperature of the ingredients at the time (just before) they are added to the pot. For example, the temperature of the ingredients is measured by a chef using a cooking thermometer.

[0112] The weight of the ingredients added is acquired and recorded as initial state information. Instead of the weight in a normal recipe, for example, the weight of the ingredients actually added to the pot and the timing of adding them are measured by the force sensor 33 and used for recording.

[0113] The above information is recorded repeatedly during cooking.

[0114] Step S5: Cooking End The information recording unit 114 records the time when cooking is ended and stops recording the sensor data. The data in which each piece of information has been recorded as described above is supplied to the navigation data set creation unit 102 as recorded data.

[0115] <Navigation Data Set Creation> A series of processes for creating a navigation data set will be described with reference to the flowchart of FIG.

[0116] Step S11: Process organization Process organization is the task of dividing the entire cooking process into multiple cooking processes. The process organization unit 121 of the navigation data set creation unit 102 divides the entire cooking process from the start to the end of the cooking to be recorded into multiple sections based on, for example, an input by the chef, and sets each section as a section that constitutes one cooking process.

[0117] The process organization of the navigation dataset defines the cooking process as a whole, i.e., what cooking task will be performed in the next cooking step when a certain goal state is reached. In the process organization, the boundaries of each cooking step are set at points where timing is important, such as changing the heat or adding ingredients, and at points where it is important to judge the amount of water evaporation, temperature, etc.

[0118] The information on each cooking step set by the step organization unit 121 in this manner is recorded by the information recording unit 124 as step configuration information.

[0119] Step S12: Setting the target state for each process The target state setting unit 122 sets the target state for each cooking process using a quantitative index value that can be measured by the cooking assistance system 1. Specifically, the target state setting unit 122 selects one of the following values: time (time elapsed since the start of the cooking process), temperature (temperature of the bottom of the pot detected by the temperature sensor 32), or weight (weight of the ingredients in the pot measured by the force sensor 33), and sets it as the index value that defines the target state.

[0120] Information on the index values ​​defining the target state set by the target state setting unit 122 is recorded by the information recording unit 124 as information on the target state.

[0121] Step S13: Setting cooking tasks for each step The cooking task setting unit 123 sets the details of the cooking tasks to be performed in each cooking step, for example, based on input from the chef. The information set by the cooking task setting unit 123 is recorded by the information recording unit 124 as cooking task information. Information on the order in which ingredients are added, the method for adding ingredients, the method for stirring ingredients, putting on and taking off a pot lid, and other task details is recorded. The cooking task setting unit 123 generates reference task information by analyzing images of the chef's movements and data such as acceleration and angular velocity measured by sensors mounted on cooking utensils such as spatulas. The reference task information set by the cooking task setting unit 123 is recorded by the information recording unit 124 as cooking task information.

[0122] A navigation data set is created by recording information on the target state and cooking tasks for each cooking step set in the step scheduling (step S11). Step scheduling may be performed again after the target state is set, or after the cooking tasks are set.

[0123] <Reproduction Cooking> A series of processes for reproduction cooking will be described with reference to the flowchart in Fig. 10. In reproduction cooking, each cooking step is reproduced based on the navigation data set. The process flow for reproduction cooking is the same as the process flow for recording cooking.

[0124] Step S21: Reading navigation dataset When the user selects a dish to be reproduced, the navigation dataset acquisition unit 131 ( FIG. 7 ) of the reproduction processing unit 103 reads the navigation dataset. The dish to be reproduced is selected, for example, using a screen displayed on the display 21 by the information presentation unit 136. Specific examples of the screen display will be described later.

[0125] Step S22: Cooking environment measurement The cooking environment measurement unit 132 measures the cooking environment and acquires the specifications and characteristics of the cooking appliances used in the replicated cooking. The cooking environment measurement for the replicated cooking is performed in the same manner as the cooking environment measurement for the recorded cooking (step S2 in FIG. 8).

[0126] If there is a difference between the cooking environment of the reproduced cooking acquired by the cooking environment measurement unit 132 and the cooking environment of the recorded cooking represented by the information included in the navigation data set, the information presentation unit 136 presents information indicating this difference to the user and warns them. The user can change cooking appliances in response to the difference in the cooking environment presented on the screen of the display 21, etc., to make the cooking environment of the reproduced cooking closer to the cooking environment of the recorded cooking.

[0127] Step S23: Start of cooking As with the start of cooking in recorded cooking (step S3 in FIG. 8), the sensor data acquisition unit 133 measures the temperatures of the IH stove 11 and the pot as the initial state of the cooking appliances, etc. If the measured temperatures are significantly different from the initial state temperatures recorded in the navigation dataset, the information presentation unit 136 postpones the start of heating and warns the user.

[0128] Step S24: During cooking The control unit 135 references information about the cooking steps included in the navigation dataset and supports the reproduction of the cooking steps. That is, the control unit 135 measures the state of the reproduced cooking in real time using sensors and controls the cooking appliances to reproduce the cooking steps (time series of the TTW profile) described in the navigation dataset. The information presentation unit 136 presents information to support the user's work as appropriate.

[0129] Step S25: Cooking completes. When all cooking steps are completed, the cooking reproduction is complete and the dish is ready. Through this series of processes, the user can consistently create dishes with a high degree of accuracy. In addition, information on the cooking tasks for each cooking step is displayed on the cooking navigation screen, allowing the user to easily proceed with the cooking reproduction.

[0130] <<Specific Examples of Screen Displays>> Specific examples of screens that are displayed on the display 21 at each timing of the replicated cooking will be described.

[0131] <Menu Selection Screen> Fig. 11 is a diagram showing an example of a menu selection screen. The menu selection screen is a screen used to select a dish to be reproduced.

[0132] The menu selection screen in Figure 11 shows the screen when "Sichuan Mapo Tofu" is selected. The name of the dish is displayed large in the approximate center of the menu selection screen, and below that are the chef's name, the time required for the re-creation, the level of re-creation, and the number of steps. A large image (photo) of the dish is displayed to the left of the dish name.

[0133] The chef's name is the name of the chef who cooked the recorded dish. In the example of Figure 11, "Mr. A" is displayed as the name of the chef who cooked the recorded dish. The reproduction cooking time is the estimated time required for the reproduction cooking. The reproduction level indicates the difficulty of the reproduction cooking. The higher the reproduction level, the more difficult the reproduction cooking will be, such as if difficult cooking tasks are required or if the time spent on each cooking step needs to be minimized. The number of steps indicates the number of cooking steps in the entire reproduction cooking. These displays are switched each time the dish to be reproduced changes.

[0134] At the bottom of the menu selection screen, genre buttons 201 used to select a cuisine genre are displayed. In the example of Fig. 11, buttons operated to select each of the genres of Japanese cuisine, Chinese cuisine, French cuisine, Italian cuisine, and sweets are displayed as genre buttons 201. The user operates the genre button 201 to select the genre of cuisine they wish to cook. In the cooking assistance system 1, each dish for which cooking has been recorded is managed by categorized by genre.

[0135] After selecting the Chinese genre by operating the genre button 201, "Sichuan Mapo Tofu" is selected. After selecting the genre, the name of the dish, the chef who recorded the cooking, and the like may be selected hierarchically, and then a screen such as that shown in FIG. 11 may be displayed.

[0136] Typically, dishes made with the same ingredients and using the same procedures are treated as the same dish. However, in the cooking assistance system 1, chef-specific information acquired during the recording of the cooking, such as the heating profile and stirring method, is also recorded, so even dishes made with the same ingredients and using the same procedures are managed as different dishes depending on the chef who performed the recording. Even when the same chef performs the recording of the cooking, their skills and preferences may vary depending on the time period. Therefore, dishes may be distinguished based on the time period during which the recording of the cooking was performed, and information on the time period may be clearly indicated. Furthermore, the chef's affiliation at the time of the recording of the cooking and the affiliation at the time of the playback of the cooking (current time) may be clearly indicated as information about the chef.

[0137] If the skill level of the user who will be recreating the dish and their familiarity with the system can be evaluated in advance, the reproduction level will be displayed, allowing the user to select a dish that suits their skill level, etc. This makes it possible to ensure a high level of reproduction of the dish.

[0138] A favorite button 202 is displayed at the bottom left of the menu selection screen. The favorite button 202 is a button that is operated when displaying dishes that have been registered as favorite dishes.

[0139] A back button 203L and a forward button 203R are displayed on the left and right ends of the menu selection screen. The back button 203L and the forward button 203R are buttons that are operated to switch dishes. When the back button 203L or the forward button 203R is operated, the dish information, including the dish image, transitions in the same way as dishes arranged on a turntable are rotated, and information on another dish is displayed. The screen transition caused by button operation does not have to be a rotational transition, but may be a linear transition in the left-right or up-down direction.

[0140] A start button 204 and an ingredient information button 205 are displayed approximately in the center of the menu selection screen. The start button 204 is a button that is operated when the user has finished selecting a dish and is about to start the reproduction cooking. The ingredient information button 205 is a button that is operated when the user is checking the ingredients to be used in the reproduction cooking. When the ingredient information button 205 is operated, information such as the ingredients to be used in the reproduction cooking, the amount of ingredients to be used, the process in which the ingredients are used, the storage location of the ingredients, the cooking equipment and utensils to be used, etc. is displayed.

[0141] The user can check what ingredients and how much of each ingredient are needed. The user can also check the state in which the ingredients should be prepared. It may also be possible to display sorting, such as sorting ingredients by process order or by storage location. This allows the user to more reliably check the preparation status of ingredients.

[0142] A recipe search may be performed using ingredients that can be prepared as keywords. When an ingredient is selected by the user, recipe candidates that can be recreated using the selected ingredient are displayed as search results.

[0143] A language switch button 206 and a search button 207 are displayed in the upper right corner of the menu selection screen, and a favorites button 208 is displayed below them. The language switch button 206 is a button used to switch the navigation language. The search button 207 is a button operated when searching for a dish by entering characters. The favorites button 208 is a button operated when registering the displayed dish as a favorite dish.

[0144] It may be possible to allow users to register on the menu selection screen. By registering a user, it becomes possible to associate and manage attribute information with the user, such as the number of past recipe replications, the history of dishes made through replications, training status, difficult tasks, and proficiency in operating equipment. The attribute information can be used, for example, to display only dishes at a replication level that matches the user's skills on the menu selection screen, automatically skip training when checking the cooking process, or highlight supplementary information for tasks that the user is difficult to perform.

[0145] <Cooking Process Confirmation Screen> Fig. 12 is a diagram showing an example of a cooking process confirmation screen. When the start button 204 on the menu selection screen is pressed, the cooking process confirmation screen starts to be displayed.

[0146] The cooking process confirmation screen is a screen used to confirm the cooking process of the replicated recipe. The entire cooking process of the replicated recipe is divided into cooking steps, which are numbered sequentially. Text, images, etc. outlining each cooking step are presented sequentially. The cooking process may be presented using audio from the speaker 22.

[0147] The numbers of all cooking steps are displayed in a row at the top of the cooking process confirmation screen. Of the numbers displayed in a row, the number of the cooking step being introduced is highlighted by inverting the color, for example. In the example of Figure 7, the cooking step being introduced is step 7 out of a total of 17 steps. The user can confirm which step out of all the steps is being introduced. The numbers of cooking steps that require particular attention among all the cooking steps are highlighted by underlining, for example. For example, the numbers of cooking steps with complex cooking operations are highlighted.

[0148] As shown in Fig. 12, a rectangular display area 211 is formed below the cooking step number, and an overview of the cooking step is displayed in the display area 211. In the example of Fig. 12, text explaining that step 7 is the step of adding chili bean paste, garlic, and black beans to the pot is displayed along with a cooking video. The cooking video is, for example, a video recording the chef's actions during the cooking process.

[0149] A back button 212L and a forward button 212R are displayed on the left and right sides of the display area 211. The back button 212L is a button that is operated to display an overview of the previous cooking step. The forward button 212R is a button that is operated to display an overview of the next cooking step.

[0150] If the task introduced on the cooking process confirmation screen is complicated, task training may be provided. The task training may be provided after an overview of all cooking steps is presented. As the task training, the user performs cooking tasks such as mixing without using ingredients. For example, the user's task score is calculated based on sensor data detected during the task training, and pass / fail is determined using reference task information included in the navigation dataset.

[0151] For example, if the error in the speed of the user's stirring during the work training is within 1% of the average speed of the standard stirring work, 10 points will be given, if it is 1-3%, 9 points will be given, and so on, and a score for the user's work is calculated. A pass / fail judgment is made for a score of 10-8 points, and a fail for a score of 7 points or less, and the judgment result is presented to the user. Only if the user passes the work training, the display 21 may transition to the cooking navigation screen.

[0152] The top of the cooking process confirmation screen also displays information such as the name of the dish, the name of the chef who recorded the cooking, the cooking time, the reproduction level, the number of steps, etc. The top right of the cooking process confirmation screen displays a start button 213 that is operated to start the reproduction cooking process.

[0153] The information used to display the cooking process confirmation screen may be included in the navigation dataset in advance, or may be automatically generated based on the information included in the navigation dataset.

[0154] <Cooking navigation screen> Configuration of the cooking navigation screen Fig. 13 is a diagram showing an example of the cooking navigation screen. When the overview of all cooking steps has been introduced on the cooking process confirmation screen, or when the start button 213 is pressed, the display on the display 21 transitions to the cooking navigation screen.

[0155] The cooking navigation screen is a screen used to present information about cooking processes. As shown in Figure 13, the cooking navigation screen is mainly composed of a time-series data area 221 and cooking process areas 222-1 and 222-2, which are three strip-shaped areas of approximately the same height. With the time-series data area 221 at the center, the cooking process area 222-1 is located above the time-series data area 221, and the cooking process area 222-2 is located below the time-series data area 221. The time-series data area 221 is located between the cooking process areas 222-1 and 222-2. A narrow process number area 223 is formed below the cooking process area 222-2.

[0156] The display of the cooking navigation screen configured in this way changes automatically over time. The horizontal direction of the cooking navigation screen indicates the time direction. The position indicated by the thick line image L1 indicates the current time. The part to the left of image L1 is displayed grayed out as shown in FIG. 13. The grayed out display indicates that the information to the left of image L1 is past information. The information to the right of image L1 is future information.

[0157] In the example of Figure 13, section T2 indicated by the arrow is the section of step 4, which is the current cooking process, and section T3 is the section of step 5, which is the next cooking process. Section T1 is the section of step 3, which is the cooking process immediately before the current cooking process. Only a portion of the sections between steps 3 and 5 is displayed, not the entire section.

[0158] The time-series data area 221 is a display area for information indicating time-series data. A graph showing time-series data (first time-series data) of sensor data detected during the recorded cooking and recorded in the navigation data set, and a graph showing time-series data (second time-series data) of sensor data detected during the reproduced cooking are displayed in the time-series data area 221. Depending on the cooking process, information other than information indicating the time-series data is also displayed in the time-series data area 221 as appropriate.

[0159] FIG. 14 is an enlarged view of the time-series data area 221. As shown in FIG.

[0160] In the example of Fig. 14, the start time of section T2 is displayed slightly to the left of the current time. The state shown in Fig. 14 is the state immediately after the start of the current cooking process (1 second later).

[0161] Graph G1 displayed in section T1 is a graph showing time-series data for the previous cooking step, and graph G2 displayed in section T2 is a graph showing time-series data for the current cooking step. In the example of FIG. 14 , the words "mixing strength" are displayed next to image L1, indicating that graph G2 is time-series data for the mixing strength of the stirring operation. The portion of time-series data area 221 for section T2 is displayed in a conspicuous color, such as orange, to distinguish it from the section for the next cooking step. Graph G3 displayed in section T3 is a graph showing time-series data for the next cooking step. Graphs G1, G2, and G3 are information displayed based on time-series data from sensor data detected during the recorded cooking, which is the cooking to be reproduced, and recorded in the navigation dataset.

[0162] The graphs G1, G2, and G3 are displayed moving leftward over time, similar to information about the next cooking step. For example, the graphs G1, G2, and G3 are all displayed moving so that the measurement results of the sensor data at the time during the recorded cooking corresponding to the current time are displayed at the position of the current time indicated by image L1. In the state shown in FIG. 14 , the value of graph G2 at the current time position indicates the value one second after the start of the cooking step corresponding to cooking step 4, measured during the recorded cooking as the "mixing strength" of the chef's stirring. In this way, the range of time-series data displayed based on the navigation data set is switched by moving the graphs G1, G2, and G3 as a whole leftward.

[0163] In the portion of the time-series data area 221 to the left of the image L1, a graph G11 is displayed in a conspicuous color and superimposed on other graphs. The graph G11 is information showing the time-series data of the sensor data detected during the reproduction recipe. The graph G11 is updated in real time and displayed so as to show the latest sensor data status at the reference position, based on the current time position shown in the image L1.

[0164] The portion of graph G11 displayed in section T1 represents time series data of the same type of sensor data as that represented by the time series data of the previous cooking step (graph G1), while the portion of graph G11 displayed in section T2 represents time series data of sensor data of "mixing strength," the same type of sensor data as that represented by the time series data of the current cooking step (graph G2).

[0165] While viewing the display of graph G11, which is updated in real time, the user performs cooking work so as to bring the current value of graph G11 closer to the value of the graph displayed based on the navigation data set. In the state shown in Fig. 14, the closer graph G11 is to graph G2, the more effectively the strength of the user's stirring work reproduces the strength of the chef's stirring during the recorded cooking.

[0166] 13, the cooking process area 222-1 and the cooking process area 222-2 are display areas for information about cooking processes. The information about cooking processes includes information about the contents of cooking work.

[0167] In the example of FIG. 13 , the cooking task of step 4, which is the current cooking process, is to add beef and stir. Text instructing the user to add beef and stir is displayed to the right of image L1 in cooking process area 222-2. A cooking animation showing how to stir the beef is also displayed to the right of the text. Above the text, an icon is displayed indicating that the cooking process of step 4 will take 70 seconds and that one second has passed since the start of the cooking process. This information is displayed statically. Information indicating the content of the task of the current cooking process is presented as appropriate using audio from speaker 22.

[0168] In the example of Figure 13, the cooking task of the next cooking process, step 5, is to add seasoning A and stir. Text instructing the user to add seasoning A and stir is displayed in section T3 of the cooking process area 222-1. Above the text, an icon is displayed indicating that the cooking process time for step 5 is 30 seconds. This information is displayed by shifting to the left.

[0169] The process number area 223 is a display area for information showing the progress of the cooking process. In the example of Fig. 13, the cooking process numbers are used to indicate that the cooking process has been completed up to step 3 and the current cooking process is step 4. The numbers of cooking processes that require particular attention among all the cooking processes are highlighted using underlines, etc. For example, the numbers of cooking processes with complex cooking operations are highlighted.

[0170] In this way, the cooking navigation screen displays information such as the task details of the current cooking step, the cooking video corresponding to the current cooking step, the time of the current cooking step and the elapsed time, etc. The cooking navigation screen also displays information such as the task details of the next cooking step, the progress status of all steps, sensor data recorded in the navigation data set, and sensor data detected during the replicated cooking.

[0171] From the cooking navigation screen display, the user can check the details of the tasks in the current cooking process, and while performing the instructed tasks, can check the details of the tasks in the next cooking process. In addition, the user can check how much their own tasks differ from the chef's tasks by viewing the graph G11, which is updated in real time.

[0172] Transition of the cooking navigation screen Fig. 15 is a diagram showing an example of the cooking navigation screen after a predetermined time has elapsed. During the reproduction cooking, the display of the cooking navigation screen shown in Fig. 13 changes to the state shown in Fig. 15. The cooking navigation screen shown in Fig. 15 is the screen when the current cooking process is still at step 4.

[0173] In the example of Figure 15, information about step 4, which is the current cooking process, is displayed in the cooking process area 222-2 at the same position as in Figure 13. 22 seconds have elapsed since the start of the cooking process. The cooking navigation screen shown in Figure 15 is the screen 21 seconds after the state shown in Figure 13.

[0174] On the other hand, information about the next cooking step, step 5, is displayed in the cooking step area 222-1 at a position closer to the position of image L1 indicating the current time than the position in Fig. 13. In the example of Fig. 15, section T3, which is the section for step 5, is displayed extending to the left of the cooking navigation screen.

[0175] Fig. 16 is a diagram showing an example of the cooking navigation screen after a predetermined time has elapsed. The cooking navigation screen shown in Fig. 16 is the screen at the timing after the current cooking process has switched to step 5. After further time has elapsed from the state in Fig. 15 and step 4 has finished, the current cooking process will switch to step 5.

[0176] 16, the information about step 5, which had previously been moved and displayed, is now displayed as the new current cooking process information, fixed in a position to the right of image L1 in cooking process area 222-1. Image L1 has also switched from being displayed across both the time-series data area 221 and the cooking process area 222-2 to being displayed across both the time-series data area 221 and the cooking process area 222-1.

[0177] 16, information indicating the details of the work in step 6, the next cooking step, is displayed in section T4 of the cooking step area 222-2. The information indicating the details of the work in step 6 is displayed by gradually moving leftward over time so that the instructions become clearer.

[0178] FIG. 17 shows the transition of the cooking navigation screen described above.

[0179] The top, middle, and bottom rows of Fig. 17 respectively show the cooking navigation screens of Fig. 13, Fig. 15, and Fig. 16. In the examples of the top and middle rows of Fig. 17, information on step 4, which is the current cooking process, is displayed fixedly, while information on step 5, which is the next cooking process, is displayed moving leftward as if approaching the position of the current time, which serves as the reference.

[0180] In addition, in the example at the bottom of Figure 17 after the current cooking process has switched to step 5, the information for the current cooking process, step 5, is displayed fixedly, while the information for the next cooking process, step 6, is displayed moving to the left as if approaching the position of the current time, which is the reference time.

[0181] In this way, on the cooking navigation screen, information about the current cooking step is displayed fixedly, and information about the next cooking step is displayed by gradually shifting to the left over time so that the instructions become clearer. Also, when switching to the next cooking step, the information that had been shifting and displayed up until then is displayed fixedly as information about the new current cooking step, and information about the next cooking step is displayed by gradually shifting to the left over time so that the instructions become clearer.

[0182] The speed at which the information about the next cooking step moves varies depending on the time required for the current cooking step. The longer the time required for the current cooking step, the slower the speed at which the information about the next cooking step moves. Conversely, the shorter the time required for the current cooking step, the faster the speed at which the information about the next cooking step moves.

[0183] This type of display is achieved using cooking process area 222-1 and cooking process area 222-2. Each time the current cooking process changes, the information about the current cooking process alternates between being fixedly displayed in cooking process area 222-1 and being fixedly displayed in cooking process area 222-2. Also, each time the current cooking process changes, the information about the next cooking process alternates between being moved within cooking process area 222-1 and being moved within cooking process area 222-2.

[0184] This transition of the cooking navigation screen allows the user to intuitively understand whether the next cooking step is imminent, and also allows the user to prepare for the next cooking step while working on the current cooking step.

[0185] Regarding switching between cooking processes As described above, the conditions for switching between cooking processes are set as a target state using the TTW value and the like. The conditions for switching between cooking processes may be set using sensor data other than the TTW value, such as the amount of water evaporation, sound, vibration, acceleration, and images. Conditions that combine multiple types of sensor data may be set instead of conditions that use a single type of sensor data. When conditions that combine multiple types of sensor data are set, multiple graphs showing the time series data of each sensor data may be displayed in the time series data area 221 of the cooking navigation screen, or one graph showing the time series data of a single sensor data item of interest may be displayed.

[0186] The weight of the ingredients may be measured before the ingredients are added, or may be measured during (after being added to) the cooking simulation by the cooking assistance system 1. When measuring ingredients during the cooking simulation, information about the current cooking process is displayed, including text instructing the content of the cooking process and a cooking video, as well as information indicating the current weight of the ingredients.

[0187] The cooking process is switched when the sensor data detected during the replica cooking meets a predetermined condition as the target state. If the target state condition is set using temperature or water evaporation, the cooking process is switched when the temperature or water evaporation detected during the replica cooking reaches a predetermined value, or when the temperature of the ingredients reaches a certain temperature and the water evaporation rate after a predetermined time has passed reaches a predetermined value. The time-series data of the sensor data detected during the replica cooking may be compared with the time-series data of the sensor data recorded in the navigation data set, and the speed at which the information about the next cooking process moves may be adjusted based on the comparison of the time-series data. For example, if the temperature or water evaporation rate during the replica cooking is small, the speed at which the information about the next cooking process moves may be slowed down, or if the temperature or water evaporation rate is large, the speed at which the information about the next cooking process moves may be increased.

[0188] The cooking process may be switched when the amount of water evaporation detected after the user performs a stirring operation reaches the amount of water evaporation set as the target state. In this case, for example, the control unit 135 detects whether or not the user performed a stirring operation during the replicated cooking based on an image captured by the camera 23.

[0189] If stirring has not been performed when the amount of water evaporation detected during the replicated cooking reaches a predetermined threshold, such as two-thirds of the amount of water evaporation set as the target state, supplemental information instructing the user to start stirring is presented. The target state conditions are set using the number of stirrings and the amount of water evaporation, and if it is difficult to perform the preset number of stirrings in the remaining time, adjustments such as heat adjustment may be made to slow down the rate of water evaporation.

[0190] This automatic adjustment, which reflects the cooking process status, is performed only for conditions that have little effect on the reproducibility of the finished dish. In this example, the number of stirrings and the amount of water evaporation are conditions that have a large effect on the reproducibility, while the rate of water evaporation is a condition that has a small effect on the reproducibility.

[0191] For tasks that are prone to timing differences, such as adding ingredients, the cooking process may be switched when a button displayed on the cooking navigation screen is operated.

[0192] FIG. 18 is a diagram showing an example of how the switching buttons are displayed.

[0193] In the example of FIG. 18 , the current cooking process is step 12. The cooking task of step 12 is dissolving potato starch in water. A switch button 231 is displayed in the time-series data area 221. When the user presses switch button 231 after putting potato starch into a bowl or container and finishing the task of dissolving it in water, the current cooking process switches to step 13.

[0194] In this way, there are cooking steps in which information other than time-series data is displayed in the time-series data area 221. While the switching button 231 is displayed in the time-series data area 221, the information for the next cooking step may be displayed in a moving manner or may be displayed fixedly. The button used to switch cooking steps may be provided on the cooking navigation screen, or may be provided in the cooking assistance system 1 as a physical button.

[0195] - Display of time-series data The sensor data recorded in the navigation dataset and the sensor data detected during the reproduction cooking are presented to the user as time-series data using graphs, etc. The graph showing the sensor data recorded in the navigation dataset and the graph showing the sensor data detected during the reproduction cooking are displayed synchronized with each other, starting from the start point of the cooking process.

[0196] Here, the information displayed using graphs, etc. is time series data obtained by converting information obtained by sensors installed in the cooking assistance system 1 and cooking utensils into temperature, wave number (wavelength, frequency), amount of water evaporation, weight, mixing strength, mixing speed, estimated amount of chemical change (Maillard reaction, etc.), heat transfer simulation values, etc.

[0197] By presenting the user with the sensor data recorded in the navigation dataset and the sensor data detected during the re-creation cooking in a comparable state, the user is made aware of the need to work to minimize the difference, thereby improving the accuracy of the cooking reproduction. Furthermore, by being able to confirm that there is no difference with the navigation data, the accuracy of the work can be guaranteed.

[0198] Feedback may be provided to the user based on the difference between the sensor data recorded in the navigation data set and the sensor data detected during the replicated cooking. For example, if the user's stirring speed is slower than the average speed of the reference stirring operation, supplemental information such as an instruction to "mix faster" may be provided to the user. If the information presented in a graph or the like is the mixing speed, the average speed of the reference stirring operation may be presented as a target value, and a warning may be displayed, such as by turning part of the screen red, as the difference between the target value and the average speed during the replicated cooking becomes larger.

[0199] 19 to 21 are diagrams showing examples of displaying supplemental information. The supplemental information is information that supplements the cooking task. In addition to displaying the cooking navigation screen, the supplemental information is also presented using voice, as appropriate.

[0200] In the example of Fig. 19, speech bubble 251 containing the words "Please stop cooking occasionally" is displayed with the time series data of the current cooking step as the source of the speech bubble. In the example of Fig. 20, speech bubble 252 containing the words "Please prepare combined seasoning A" is displayed with the time series data of the current cooking step as the source of the speech bubble. Speech bubbles 251 and 252 are displayed in positions that do not overlap with information about the current cooking step or the next cooking step.

[0201] When recreating a recipe, the user performs the necessary steps by referring to the text, cooking video, and audio guidance displayed on the cooking navigation screen. However, there are times when supplemental instructions, such as "temporarily stop stirring" or "prepare the next ingredient," are necessary. In such situations, the displays shown in Figures 19 and 20 make it possible to provide supplemental explanations of the necessary steps at the appropriate time.

[0202] The supplemental information may also be information indicating that the work is proceeding as scheduled, such as "wait until heating is complete," even though no user action is required. Fig. 21 shows an example of the display of supplemental information explaining waiting. In this case, the supplemental information may be displayed superimposed on information instructing the content of the work in the current cooking process, as in speech bubble 253 in Fig. 21.

[0203] These supplemental information are displayed when the sensor data detected during the replicated cooking meets preset conditions. For example, the supplemental information "Prepare the next ingredients" is displayed when the elapsed time of a certain cooking step is 60 seconds before the scheduled time to switch to the next cooking step. Also, the supplemental information "Continue stirring" is displayed when the temperature of the ingredients detected during a certain cooking step exceeds a threshold temperature, and the supplemental information "Temporarily stop stirring" is displayed when the stirring operation is completed.

[0204] 22 and 23 are diagrams showing examples of countdown displays as other supplemental information.

[0205] 22 and 23, countdowns indicating the time remaining until the start and end of a cooking process, and countdowns indicating the time remaining until the start and end of cooking work, are displayed as supplemental information. Fig. 22 shows an example of a countdown display until the start time of step 1. In this way, the supplemental information may be displayed large, occupying the entire display area 211.

[0206] 23 shows an example of a countdown displayed overlaid on a cooking video. The countdown 261 displayed overlaid on the cooking video indicates the time remaining until the mixing step begins. By displaying such supplemental information, the user can more easily time the cooking step.

[0207] For cooking steps that are performed less frequently, supplemental information explaining that the user can temporarily leave the replica cooking site may be presented to the user. In this case, a warning to return to the replica cooking site before the next cooking task begins may also be presented as supplemental information. This supplemental information is also presented when preset conditions are met for the time, wavenumber (wavelength, frequency), temperature, amount of water evaporation, weight, stirring strength, stirring direction, stirring speed, estimated amount of chemical change (Maillard reaction, etc.), heat transfer simulation value, etc.

[0208] The warning to return to the reproduction cooking site may not be displayed all the time, but may be displayed only when the user leaves the reproduction cooking site. The user's departure from the reproduction cooking site is detected, for example, by the user operating a button or by the human presence sensor 35. The user's departure from the reproduction cooking site may be detected by a proximity sensor using radio waves instead of the human presence sensor 35.

[0209] In cooking processes where the cooking time can be changed by controlling the heat of the IH stove 11, the time until the next cooking process may be extended. The extension of the time until the next cooking process is performed, for example, by a user operation. This allows the user to have time to leave the cooking process. Such an extension can be performed only for cooking processes that have little impact on the degree of reproduction of the finished dish.

[0210] <Score Display Screen> Fig. 24 is a diagram showing an example of the score display screen. The score display screen is a screen used to present the reproducibility score. The reproducibility score is a score that indicates the evaluation result of the reproduced recipe.

[0211] The score display screen displays a numerical value indicating the overall reproducibility score in the upper center, and the reproducibility scores for each individual evaluation index are displayed below that. In the example of Figure 24, the overall reproducibility score is 92 points, and the reproducibility scores for each evaluation index (force, timing, and mixing) are displayed using bar graphs. The bar graphs show the ratio of the reproducibility score for each individual evaluation index to the perfect score.

[0212] The amount of force refers to the amount of force used when mixing ingredients. Timing refers to the timing of cooking tasks such as when to add ingredients, when to open and close the lid, and when to mix the ingredients. Mixing method refers to the speed, stroke (mixing area), and direction of mixing the ingredients. Reproducibility scores for other evaluation indices related to straining, squeezing, switching of cooking appliances, etc. may also be displayed. The reproducibility score for mixing, which is one type of cooking task, may be presented using scores for multiple evaluation indices such as speed, stroke, and direction.

[0213] A pass / fail determination may be made based on the reproducibility score, such that if the reproducibility score is higher than a threshold value, it is deemed to be passed, and if it is lower than the threshold value, it is deemed to be failed, and the result may be displayed.

[0214] The reproducibility score described above is calculated by the control unit 135, for example, when all cooking steps of the replicated recipe are completed. The calculation of the reproducibility score is performed so that the closer the cooking operations of the user during the replicated recipe are to the cooking operations of the chef during the recorded recipe, the higher the reproducibility score.

[0215] Specifically, the reproducibility score for the timing of adding ingredients is calculated by comparing the start time of the ingredient adding operation with a threshold range, such as 10 points if the difference between the start time of the ingredient adding operation and the time the user added the ingredients is within ±5 seconds, 9 points if it is +5 to 10 seconds, 8 points if it is +10 to 15 seconds, etc.

[0216] The reproducibility scores for each individual evaluation index may be displayed, or the sum of the reproducibility scores may be displayed. The total score may be determined in advance and normalized according to the number of cooking steps.

[0217] For example, a reproducibility score for each individual evaluation index is calculated for each cooking step, and an overall reproducibility score is calculated based on the reproducibility scores for the multiple individual evaluation indexes. A difficulty level may be set for each evaluation index or each cooking step, and the overall reproducibility score may be calculated using a weight according to the difficulty level. A reproducibility score for each cooking step may be calculated and presented.

[0218] From the display shown in FIG. 24, the user can check to what extent he or she has been able to reproduce the chef's cooking operations. The user can also check how to further improve the degree of reproduction. In the example of FIG. 24, advice on areas for improvement is displayed below the reproduction score. If there are many cooking steps for which the reproduction score for the timing of adding ingredients is 8 points, feedback such as "If you add the ingredients earlier, the taste will be more uniform" is provided as shown in FIG. 24.

[0219] When a user with a low reproducibility score performs a reproducible cooking using the cooking assistance system 1, the user may be prompted to confirm ingredient information when selecting a dish on the menu selection screen, or the previous feedback may be presented again. On the cooking navigation screen, supplementary information may be highlighted by enlarging the text or changing the color to draw attention. Based on the reproducibility score, the procedure for the reproducible cooking from the next time onwards will change.

[0220] <Cooking Navigation Processing> Fig. 25 is a flowchart illustrating the cooking navigation processing. The processing shown in Fig. 25 starts, for example, when the selection of the dish to be reproduced and the introduction of an overview of all cooking steps are completed and the navigation data set is read.

[0221] In step S101, the information presenter 136 displays a cooking navigation screen.

[0222] In step S102, the sensor data acquisition unit 133 acquires sensor data after the start of the reproduction cooking.

[0223] In step S103, the control unit 135 determines whether the value of the sensor data indicating the state of the ingredients and the like has reached the value set as the target state of the current cooking process.

[0224] If it is determined in step S103 that the sensor data value is not the value set as the target state of the current cooking process, the control unit 135 determines in step S104 whether it is time to display supplemental information. The determination of whether it is time to display supplemental information is also made by comparing the sensor data value indicating the state of ingredients, etc. with the value of a preset condition.

[0225] If it is determined in step S104 that it is time to display the supplemental information, the information presenter 136 displays the supplemental information on the cooking navigation screen in step S105. After the supplemental information is displayed in step S105, or if it is determined in step S104 that it is not time to display the supplemental information, the process returns to step S101, and the above process is repeated.

[0226] On the other hand, if it is determined in step S103 that the value of the sensor data has reached the value set as the target state of the current cooking process, then in step S106, the control unit 135 determines whether all cooking processes have been completed.

[0227] If it is determined in step S106 that all cooking steps have not been completed, the information presenter 136 switches the cooking step information to be fixedly displayed in step S107. That is, the information that had been shifted and displayed as information about the next cooking step is fixedly displayed as information about the new current cooking step. After the fixedly displayed information has been switched, the process returns to step S101, and the above processing is repeated.

[0228] On the other hand, if it is determined in step S106 that all cooking steps have been completed, the control unit 135 calculates a reproducibility score in step S108.

[0229] In step S109, the information presenting unit 136 displays a score display screen including the reproducibility score calculated by the control unit 135, and then ends the process.

[0230] <<Modifications>> <Other Display Examples of the Cooking Navigation Screen> Figures 26 and 27 are diagrams showing other display examples of the cooking navigation screen. In Figures 26 and 27, the same symbols are used for displays that are the same as those described with reference to Figure 13, etc. Duplicate descriptions will be omitted as appropriate.

[0231] The cooking navigation screens shown in Figures 26 and 27 differ from the screens described with reference to Figure 13 etc. in that information moves up and down. That is, the cooking navigation screens shown in Figures 26 and 27 are composed of a time-series data area 221 and cooking process areas 222-1 and 222-2, which are three strip-shaped areas of approximately the same width. With the time-series data area 221 at the center, the cooking process area 222-1 is located to the right of the time-series data area 221, and the cooking process area 222-2 is located to the left of the time-series data area 221.

[0232] The vertical direction of the cooking navigation screen indicates the time direction. The position indicated by the thick line image L1 indicates the current time. The grayed-out portion above image L1 indicates that the information in that portion is past information. The information below image L1 indicates information from a later time.

[0233] Information about the cooking work of the current cooking process, step 4, is fixed and displayed in the cooking process area 222-2, while information about the cooking work of the next cooking process, step 5, is displayed by moving upward in the cooking process area 222-1, as indicated by the white arrow. Figure 26 shows the transition of the cooking navigation screen when the information about the next cooking process moves upward. Conversely, Figure 27 shows the transition of the cooking navigation screen when the information about the next cooking process moves downward.

[0234] In this way, it is also possible to form the time-series data area 221 and the cooking process areas 222-1 and 222-2 as vertical areas, and move the information on the next cooking process in the vertical direction.

[0235] 28 to 30 are diagrams showing other display examples of the cooking navigation screen.

[0236] 28 to 30, a circular area including a clock face-like scale is positioned in the center of the cooking navigation screen. A rectangular image 301 including information about cooking tasks for step 4, the current cooking process, is displayed fixed at the position corresponding to 12 o'clock. The position corresponding to 12 o'clock is the reference position. As shown by the arrowhead in FIG. 28, a straight line image L11 corresponding to the hands of a clock moves clockwise over time.

[0237] When a predetermined time has passed from the state shown in the lower part of Fig. 28, rectangular image 302 containing information about the cooking work of step 5, the next cooking process, is displayed only halfway, and then the entire image is displayed, as shown in the upper part of Fig. 29. After the entire image is displayed, image 302 is displayed by moving clockwise so as to approach image 301 over time, as shown in the lower part of Fig. 29 and Fig. 30.

[0238] When image 302 reaches the position of image 301, the current cooking process switches to the cooking process of step 5, and image 302, which had been moving up until then, is fixed and displayed at the reference position where image 301 was displayed. In this way, it is also possible to make the movement direction of the information on the next cooking process clockwise.

[0239] <Example of System Configuration> FIG. 31 is a diagram showing an example of the configuration of a cooking assistance system.

[0240] Although the processes of recording cooking, creating a navigation data set, and reproducing cooking are each performed in the same cooking assistance system 1 having a recording processing unit 101, a navigation data set creation unit 102, and a reproduction processing unit 103 as shown in A of Figure 31, each process may also be performed in a different information processing device.

[0241] 31B, the recording and reproducing cooking may be performed in the cooking assistance system 1, and the navigation dataset creation may be performed in the web server 42. In this case, the recording processing unit 101 and the reproducing processing unit 103 are realized in the cooking assistance system 1, and the navigation dataset creation unit 102 is realized in the web server 42.

[0242] The recording data generated by the recording processing unit 101 of the cooking assistance system 1 is transmitted to the web server 42 via a network such as the Internet, and is used by the navigation dataset creation unit 102 to create a navigation dataset. The navigation dataset created by the navigation dataset creation unit 102 of the web server 42 is transmitted to the cooking assistance system 1 via the network, and is used by the reproduction processing unit 103 to reproduce the recipe.

[0243] 32, the recording cooking and the reproducing cooking may be performed in different cooking assistance systems 1. In the example shown in Fig. 32, the recording processing unit 101 is realized in the cooking assistance system 1-1, and the reproducing processing unit 103 is realized in the cooking assistance system 1-2. The cooking assistance systems 1-1 and 1-2 each have the same configuration as the cooking assistance system 1 described with reference to Fig. 3 etc.

[0244] The recording data generated by the recording processing unit 101 of the cooking assistance system 1-1 is transmitted to the Web server 42 via a network such as the Internet, and is used in the creation of a navigation data set by the navigation data set creation unit 102. The navigation data set created by the navigation data set creation unit 102 of the Web server 42 is transmitted to the cooking assistance system 1-2 via the network, and is used in the reproduction processing unit 103 for the reproduction of the cooking.

[0245] By providing a plurality of cooking assistance systems 1-2 that perform cooking reproduction, it becomes possible to reproduce a dish based on one navigation data set in various locations.

[0246] In this way, it is possible to arbitrarily change which device is to perform at least one of the processes of recording cooking, creating a data set for navigation, and reproducing cooking.

[0247] - Computer Configuration Example The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed from a program recording medium into a computer built into dedicated hardware or a general-purpose personal computer.

[0248] Fig. 33 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. A computer having the configuration shown in Fig. 33 is provided in the induction cooker 11. External devices such as a web server 42 are also configured using a computer having the configuration shown in Fig. 33.

[0249] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0250] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0251] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0252] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0253] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0254] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0255] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0256] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0257] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0258] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0259] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0260] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0261] (1) An information processing device including a presentation control unit that controls the display of a cooking navigation screen on which information about a current cooking process is fixedly displayed and information about a next cooking process is displayed in a moving manner over time. (2) The information processing device described in (1), wherein the presentation control unit moves the information about the next cooking process closer to the information about the current cooking process. (3) The information processing device described in (1) or (2), wherein the presentation control unit moves the information about the next cooking process at a speed corresponding to the time required for the current cooking process. (4) The information processing device described in any of (1) to (3), wherein the presentation control unit displays the information about the next cooking process as fixed information about the new current cooking process when the current cooking process is finished. (5) The information processing device described in (4), wherein the presentation control unit switches the current cooking process when sensor data measured during the current cooking process meets a preset condition. (6) The information processing device described in any of (1) to (5), wherein the presentation control unit presents supplemental information about the current cooking process when sensor data measured during the current cooking process meets a preset condition. (7) The information processing device according to any one of (1) to (6), wherein the presentation control unit displays information indicating first time series data of sensor data measured during cooking of an object to be reproduced and information indicating second time series data of sensor data measured in the current cooking step. (8) The information processing device according to (7), wherein the presentation control unit updates the display of the information indicating the second time series data based on a predetermined position indicating the current time. (9) The information processing device according to (8), wherein the first time series data is data indicating a time series of sensor data measured in a cooking step corresponding to the current cooking step during cooking of the object to be reproduced, and the presentation control unit moves a display range of the information indicating the first time series data so that the measurement result of the sensor data at the time corresponding to the current time is displayed at the predetermined position.(10) The information processing device described in (9), wherein the presentation control unit updates the display of the information indicating the second time-series data so that the latest measurement result of the sensor data for the current cooking process is displayed in the predetermined position. (11) The information processing device described in any of (7) to (10), wherein the presentation control unit displays the cooking navigation screen in which the display areas for the information indicating the first time-series data and the information indicating the second time-series data are arranged between the display areas for information related to the current cooking process and information related to the next cooking process. (12) The information processing device described in (11), wherein the presentation control unit switches the display areas for the information related to the current cooking process and the information related to the next cooking process each time the current cooking process changes. (13) The information processing device described in any of (1) to (12), wherein the presentation control unit displays information indicating the content of tasks for each cooking process as information related to the current cooking process and the next cooking process. (14) An information processing method, wherein an information processing device controls the display of a cooking navigation screen in which information related to the current cooking process is fixed and information related to the next cooking process is displayed in a moving manner over time. (15) A program that causes a computer to execute a process to control the display of a cooking navigation screen in which information about the current cooking process is displayed fixedly and information about the next cooking process is displayed moving over time.

[0262] 1 Cooking assistance system, 11 IH stove, 21 Display, 22 Speaker, 23 Camera, 24 Microphone, 25 Lighting equipment, 26 Cooking environment measuring device, 27 Router, 31 Processor, 32 Temperature sensor, 33 Force sensor, 34 Storage, 41 Smartphone, 42 Web server, 101 Recording processing unit, 102 Navigation dataset creation unit, 103 Reproduction processing unit, 111 Cooking environment measurement unit, 112 Sensor data acquisition unit, 113 User input detection unit, 114 Information recording unit, 121 Process organization unit, 122 Target state setting unit, 123 Cooking task setting unit, 124 Information recording unit, 125 Navigation dataset management unit, 131 Navigation dataset acquisition unit, 132 Cooking environment measurement unit, 133 Sensor data acquisition unit, 134 User input detection unit, 135 Control unit, 136 Information presentation unit

Claims

1. An information processing device having a presentation control unit that controls the display of a cooking navigation screen on which information about the current cooking step is displayed fixedly and information about the next cooking step is displayed moving over time.

2. The information processing device according to claim 1, wherein the presentation control unit moves the information relating to the next cooking step closer to the information relating to the current cooking step.

3. The information processing device according to claim 1, wherein the presentation control unit moves the information about the next cooking step at a speed corresponding to the time required for the current cooking step.

4. The information processing device according to claim 1, wherein the presentation control unit, when the current cooking process is completed, fixes and displays information about the next cooking process as new information about the current cooking process.

5. The information processing device according to claim 4, wherein the presentation control unit switches the current cooking process when sensor data measured in the current cooking process satisfies a preset condition.

6. The information processing device according to claim 1, wherein the presentation control unit presents supplemental information regarding the current cooking process when sensor data measured during the current cooking process meets a preset condition.

7. The information processing device according to claim 1, wherein the presentation control unit displays information indicating first time series data of sensor data measured during cooking of the object to be reproduced and information indicating second time series data of sensor data measured during the current cooking process.

8. The information processing device according to claim 7, wherein the presentation control unit updates the display of the information indicating the second time-series data based on a predetermined position indicating the current time.

9. The information processing device according to claim 8, wherein the first time series data is data indicating a time series of sensor data measured in a cooking step corresponding to the current cooking step during cooking of the object to be reproduced, and the presentation control unit moves the display range of information indicating the first time series data so that the measurement result of the sensor data at the time corresponding to the current time is displayed at the specified position.

10. The information processing device according to claim 9, wherein the presentation control unit updates the display of the information indicating the second time series data so that the latest sensor data measurement results for the current cooking process are displayed at the specified position.

11. The information processing device of claim 7, wherein the presentation control unit displays the cooking navigation screen in which the display areas for the information indicating the first time series data and the information indicating the second time series data are arranged between the display areas for the information regarding the current cooking process and the information regarding the next cooking process.

12. The information processing device according to claim 11, wherein the presentation control unit switches the display area for the information on the current cooking process and the information on the next cooking process every time the current cooking process changes.

13. The information processing device according to claim 1, wherein the presentation control unit displays information indicating the content of work in each cooking step as information relating to the current cooking step and the next cooking step.

14. An information processing method in which an information processing device controls the display of a cooking navigation screen in which information about the current cooking process is displayed fixedly and information about the next cooking process is displayed moving over time.

15. A program that causes a computer to execute a process to control the display of a cooking navigation screen in which information about the current cooking process is displayed fixedly and information about the next cooking process is displayed moving over time.

Images