Managing a cooking process

Abstract

A method (200) comprises steps of optically scanning (205) a work area of ​​a kitchen; capturing (215) a cooking activity of a user (105) in the work area; storing (220) a reference to the cooking activity and a time of the cooking activity; and providing (225) a reference to a past cooking activity.

Description

[0001] The present invention relates to the control of a cooking process. In particular, the invention relates to assisting a user in carrying out cooking operations.

[0002] A cooking process for a dish comprises a sequence of cooking actions performed by a user. For example, ingredients may be chopped, processed, or combined, or a kitchen appliance may be operated in a predetermined manner, and an intermediate product may be further processed using the kitchen appliance.

[0003] Especially for a user who doesn't cook regularly, it can be difficult to execute the correct sequence of cooking steps. For example, it may be unclear exactly when a cooking or baking process began, making it difficult to determine the end. If a cooking step is stopped too early, the food may not be cooked sufficiently; if it is stopped too late, the food may overcook, fall apart, or stick to the cookware.

[0004] The problem of coordinating cooking steps in a good sequence can arise particularly when the user is not following a predetermined recipe but rather working by intuition. A recipe can be successful if the user prepares it frequently enough; however, if too much time has passed since the last preparation, the end of a cooking step may no longer be accurately determined.

[0005] One of the problems underlying the present invention is therefore to provide an improved technique for assisting a user in carrying out a sequence of cooking operations. The invention solves this problem by means of the subject matter of the independent claims. Dependent claims describe preferred embodiments.

[0006] According to a first aspect of the present invention, a method comprises steps of optically scanning a work area of ​​a kitchen; capturing a cooking action of a user in the work area; storing a reference to the cooking action and a time of the cooking action; and providing a reference to a past cooking action.

[0007] This method allows cooking actions to be saved along with their associated times, making it possible to later determine which cooking step was performed at which time. This enables the user to schedule subsequent cooking steps at the appropriate time. This can be particularly helpful when the user is managing multiple cooking processes simultaneously, such as different components of a dish. By providing these cues, the user can better coordinate the completion of the individual components, ensuring that all components are finished in a timely manner.

[0008] In one embodiment, a provided indicator includes the time elapsed since the cooking action. For example, the user can immediately see how long a cooking step initiated by the cooking action has already lasted. For instance, the cooking action might involve adding pasta to boiling salted water, with the cooking step being the cooking of the pasta. The elapsed time can easily be compared with the required cooking time. The user can then immediately determine how much time remains until the end of the cooking step.

[0009] In another embodiment, a history of past cooking steps is provided. This history can be displayed, for example, as a chronologically sorted list, similar to a logbook or journal. The user can thus directly view the sequence of completed cooking steps. Optionally, the user can delete a less relevant step so that it is no longer displayed. This makes it easier to visualize the temporal relationship between relevant cooking steps. Furthermore, the user can see the actual time required for a cooking step or for preparing an entire dish. This information can be valuable for planning future cooking processes.

[0010] In one variation, the cooking process is based on a recipe. The method can include capturing the recipe, allowing the system to track the user's current position within the recipe. For example, the system can focus on capturing cooking actions that are expected at the user's current stage of the cooking process. This can improve the recognition rate of captured cooking actions. Furthermore, cooking actions that don't fit the recipe can be either not detected or discarded after detection.

[0011] In another embodiment, a follow-up action is defined that is to be carried out a predetermined time after the cooking action. Accordingly, a time-based indication of the follow-up action can be provided.

[0012] In particular, it can be shown when the next action should be performed relative to the current time. In one embodiment, the remaining time until the next action is displayed. Optionally, an alarm can also be triggered when the appropriate time for performing the next action has arrived. The next action and the predetermined time can be determined, for example, based on a recipe. This document will show further ways to determine this information.

[0013] A condition can be specified that must be met for a subsequent action to be performed after the cooking action, and a reference to this condition can be provided. The condition might, for example, refer to the degree of doneness of an intermediate product that is currently being boiled, fried, grilled, or baked. The condition can be presented, in particular, in conjunction with a predetermined time. This illustrates that many cooking steps cannot be well defined solely on the basis of time and that some deviation is practically unavoidable. For example, it might be stated that the cooking of pasta should be stopped after approximately 12 minutes, although a desired firmness (al dente) is usually achieved within a cooking time of approximately 9 to 11 minutes.

[0014] The subsequent action and / or the condition are preferably determined based on the cooking recipe. In another embodiment, an ingredient that is the subject of the cooking action can be determined, whereby the subsequent action can be determined based on the ingredient. The ingredient can be identified based on scanning the working area. The subsequent action can, for example, be determined based on generic knowledge of common cooking recipes. In this way, at least to a first approximation, it is possible to determine the cooking time to be expected for a roast, vegetables, or rice.

[0015] In a further development, the ingredient is contained within packaging, and instructions for subsequent processing are determined based on the packaging. The packaging can be identified, for example, by a brand name, a barcode, or a QR code. Processing instructions can then be obtained, for instance, from a recipe collection or manufacturer's information. The packaging can also include a description or image of the contents. In this case, generic information about the ingredient can be assigned. In yet another embodiment, processing instructions can be printed directly on the packaging. For example, it is common to provide an approximate cooking time for pasta. Such information can be optically detected, recognized, and evaluated accordingly.

[0016] A cooking operation can comprise a cooking step, the beginning of a cooking step, or the end of a cooking step. Common cooking steps include combining, chopping, blending, slicing, boiling, grilling, frying, steaming, or similar heat treatment. A cooking operation, as defined in the technique presented herein, can also include setting a kitchen appliance. In particular, a temperature, power, duration, or configuration of a kitchen appliance can be determined. For example, it can be reflected that a cooking time can be shortened by increasing the cooking temperature or lengthened by lowering it. A configuration or setting of the kitchen appliance can be determined based on sensing or direct communication with the appliance.For example, the kitchen appliance can provide information regarding a setting, a configuration, or another operating parameter to the outside via a predetermined interface.

[0017] It is particularly advantageous to determine the timing of a subsequent action based on the setting. This can be achieved by determining a deviation of the setting from a predetermined setting, and based on this, a deviation from a predetermined timing of the subsequent action can be determined. For example, the subsequent action can be brought forward if a current cooking step is shortened due to a setting on a kitchen appliance, and vice versa.

[0018] In a particularly preferred embodiment, a sequence of detected cooking actions can be learned. Based on a cooking action, an indication of a likely subsequent action can be determined, and an indication of the subsequent action can be provided.

[0019] This option is particularly useful when the user isn't following a defined recipe, for example, when cooking a familiar recipe or improvising from an existing one. A sequence could, for instance, include a frying or cooking time. If the user flips pancakes for the first time after approximately four and a half minutes, an automatic reminder to flip them at that time can be provided after pouring batter into the pan or placing the pan on the stove.

[0020] It is not necessary to learn complete recipes; rather, it can be sufficient to learn specific sequences that can be repeated between different recipes. Examples of such sequences include sorting or blanching vegetables, searing meat in a pan, or braising a roast. By learning these sequences, a user can develop a habit that isn't tied to a vast array of familiar recipes. This can help the user repeat a cooking process according to their own preferences. As a result, the user can cook more confidently and achieve better results by deviating from familiar procedures by adjusting individual parameters. This can also enhance and foster the user's ability to cook creatively.

[0021] In another embodiment, a deviation from a learned sequence of cooking actions is determined, and the guidance is provided taking this deviation into account. This allows for an attempt to compensate for one user-initiated deviation with another. For example, the user might decide to use a higher cooking temperature than usual, and a hint can be given that the cooking time will end sooner than usual. This provides a suggested correction that the user can either follow or ignore. As a result, the user can be guided more effectively through the cooking process without being perceived as an error if their actions deviate intentionally from a familiar sequence.Rather, the user can be encouraged to experiment according to their own ideas without having to risk that the result of their cooking efforts deviates significantly from a typical result.

[0022] According to a further aspect of the present invention, a device comprises an optical scanning device for scanning a work area of ​​a kitchen; a storage device and a processing device. The processing device is configured to detect a cooking action performed by a user in the work area, to store a reference to the cooking action and the time of the cooking action; and to provide a reference to a past cooking action.

[0023] The processing equipment may be configured to partially or completely execute a method described herein. For this purpose, the processing equipment may be electronic and may, for example, include a programmable microcomputer or microcontroller. The method may be in the form of a computer program product containing program code. The computer program product may also be stored on a computer-readable data carrier. Features or advantages of the method may be transferred to the equipment and vice versa.

[0024] The invention will now be described in more detail with reference to the accompanying figures, in which: Fig. 1. A system to assist in a cooking process; and Fig. 2. Present a flowchart of a process.

[0025] Fig. Figure 1 shows a system 100 for assisting a user 105 in carrying out a cooking process. The cooking process generally takes place in a household, particularly in a kitchen, and more preferably in a work area 102. The device 110 comprises a scanning device 115 for the work area 102, a processing device 120, a storage device 125, and preferably an output device 130. The scanning device 115 is configured to optically scan the work area 102 and can, for example, include a camera, a depth camera, a stereo camera, an infrared camera, a LiDAR sensor, or a similar device. Optionally, the scanning device 115 includes multiple sensors. The scanning device 115 is preferably arranged above the work area 102 to provide a good overview of the cooking process.For example, the scanning device 115 can be attached to a wall cabinet or a range hood; alternatively, it can also be attached to a wall or ceiling.

[0026] The storage device 125 is preferably located locally in the kitchen or at least in the household, but in a further embodiment it can also be implemented at another location and connected via an interface, in particular a wireless interface. For example, a central location, a server, or a service, such as in a cloud, can be used to store information. Such a service can be communicatively connected to the device 110, for example, via the internet.

[0027] The output device 130 is designed as an example and, in this case, comprises a tablet computer that can be positioned in the work area 102 using a holder. In any case, the output device 130 is designed to be used by the user 105 during the cooking process.

[0028] In one embodiment, the device 110 comprises an artificial neural network (ANN) 135, which can be used to learn a sequence of cooking actions in the work area 102 or to determine a probable subsequent action based on a cooking action and the learned content.

[0029] Optionally, an interface 140 is provided, via which a kitchen appliance 145 can be connected, which is used by the user 105 to carry out a cooking process. The interface 140 can be wired or wireless.

[0030] The kitchen appliance 145 can, in particular, comprise a stove, an oven, a mixer, a blender, a mixing or kneading machine, or a similar device. In the illustrated embodiment, the kitchen appliance 145 is, by way of example, configured as a stove.

[0031] A cooking process comprises a number of cooking steps, which are usually performed in a predetermined sequence. Occasionally, variations in the sequence or an interruption at a predetermined point are possible. A recipe can specify a sequence of cooking steps to prepare a predetermined dish. The recipe can come from any source, such as a locally stored collection of recipes or from an external source, such as an online recipe collection.

[0032] A cooking action can comprise a simple event, such as setting a heat setting on the stove 145. A cooking action can also initiate or terminate a cooking step. For example, a cooking step might involve heating an ingredient 150 in a cooking vessel 155 for a predetermined time in a predetermined manner. A first exemplary cooking action can initiate the cooking step, and a second cooking action can terminate it.

[0033] In the representation of Fig. 1 is shown as an example ingredient 150, pasta (rigatoni). The cooking vessel 155 comprises a pot in which salted water is boiling.

[0034] It is proposed that the device 110 detects a cooking action performed by the user 105 in the work area 102 using the scanning device 115 and recognizes it using image processing methods. A reference to the cooking action, along with the time of the cooking action, can be stored in the memory device 125. For example, it can be recognized that the user 105 adjusts the heat setting on the stove 145 or that the user 105 adds the rigatoni 150 to the pot 155. Optionally, a cooking event can also be interpreted as a cooking action. A cooking event occurs during the cooking process and can be caused, in particular, by the use of a kitchen appliance 145. For example, the boiling of the salted water in the pot 155 can be defined as a cooking event.

[0035] One or more past cooking operations can be made available to user 105 via output device 130. For example, a chronologically sorted list of completed cooking operations can be displayed graphically.

[0036] To further assist the user 105, a subsequent action can be determined based on a cooking action. This subsequent action is conditioned by the cooking action and should be performed by the user 105 at a predetermined time. In one embodiment, a subsequent action can be determined based on a cooking recipe that the user 105 follows when preparing a dish. In another embodiment, a subsequent action can be determined based on an ingredient 150. For example, an ingredient 150 can be identified, and its associated cooking time can be determined. This can be done, in particular, by drawing on a collection of such information. In the present example, the rigatoni 150 themselves or the packaging 160 in which they are contained can be identified. For this purpose, for example, a label 165, a representation 170, or a trade code 175 can be optically detected and evaluated.In another embodiment, a recommendation 180 for handling the ingredient 150, as indicated on the packaging 160, can be recognized and evaluated. In the example shown, a manufacturer of the rigatoni 150 recommends a cooking time of approximately 12 minutes. If the cooking action specified is to place the rigatoni 150 into the boiling salted water in the pot 155, then, 12 minutes later, the subsequent action specified can be to remove the rigatoni 150 from the salted water in the pot 155. A notification of a specific subsequent action can also be provided to the user 105 via the output device 130.

[0037] In a further preferred embodiment, a machine learning technique 135 is provided to learn a typical sequence of cooking actions performed by the user 105. Reinforcement learning can be used for this purpose. A sequence can be placed in a predetermined context, for example, a cooking recipe that is being followed, a parallel cooking process, or the person 105, if different persons 105 can be distinguished from one another.

[0038] If the device 110 detects a cooking action by the user 105, it can determine, based on previously observed or learned sequences of cooking actions, which further cooking action or subsequent action is likely to follow. A notification of such a cooking action can then be provided to the user 105 via the output device 130.

[0039] Fig.Figure 2 shows a flowchart of a method 200 for controlling a cooking process. It should be noted that not all embodiments of the present invention necessarily include all of the process steps mentioned. The sequence of the specified process steps may also be modified, as a person skilled in the art will readily recognize.

[0040] In step 205, a working area 102 can be optically scanned using the scanning device 115.

[0041] In step 210, further information about a cooking progress or process can be collected. For example, a recipe can be recorded, followed by a cooking process. Furthermore, an ingredient 150 located in the work area 102 can be detected. Optionally, the ingredient 150 is detected while the user 105 is processing it. Optionally, the status of another kitchen appliance 145 can also be determined via the interface 140. The status can also be determined optically based on the scanning of the scanning device 115.

[0042] In step 215, a cooking action performed by user 105 in workspace 102 can be recorded. The cooking action typically involves an action with an ingredient 150 or an intermediate product.

[0043] In step 220, information about the cooking operation and the time at which it occurred can be stored in the memory device 125. Optionally, a boundary condition of the cooking operation can also be stored, for example, the identity of the user 105 or a setting or parameter of a kitchen appliance 145. For example, a set temperature or a current temperature can be stored as a parameter for an oven.

[0044] Step 225 allows you to display a history of cooking actions. This history can include a predetermined number of past cooking actions or cooking actions within a predetermined, past time period.

[0045] In step 230, a follow-up action can be provided. The follow-up action can be determined based on a recognized cooking action. Optionally, information from a cooking recipe or an ingredient 150 can be applied. The follow-up action can have an assigned time at which it should be carried out.

[0046] In a preferred embodiment, the subsequent action, which may also include a cooking action, is determined based on previously observed sequences of cooking actions. Such a sequence can be determined based on a history of cooking actions. In one embodiment, only cooking actions of one user 105 are considered. In another embodiment, cooking actions observed by the device 110 can be considered. In yet another embodiment, cooking actions observed by other devices 110 in other households can also be considered. For this purpose, stored cooking actions in a central location can be accessed. The determined subsequent action and an associated condition, in particular a predetermined time, can be provided to the user 105. Reference sign 100 System 102 Work area 105 users 110 Device 115 Scanning device 120 processing unit 125 Storage device 130 Output device 135 artificial neural network (ANN) 140 interface 145 Kitchen appliance, e.g. stove 150 ingredients, e.g. pasta 155 Cooking vessel 160 Packaging 165 inscription 170 illustration 175 Trading Code 180 Recommendation 200 procedures 205 Scan work area 210 Recipe, record ingredient 215 Cooking activity Save 220 cooking action and time Show 225 history 230 Provide follow-up action

Claims

[1] Procedure (200), wherein the procedure (200) comprises the following steps: - optical scanning (205) of a kitchen work area; - Recording (215) a cooking action of a user (105) in the work area; - Saving (220) a reference to the cooking action and a time of the cooking action; and - Providing (225) a reference to a past cooking activity. [2] Method (200) according to claim 1, wherein the provided indication includes a time elapsed since the cooking operation. [3] Method (200) according to claim 1 or 2, wherein a history of past cooking operations is provided (225). [4] Method (200) according to one of the preceding claims, wherein a cooking recipe is captured (210); and the cooking operation is based on the cooking recipe. [5] Method (200) according to any of the preceding claims, wherein a subsequent action is determined (230) to be carried out a predetermined time after the cooking action; and a time-based indication of the subsequent action is provided (230). [6] Method (200) according to one of the preceding claims, wherein a condition is determined which is to be satisfied for the execution of a subsequent action following the cooking action; and a reference to the condition is provided (230). [7] Method (200) according to claim 5 or 6, referring back to claim 4, wherein the subsequent action is determined on the basis of the cooking recipe. [8] Method (200) according to claim 5 or 6, wherein an ingredient is determined which is the subject of the cooking operation; wherein the subsequent operation is determined on the basis of the ingredient. [9] Method (200) according to claim 8, wherein the ingredient is contained in a package; wherein an indication of the subsequent action is determined on the basis of the package. [10] Method (200) according to one of the preceding claims, wherein a cooking operation comprises setting a kitchen appliance (145). [11] Method (200) according to claim 10, wherein a time of a subsequent action is determined depending on the setting. [12] Method (200) according to one of the preceding claims, wherein a sequence of detected cooking operations is learned; wherein, on the basis of a cooking operation, an indication of a probable subsequent operation is determined; and an indication of the subsequent operation is provided. [13] Method (200) according to claim 12, wherein a deviation from a learned sequence of cooking operations is determined; and the instruction is provided taking into account the deviation. [14] Device (110), wherein the device (110) comprises the following elements: - an optical scanning device (115) for scanning a work area of ​​a kitchen; - a storage device (125); - a processing device (120) that is configured to record a cooking action performed by a user (105) in the work area, to store a reference to the cooking action and a time of the cooking action; and to provide a reference to a past cooking action.

Images