Method and control unit for operating a cooking appliance and cooking appliance

Abstract

The invention relates to a method (300) for operating a cooking appliance (100) comprising a cooking chamber (105), a cooking chamber door (110) closing the cooking chamber (105), and a sensor device (115). The method (300) includes a step of reading (305) a plurality of image data (425) via an interface to the sensor device (115) in response to an actuation signal (430) representing a door state of the cooking chamber door (110), wherein the plurality of image data (425) represents a plurality of images of the cooking chamber (105). Furthermore, the method (300) includes a step of evaluating (310) the plurality of image data (425) using the door state to detect movement or non-movement (435) within the cooking chamber (105).The method (300) also includes a step of performing (320) food detection responding to non-movement and a step of providing (325) a program signal (445) to an interface to a display device (120) of the cooking appliance (100) in order to suggest a cooking program to a user that is assigned to the food being cooked.

Description

[0001] The invention relates to a method and a control unit for operating a cooking appliance and to a cooking appliance.

[0002] In the course of digitalization and the goal of offering people the greatest possible comfort and making everyday life easier, more and more areas and, for example, everyday devices are being electronically networked, which is related to the so-called smart home.

[0003] EP3067828B1 describes a method for detecting the direction of movement of a predetermined object, in particular a cooking accessory, in the area of ​​a cooking chamber opening of a cooking appliance.

[0004] The approach presented here aims to create an improved method and control unit for operating a cooking appliance, and an improved cooking appliance itself.

[0005] According to the invention, this problem is solved by a method and control unit for operating a cooking appliance, as well as by a cooking appliance with the features of the main claims. Advantageous embodiments and further developments of the invention are described in the following dependent claims.

[0006] The presented approach offers a way to support users in a time-efficient manner during a cooking process, for example. This means that waiting times for the user can be advantageously reduced, allowing them to attend to other tasks without any loss of time.

[0007] A method for operating a cooking appliance is presented, comprising a cooking chamber, a cooking chamber door that closes the cooking chamber, and a sensor device. The method includes a step of reading a plurality of image data via an interface to the sensor device in response to an actuation signal representing the door state of the cooking chamber door. The plurality of image data represent a plurality of images of the cooking chamber. Furthermore, the method includes a step of evaluating the plurality of image data using the door state to detect movement or lack thereof within the cooking chamber.The procedure also includes a step of performing food detection responding to the absence of movement within the cooking chamber and a step of providing a program signal to an interface to a display device of the cooking appliance in order to suggest a cooking program to the user that is assigned to the food being cooked.

[0008] The cooking appliance can be, for example, a built-in or freestanding unit, such as an oven. The cooking chamber can be heat-resistant, allowing food items or entire meals to be prepared in the appliance. The cooking chamber door can advantageously be a hinged door, which can be pivotally mounted on the appliance body, for example, via hinges. The sensor device can have one or more sensors, but at least one image sensor capable of capturing the majority of image data, advantageously cyclically. The sensor device can therefore be located within the cooking chamber. The door status can advantageously represent either the opening or the open state of the appliance door. Consequently, the activation signal can, for example, be triggered initially by opening the door and then, upon repetition of the reading step, be read cyclically after a set interval.In other words, the appliance door is opened only once, and then images can be continuously captured cyclically. Only the first image is triggered by the door opening, while subsequent images are captured at a set frame rate. During the evaluation phase, an image comparison can be advantageously performed to identify differences between the images, allowing conclusions to be drawn about movement or the absence of movement. Movement could be, for example, the insertion of a baking tray or wire rack into the oven, or alternatively, the insertion of food into the oven. In the case of no movement, for example, no movement was detected in the oven. No movement could also represent a detected standstill. Advantageously, movement or no movement can be detected while the oven door is open.Advantageously, motion direction detection can identify whether there has been no movement after an object, such as a cooking tray or food, has been removed or after an object has been inserted into the cooking chamber. For food detection, the shape of a container can be analyzed to determine the type of container in which the food might be placed. For example, a baking pan can be distinguished from a casserole dish, allowing the user to be offered a cooking program during the placement step that specifies at least one cooking parameter for the corresponding dish. Optionally, the food detection process can also choose not to detect any food at all. Food detection can be advantageously implemented as a trained machine model, such as a neural network. Furthermore, food detection can take place outside the cooking appliance on a server or in the cloud.The display device can advantageously be designed as a device display, for example a touchscreen. Additionally, the suggested cooking program can also be displayed on a mobile device.

[0009] According to one embodiment, in the evaluation step, the majority of image data can be evaluated to detect movement or lack thereof. Movement can be detected if the image and at least one other image of the cooking chamber are different. Lack of movement can be detected if the image and at least one other image are identical. Preferably, the majority of captured images can be compared with one another to detect movement. If, for example, no movement is detected, it is advantageous to check whether there is any food being cooked in the cooking chamber or whether the cooking chamber is empty. Advantageously, this allows images to be captured and processed from the moment the cooking chamber door is opened.

[0010] According to one embodiment, the movement can also be assigned a direction of movement into or out of the cooking chamber, and the lack of movement can be assigned a state of being pulled out or pushed in. Such an embodiment offers the advantage of a clear and easily recognizable movement, which can then be used for further evaluation.

[0011] The process can also include a determination step in which the cooking appliance's state can be detected and additionally or alternatively verified before the food being cooked is detected. The cooking appliance's state can represent the presence of food in the cooking chamber.

[0012] The provisioning step can be triggered by receiving a door operation signal, where the door operation signal can represent the operation of the oven door, in particular the closing of the oven door. The cooking program can therefore advantageously be displayed on the display device as soon as the oven door is closed, so that a user does not have to wait to see the program suggestion on the display device. The display can thus be advantageously time-efficient.

[0013] According to one embodiment, in the evaluation step, the majority of image data can be evaluated to detect movement or the absence of movement. In particular, if the evaluation result is ambiguous, the determination step can be performed. The ambiguous evaluation result could, for example, be due to a dirty camera lens or partially obscured movements. To avoid a continuous loop of the same step, the determination step can clarify whether movement has occurred or not in the case of an ambiguous evaluation result. In other words, the presence of the food being cooked can provide a clue as to whether movement has occurred.

[0014] Furthermore, during the evaluation step, the majority of image data can be analyzed using vibration analysis, a measured value from a force sensor, or a monitoring signal from a monitoring sensor to monitor a cooking tray. The cooking tray can be, for example, a baking sheet or a baking rack on which the food can be placed. Advantageously, data from other sensors of the cooking appliance can be used during the evaluation step.

[0015] According to one embodiment, in the evaluation step, the majority of image data can be evaluated to detect movement or lack thereof, whereby a start time and / or an end time of the movement can be recorded if the movement has been detected. This advantageously allows the identification of which object was inserted into the cooking chamber or at what position this object was placed, so that a conclusion can be drawn regarding a recommended cooking program.

[0016] Furthermore, in the evaluation step, the majority of image data can be analyzed to detect movement or non-movement based on edge images and additionally or alternatively via a similarity measure of the images. Advantageously, this can be done by subtraction or correlation. According to one embodiment, the image data can also be analyzed in the evaluation step to determine the direction of movement. This can advantageously be determined using the optical flow between images. This information is particularly advantageous for distinguishing between the removal and loading of the cooking chamber. Preferably, food identification can only be performed after loading. In this case, a distinction can be made between movement with removal, movement with insertion, non-movement with removal, and movement with insertion.If necessary, a fifth state, non-movement, in which no prior movement is known, can be used as a state.

[0017] The steps of the procedure can be repeated at least partially, in particular the reading step can be performed at a repetition interval. Advantageously, the evaluation, execution, and provisioning steps can also be repeated to perform a continuous cyclic determination while the oven door is open. A repetition interval between 100 and 2000 ms is advantageous. 200 to 500 ms is particularly preferred.

[0018] The approach presented here further creates a control unit designed to execute, control, and implement the steps of a variant of the method presented here in appropriate devices. This embodiment of the invention, in the form of a device, also allows the underlying problem to be solved quickly and efficiently.

[0019] The control unit can be configured to read input signals and use these input signals to determine and provide output signals. An input signal can, for example, be a sensor signal readable via an input interface of the control unit. An output signal can be a control signal or a data signal that can be provided at an output interface of the control unit. The control unit can be configured to determine the output signals using a processing instruction implemented in hardware or software. For example, the control unit can include a logic circuit, an integrated circuit, or a software module and may be implemented as a discrete component or comprised of a discrete component.

[0020] A computer program product or computer program with program code that can be stored on a machine-readable medium such as semiconductor memory, hard disk storage, or optical storage is also advantageous. If the program product or program is executed on a computer, a control unit, or, for example, in a cloud, it can be used to carry out, implement, and / or control the steps of the method according to one of the embodiments described here.

[0021] Furthermore, a cooking appliance is presented which has a cooking chamber for holding food to be cooked, a cooking chamber door closing the cooking chamber, a sensor device, a display device and a control unit in a previously mentioned variant.

[0022] The cooking appliance can be, for example, an oven that is either built into a kitchen or a freestanding unit. Although the described approach is based on a household appliance, it can also be applied to commercial or professional equipment.

[0023] According to one embodiment, the sensor device can comprise at least one image sensor and additionally or alternatively a further sensor, in particular wherein the further sensor can be configured as a vibration sensor, a force sensor, and / or a scale. Advantageously, the sensor device can utilize sensors already installed in the cooking appliance to carry out the method in one of the aforementioned variants. The sensor can advantageously be configured as an optical sensor, such as a camera. The at least one further sensor can advantageously be a sensor already arranged in the cooking appliance.

[0024] An embodiment of the invention is shown purely schematically in the drawings and is described in more detail below. It shows Fig. 1 a schematic representation of an exemplary embodiment of a cooking appliance; Fig. 2 a schematic sectional view of an exemplary embodiment of a cooking appliance; Fig. 3 a flowchart of an exemplary embodiment of a method for xxx for a cooking appliance; and Fig. 4 a block diagram of an exemplary embodiment of a control unit.

[0025] Fig. Figure 1 shows a schematic representation of an embodiment of a cooking appliance 100. According to this embodiment, the cooking appliance 100 is implemented as an oven comprising a cooking chamber 105 for holding food, a cooking chamber door 110 closing the cooking chamber 105, a sensor device 115, a display device 120, and a control unit 125. The cooking chamber door 110 is pivotable in this embodiment. The schematically depicted sensor device 115 includes, for example, at least one image sensor and / or another sensor, the latter being, in particular, a vibration sensor, a force sensor, and / or a scale. The sensor device 115, as well as the display device 120, is electrically coupled to the control unit 125.The control unit 125 is configured to control and / or execute a method for operating the cooking appliance 100, as described, for example, in at least one of the following figures. The control unit 125 is configured to read signals, evaluate them, and output signals. The display device 120 and / or the control unit 125 can be integrated, for example, into a control panel 130 of the cooking appliance 100.

[0026] The display unit 120 is implemented, for example, as a touch-sensitive display, via which a user can, for instance, set the cooking appliance 100 or a cooking program of the cooking appliance 100. Furthermore, the display unit 120 is designed to show the user information about the cooking appliance 100 or a cooking process and / or to suggest possible cooking programs to the user.

[0027] According to this embodiment, the cooking appliance 100 has a plurality of rail elements 135 arranged in the cooking chamber 105, which are shaped to receive at least one food carrier, for example by sliding it in. The food carrier is, for example, a baking tray or a baking rack.

[0028] The described approach, for example, involves the first identification of a period of non-movement in the cooking appliance 100. This is preferably achieved using images captured in the cooking chamber 105 and / or, for example, measurement data from other sensors, such as a scale. Furthermore, depending on the period of non-movement, images are either captured directly or, for example, the most recent image is retrieved from memory, which is then the first image taken when no movement occurred. These images are then used for food identification, and a cooking suggestion is displayed, for example, on the display unit 120.

[0029] Fig. Figure 2 shows a schematic sectional view of an exemplary embodiment of a cooking appliance 100, as is found, for example, in Fig. 1 was described. According to this embodiment, the sensor device 115 comprises the image sensor 200 and at least one further sensor 205, wherein the further sensor 205 is configured as a vibration sensor, a force sensor and / or a scale. A position of the further sensor 205 is described in Fig. 2 is chosen only as an example. According to this embodiment, the image sensor 200 is designed as a camera whose viewing direction is directed into the cooking chamber 105.

[0030] Fig. Figure 3 shows a flowchart of an embodiment of a method 300 for operating a cooking appliance, as is found, for example, in at least one of the Fig. The procedure 300 comprises a reading step 305, an evaluation step 310, an optional determination step 315, an execution step 320, and a provision step 325.

[0031] In step 305 of the acquisition process, image data is read into the sensor device via an interface in response to an activation signal representing the door status of the oven door, such as door operation or an open oven door. The image data represents at least one image of the oven. In step 310 of the evaluation process, the image data is evaluated, for example, by comparing the images, to detect movement or lack of movement within the oven. This takes place while the oven door is open. In the optional step 315 of the determination process, an oven state is determined based on the absence of movement within the oven. This oven state represents the presence of food in the oven. Specifically, it indicates whether the oven is empty or contains food.If, for example, food is present in the cooking chamber, step 320 of the execution process follows. In step 320, food detection is performed, for example, using the cooking appliance's state, if no movement is detected and / or if food is present in the cooking chamber. Reference data is used, for example, for food detection. In step 325, the provisioning process, a program signal is provided to an interface for a display unit of the cooking appliance to suggest a cooking program to the user that is associated with the food. The user is then given the option to accept the suggested cooking program and thereby start it, or to reject the suggested cooking program. According to this embodiment, movement is detected in step 310 of the evaluation process if the image and at least one other image of the cooking chamber are different.Non-movement is detected when the image and at least one other image are identical. For example, if the evaluation result is ambiguous, step 315 of the determination process is performed. Optionally, in step 310 of the evaluation, the image data is analyzed using vibration analysis, a measured value from a force sensor, or a monitoring signal from a sensor for monitoring a cooking tray. Optionally, a start time and / or an end time of the movement is recorded if the movement was detected. According to this embodiment, the detection of movement or non-movement is based on edge images and / or a similarity measure of the images, such as by subtraction or correlation.

[0032] According to this embodiment, step 325 of providing a door actuation signal is triggered by step 330 of receiving a door actuation signal, wherein the door actuation signal represents an actuation of the oven door, in particular a closing of the oven door. Step 330 of receiving is triggered, for example, by the closing of the oven door.

[0033] Steps 315 and 320 of the detection and execution phases can also be performed together. Accordingly, the load is detected immediately upon detection of no movement.

[0034] Furthermore, steps 305, 310, 315, 320, 325, and 330 of procedure 300 are repeated at least partially, in particular where at least step 305, the reading step, is performed at a repetition interval of, for example, 2 seconds. This would mean, for example, that an image is captured every 2 seconds. For example, the other steps 310, 315, 320, 325, and 330 of procedure 300 can be performed continuously in a cyclical manner.

[0035] In other words, the approach presented here describes a food detection system using a camera with the oven door open, which is also referred to simply as the door. With this sensor unit, which can also be called a camera, a cooking suggestion, determined based on image data, is displayed on the appliance's display (i.e., the display unit) immediately upon closing the door. This is achieved through the... Fig. 3 described methods 300 make this possible directly and therefore faster after a door has closed.

[0036] Even while the oven door is still open, images from inside the oven are captured and analyzed. This allows the food recognition results to be available, for example, 2 to 3 seconds before the door closes, and the user receives a suggestion immediately or as soon as the door closes.

[0037] Additionally or alternatively, continuous cyclical identification of the food being cooked is enabled while the oven door is open. This cycle lasts, for example, 2 seconds. For every minute with the oven door open, the device sends only 30 requests for object identification to, for example, a cloud service or an image recognition algorithm. To minimize the number of requests, a preliminary image analysis is advisable. This is performed by the control unit, as found in at least one of the Fig. 1 to 2 were described or at least mentioned. The preliminary analysis includes, for example, motion detection in step 310 of the evaluation.

[0038] In step 310 of the evaluation process, a comparison of images (taken, for example, every second, half a second, or a quarter of a second) or of image features is used to determine whether the food being cooked or the cooking container is in motion. If so, it can be assumed that the food is only partially visible, for example, through a cropped section of the image, or that the image exhibits artifacts such as blurring due to the movement. Only when the image is stable, meaning no further changes over time are discernible, is the next step, 315, initiated.

[0039] Optionally, motion detection can be achieved using other sensors, enabling, for example, shrinkage analysis, force sensing, or monitoring of slide-in rails. Other 3D sensors, such as ToF cameras, can also be used.

[0040] Step 315 of the determination process can also be described as a preliminary empty analysis. To minimize the number of cloud requests, the static image is roughly evaluated beforehand with regard to the load. This involves filtering out images that show an empty cooking chamber. This evaluation is based, for example, on image features or the image itself. This step can optionally be omitted, which, however, can result in a higher number of cloud requests. The evaluation can also optionally be performed using other sensors, such as a scale in the appliance, the cooking tray, or 3D triangulation of one or more points. Alternatively, a dual motion analysis can be performed, in which, for example, the start of a movement is recorded after the door is opened, and optionally, the end of the movement is then determined.

[0041] Motion detection is based, for example, on edge images and / or a similarity measure of images, such as image subtraction or correlation. Empty food detection within the appliance is implemented, for instance, by ensuring that if the detection is inconclusive and the cooking chamber does not register as "empty," step 320, the food detection process, is executed. The time between the last detected food item and the door closing (i.e., step 330), the receiving step, is limited by a predefined time interval. Food detection may be repeated cyclically.

[0042] Fig. Figure 4 shows a block diagram of an exemplary embodiment of a control unit 125, as used, for example, in at least one of the Fig. 1 to 2. The control unit 125 is designed to control and / or execute a procedure for operating a cooking appliance, as described, for example, in Fig. 3 was described. The control unit 125 has a reading unit 400, an evaluation unit 405, a determination unit 410, an execution unit 415 and a provision unit 420.

[0043] The reading unit 400 is configured to read image data 425 via an interface to the sensor device 115 in response to an actuation signal 430, which represents a door status of the cooking chamber door. The image data 425 represents at least one image of the cooking chamber. The evaluation unit 405 is configured to evaluate the image data 425 in order to detect movement or non-movement 435 within the cooking chamber. The detection unit 410 is configured to determine a cooking appliance state 440 in response to non-movement 435 within the cooking chamber, wherein the cooking appliance state 440 represents the presence of food in the cooking chamber. The execution unit 415 is configured to perform food detection using the cooking appliance state 440 when food is present in the cooking chamber.The provision unit 420 is therefore designed to provide a program signal 445 to an interface with the display unit 120 of the cooking appliance in order to suggest a cooking program to the user that is appropriate for the food being cooked. This is done optionally in response to the receipt of a door operation signal 450, which represents the operation of the cooking chamber door, in particular the closing of the cooking chamber door. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 3067828B1

[0003]

Claims

[1] Method (300) for operating a cooking appliance (100) comprising a cooking chamber (105), a cooking chamber door (110) closing the cooking chamber (105) and a sensor device (115), wherein the method (300) comprises the following steps: - Reading (305) a plurality of image data (425) via an interface to the sensor device (115) responding to an actuation signal (430) representing a door state of the oven door (110), wherein the plurality of image data (425) represents a plurality of images of the oven (105); - Evaluating (310) the majority of image data (425) using the door state to detect movement or non-movement (435) within the cooking chamber (105); - Performing (320) food detection responding to non-movement; and - Providing (325) a program signal (445) to an interface to a display device (120) of the cooking appliance (100) in order to suggest a cooking program to a user that is assigned to the food being cooked. [2] Method (300) according to claim 1, wherein in step (310) of evaluation the image data (425) are evaluated to detect the movement or the non-movement (435), wherein the movement is detected when the image and at least one further image of the cooking chamber (105) are different, wherein the non-movement (435) is detected when the image and the at least one further image are the same. [3] Method (300) according to claim 2, wherein the movement is assigned a direction of movement into or out of the cooking chamber, and a non-movement is assigned a state of being pulled out or pushed in. [4] Method (300) according to one of the preceding claims, wherein in a step (315) of determining a cooking appliance state (440) is detected and / or checked before the food detection (320) is carried out, wherein the cooking appliance state (440) represents the presence of food in the cooking chamber (105). [5] Method (300) according to one of the preceding claims, wherein the step (325) of providing responds to a step (330) of receiving a door actuation signal (450), wherein the door actuation signal (450) represents an actuation of the oven door (110), in particular a closing of the oven door (110). [6] Method (300) according to one of the preceding claims, wherein in the evaluation step the plurality of image data is evaluated in order to detect the movement or the non-movement, in particular wherein, in the event of an ambiguous evaluation result, the determination step is carried out. [7] Method (300) according to one of the preceding claims, wherein in step (310) of evaluation the plurality of image data (425) is further evaluated using a vibration analysis, a measured value from a force sensor or a monitoring signal from a monitoring sensor for monitoring a cooking carrier. [8] Method (300) according to one of the preceding claims, wherein in the step (310) of evaluation the plurality of image data (425) is evaluated to detect the movement or the non-movement (435), wherein a start time and / or an end time of the movement is recorded when the movement has been detected. [9] Method (300) according to one of the preceding claims, wherein in the step (310) of evaluation the plurality of image data (425) is evaluated in order to detect the movement or non-movement (435) on the basis of edge images and / or via a similarity measure of the images. [10] Method (300) according to one of the preceding claims, wherein in step (310) of evaluation the plurality of image data (425) is evaluated with respect to the direction of the movement and this movement is calculated by an optical flow. [11] Method (300) according to one of the preceding claims, wherein the steps (305, 310, 315, 320, 325, 330) of the method (300) are repeated at least partially, in particular wherein at least the reading step (305) is performed in a predefined repetition time interval. [12] Control unit (125) configured to perform and / or control the steps (305, 310, 315, 320, 325, 330) of the method (300) according to any of the preceding claims in corresponding units (400, 405, 410, 415, 420). [13] Computer program product with program code for carrying out the method (300) according to any one of claims 1 to 11, when the computer program product is executed on a control unit (125) according to claim 12. [14] Cooking appliance (100) with the following features: - a cooking chamber (105) for holding food being cooked; - a cooking chamber door (110) that closes the cooking chamber (105); - a sensor device (115); - a display device (120); and - a control unit (125) according to claim 12. [15] Cooking appliance (100) according to claim 14, wherein the sensor device (115) has at least one image sensor (200) and / or a further sensor (205), in particular wherein the further sensor (205) is designed as a vibration sensor, as a force sensor and / or as a scale.

Images