Operating a domestic cooking appliance with at least one cooking chamber camera

Abstract

The invention relates to a method (S1-S5) for operating a domestic appliance (1) with a treatment chamber (2) and at least one treatment chamber camera (6) with at least one variable camera setting, in which, by means of the at least one treatment chamber camera (6), a plurality of images are generated (S4) in a manner staggered over time (t0 - t7) and at least two of the images are used for at least two different purposes (Z1 - Z3), wherein at least one camera setting of the at least one treatment chamber camera (6) is varied depending on the purpose (S2, S3). The invention also relates to a domestic appliance (1), in particular a domestic cooking appliance, comprising a treatment chamber (2) and at least one treatment chamber camera (6) with at least one variable camera setting, wherein the domestic appliance (1) is configured for carrying out the method (S1-S5).

Description

[0001] Operating a household appliance with at least one treatment room camera

[0002] The invention relates to a method for operating a household appliance having a treatment room and at least one treatment room camera, in which a plurality of images are generated at different times using the at least one treatment room camera, and at least two of the images are used for at least two different image-based purposes. The invention also relates to a household appliance having a treatment room and at least one treatment room camera with at least one variable camera setting, wherein the household appliance is configured to carry out the method. The invention is particularly advantageously applicable to household cooking appliances, specifically ovens, in particular baking ovens.

[0003] Household appliances with a cooking chamber and at least one cooking chamber camera, e.g., ovens with cooking chamber cameras or refrigerators with cooling chamber cameras (so-called "Camera-in-Fridge" or CIF), are generally known. Especially with cooking appliances, it is known to detect the position of food through image-based analysis and then to determine its degree of browning based on the image section showing the food. It is also known to determine a type of food through image analysis.

[0004] It is the object of the present invention to at least partially overcome the disadvantages of the prior art and in particular to provide a structurally particularly simple possibility of using images of a treatment chamber, in particular a cooking chamber, of a household appliance for different purposes.

[0005] This object is achieved according to the features of the independent claims. Preferred embodiments can be found in particular in the dependent claims.

[0006] The object is achieved by a method for operating a household appliance with a treatment room and at least one treatment room camera with at least one variable camera setting, in which

[0007] - several images are generated at different times by means of the at least one treatment room camera and at least two of the images are used for at least two different purposes, whereby

[0008] - at least one camera setting of at least one treatment room camera is varied depending on the purpose.

[0009] This has the advantage that the same camera can be used to provide images for different purposes, each with optimized camera settings. This, in turn, enables cost reduction compared to an arrangement in which different cameras or camera hardware are provided for different purposes. It also advantageously prevents any loss of precision or quality when images are used for specific purposes. A further advantage is that storage space and computing power can be kept to a minimum if the image quality is only as high as required for the respective purpose. Furthermore, new image-based purposes can be added with a software update without affecting the performance of existing purposes.

[0010] This method exploits the fact that camera settings or image parameters such as luminance, color channels, pixel count, and / or image resolution have a significant impact on the image processing algorithms that use them. Furthermore, it takes advantage of the fact that the maximum sampling rates of cameras are generally higher than the image acquisition intervals required for specific purposes in household appliances. Consequently, camera settings can be multiplexed between multiple settings or sets of settings to capture images with different camera settings for different purposes.

[0011] The treatment room is provided for treating objects to be treated by the household appliance, such as food, dishes, laundry, etc., e.g., to cook, clean, cool, etc. The treatment room can have a loading opening, e.g., on the front, which can be closed by a door. The treatment room camera is designed and arranged to record images of the treatment room, in particular also of objects present in the treatment room. A household appliance can have one or more cameras. The treatment room camera is, in particular, a pixel-based camera, in particular a digital camera. In a further development, the digital camera is a color camera, i.e., records color images, e.g., RGB or HSV images. At least one camera setting of the camera can be varied, in particular by means of a control device of the household appliance. The recording of an image can be triggered, for example, by means of the control device.It is a further development that the camera has a data processing device for image post-processing, for example, for image compression, setting a color depth, etc. It is a further development that a data processing device for image post-processing is an independent unit or is integrated into another functional unit of the household appliance, such as its control device. In this case, the camera can also be viewed as a distributed system ("system camera") in which image acquisition and image post-processing to obtain the ultimately generated image are spatially distributed across several components of the household appliance. Camera settings can be understood to mean, in particular, settings for image acquisition as such (e.g., an exposure time) and / or settings on the data processing device for image post-processing (e.g., a compression ratio). The treatment room camera can be located in the treatment room, e.g.be arranged on a wall of the treatment room, e.g. on the ceiling or door side, or it can be arranged outside the treatment room and look into the treatment room through a viewing window, e.g. a door window or a porthole.

[0012] The use of an image for a specific purpose is understood in particular to mean that the image is used for a specific functionality of the cooking appliance that utilizes the image. The "purpose" can also be referred to as (cooking appliance) functionality, application, use case, intended use, or application purpose.

[0013] In one embodiment, the at least one camera setting comprises at least one camera setting from the group of image resolution, compression level, color depth, white balance, sensor gain, gamma correction, image file format (e.g., PNG, GIF and / or RGB, HSV, NC'F image, etc.), use of a flash or flashlight, lens setting if the treatment room camera has an adjustable lens, and / or exposure time. However, the at least one camera setting is not limited to this and may include further camera settings. At least one camera setting can be predefined for at least one purpose, in particular always be the same for this purpose. At least one camera setting can be predefined for at least one purpose for a current cooking process depending on at least one further variable, e.g.,from a type of treatment, a type of food and / or the presence of certain objects in the cooking chamber such as baking paper, etc.

[0014] The image resolution can, for example, correspond to a generic image resolution of a camera sensor element, e.g., a CMOS sensor, e.g., 1024 x 768 pixels. The image resolution can also correspond to a downscaled image resolution—for example, through image post-processing—e.g., 640 x 480 pixels. A low resolution advantageously reduces image processing time, data size, and data transfer volume.

[0015] The compression level can be zero (no compression) or greater than zero (compressed image). In the following, a compression level of zero is also referred to as no compression or without compression. Compression can be achieved using suitable, generally known methods, e.g. JPEG, MPEG, PNG, WebP lossless, GIF, etc., whereby the compression level or the number of pixels is typically adjustable. However, it is also possible to choose between no compression and compression to a specific image size or number of pixels. The compression can be lossless or lossy. By specifying a compression level (other than zero), the time required for image processing, the data size, and the data transfer volume can advantageously be reduced.By means of a predetermined degree of compression (other than zero), it can also be ensured that no image information relevant for determining the condition of the food is lost, in particular if the predetermined degree of compression is selected - for example by experimental determination - in such a way that sufficient image information is available to carry out the specific purpose.

[0016] By reducing the color depth, image processing time, data size, and data transfer volume can also be advantageously reduced. Setting the white balance and / or exposure time to the same values ​​for image acquisition for specific purposes can, for example, be advantageous for creating a set of images that are particularly comparable within the same cooking process. This is particularly advantageous when capturing images to determine the degree of browning or browning progress. A further advantage is that the camera settings for such images are particularly easy to adapt to training data, so that the images can also be easily used by AI systems.

[0017] It is an embodiment that the household appliance is a household cooking appliance, in particular an oven, especially an baking oven, the treatment room is a cooking room and the treatment room camera is a cooking room camera.

[0018] It is a design that the different purposes include at least two purposes from the group:

[0019] - Geometric detection of food in the cooking chamber;

[0020] - Detection of the contents of the food in the cooking chamber;

[0021] - Detecting the condition of food in the cooking chamber;

[0022] - Detection of contamination of the cooking chamber;

[0023] - Detecting a shelf level on which the food to be cooked is located;

[0024] - Displaying at least one image to a user;

[0025] - Include data for later analysis. This advantageously includes many options for using a camera with optimized camera settings.

[0026] Geometric detection of food in the cooking chamber specifically means identifying which area of ​​an image depicts the food. This particularly facilitates the detection of the condition of food in the cooking chamber.

[0027] The detection of the content of food in the cooking chamber is understood in particular to mean that the type of food being cooked is detected ("automatic food detection"). The detection of the condition of food in the cooking chamber is understood in particular to mean that the cooking condition or cooking phase of the food is detected, e.g., still thawed, partially cooked, thoroughly cooked, etc., or, especially with dough, whether the dough is still in its rising phase or in its browning phase, etc.

[0028] Detecting contamination of the cooking chamber can be understood in particular to mean that it should be detected how heavily a cooking chamber wall, a surface of a cooking chamber lighting exposed to the cooking chamber (e.g. a protective window of a light or a surface of the light itself exposed to the cooking chamber, e.g. a light guide of the light) and / or a surface of the camera exposed to the cooking chamber (e.g. its lens) is dirty, e.g. is covered with grease.

[0029] The recognition of the shelf level on which the food to be cooked is located can be understood in particular as meaning that it should be recognized on which shelf level of several possible shelf levels a food carrier carrying the food to be cooked has been inserted or placed.

[0030] Displaying an image to a user can be understood in particular as showing one or more images to the user from a display, for example as a single image, a group of images, or as an image sequence (e.g., a video, for example, in the form of a time-lapse video of images taken at different times during the cooking process). The display can be a display of the household appliance or a display of an external instance such as a smartphone, tablet, etc.

[0031] Holding data available for analysis can be understood in particular to mean that an image or a sequence of images is to be evaluated. This data analysis can be carried out at a later time than the cooking process. To hold data available for analysis, one or more images can be saved in a data storage device of the household appliance or in an external data storage device. The data analysis can be used, for example, to identify user preferences, to train AI systems, etc. In one embodiment, the state of the food in the cooking chamber comprises at least one state from the group of degree of browning on a surface of the food and / or volume or change in volume of the food. This is particularly advantageous for determining a current degree of cooking and / or for determining the end of a cooking time.

[0032] One embodiment provides that, for a first purpose, at least one image is generated with a high resolution, without image compression, and with an automatically set exposure time. This is advantageous, for example, if the primary purpose is to display a high-quality image to a user and / or to use the images to determine the cooking process.

[0033] To create an image sequence, several images are generated with a high resolution, a specified compression ratio, and an automatically set exposure time. This has the advantage of creating an image sequence with a small data volume, in which the individual images are particularly easy to recognize for a user.

[0034] In one embodiment, at least one image with a low resolution, a predetermined compression level, a predetermined exposure time, and a predetermined white balance is generated to detect the condition of the food being cooked. The low resolution advantageously reduces the time required for image processing, data size, and data transfer volume. The predetermined compression level can advantageously ensure that this is not increased to a level at which image information relevant for determining the condition of the food being cooked is lost. The predetermined exposure time and / or the predetermined white balance advantageously ensure that a set of images can be generated that are particularly easy to compare within the same cooking process. This is particularly advantageous when capturing an image to determine a degree of browning or browning progress.

[0035] In one embodiment, at least one image with a low resolution, a predefined compression level, and a reduced color depth is generated to detect the shelf level on which the food is located. The low resolution and reduced color depth advantageously reduce image processing time, data size, and data transfer volume. The predefined compression level advantageously ensures that no image information relevant to determining the condition of the food is lost. The exposure time can be set automatically or predefined.

[0036] One embodiment provides that the image generated for detecting the insertion level is generated as a grayscale image. This advantageously allows image processing time, data size, and data transfer volume to be further reduced, particularly if an algorithm is used that does not require color information to determine the insertion height.

[0037] One embodiment is that, to detect soiling in the cooking chamber, at least one image is generated with a predefined resolution, a predefined compression level, and a reduced color depth. When detecting soiling in the cooking chamber, significant brightness fluctuations can occur, e.g., due to the contents of the cooking chamber, e.g., between an empty cooking chamber with dark walls and a cooking chamber in which a shelf level is lined with baking paper. To account for this, several images can be taken with different exposure times, for example. The exposure time can be set automatically or predefined.

[0038] A further development involves creating a so-called HDR ("High Dynamic Range") image from a series of images with different exposure times. In this case, the time interval between the images taken for this purpose is advantageously short. This can be achieved, for example, by taking images with no delay between the images.

[0039] In one embodiment, at least one image is stored for data analysis. In particular, images can be stored for all purposes or only for at least one of the purposes for data analysis. One embodiment is that, to retain the image for data analysis, at least one image is generated with a low resolution, a predefined compression ratio, and / or a predefined exposure time.

[0040] It is an embodiment that the at least one generated image is transmitted to an external instance for later data analysis, e.g. to a smartphone, a network server and / or a cloud computer.

[0041] It is a design that, e.g. during a cooking process or a cleaning process (e.g. pyrolysis, catalytic self-cleaning or manual cleaning)

[0042] - If it is decided to take a picture for a specific purpose,

[0043] - the camera settings associated with this specific purpose are provided or retrieved,

[0044] - at least one treatment room camera is configured according to the provided camera settings,

[0045] - the image is created (in particular recorded and possibly post-processed) and

[0046] - the generated image is made available for use for the specific purpose.

[0047] For each image acquisition purpose, a specific set of camera settings can be assigned, which is stored in a data storage device and thus made available for retrieval.

[0048] The fact that the generated image is provided for use for the specific purpose can include adjusting a control of a cooking process or a cleaning process based on the generated image - and possibly further images generated for the same purpose - e.g. adjusting at least one cooking parameter (e.g. a cooking chamber temperature), ending the cooking process when the end of a cooking time is reached, switching between cooking phases with different cooking parameter settings, etc.

[0049] It is a further development that the purpose of the image capture is stored as additional information for each image. This allows the images to be advantageously sorted by purpose in a data storage system and analyzed for each specific purpose. It is a further development that the camera settings used for a specific purpose are stored linked to the image, e.g., as metadata.

[0050] The object is also achieved by a household appliance comprising a treatment room and at least one treatment room camera with at least one variable camera setting, wherein the household appliance is configured to carry out the method as described above. The household appliance can be designed analogously to the method and has the same advantages.

[0051] The household appliance is in particular a household cooking appliance, especially an oven, possibly with a microwave and / or steam treatment function and / or automatic self-cleaning function (pyrolysis, catalytic self-cleaning, etc.).

[0052] The above-described properties, features and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more clearly understood in connection with the following schematic description of an embodiment, which is explained in more detail in connection with the drawings.

[0053] Fig.1 shows a simplified sketch of a household appliance in the form of an oven as a sectional side view;

[0054] Fig. 2 shows possible image acquisition times as a timeline over a time t; and Fig. 3 shows a possible embodiment of the method according to the invention.

[0055] Fig. 1 shows a simplified sectional side view of a cooking appliance in the form of an oven 1. The oven 1 has a cooking chamber 2 whose front loading opening can be closed by a pivoting door 3. Cooking chamber 2 has several shelf levels E1 to E3 into which a food carrier T such as a baking tray or a wire rack can be inserted. Food G can be placed on the food carrier T, here: on shelf level E2. The oven 1 further has a control device 4 for carrying out operating sequences such as cooking and cleaning sequences, as indicated here by an electric bottom heat element 5 shown as an example.

[0056] The control device 4 is also configured, e.g., programmed, as a data processing device for carrying out the method. A cooking chamber camera 6 in the form of a digital color camera is connected to the control device 4 for data purposes and can, in particular, record images from the cooking chamber 2 and transmit them to the control device 4. A lens 7 on the objective side of the camera 6 is exposed to the cooking chamber 2. In a further development, the objective having the lens 7 can be adjustable, e.g., with regard to focus and / or image width, etc.

[0057] The oven 1 also has a control panel 8 with a particularly touch-sensitive screen ("touchscreen"), on which, in a further development, images or sections thereof taken by the camera 6 can also be displayed.

[0058] The oven 1 can also be equipped with a communications module 9, via which data (in particular including images or image data) can be transmitted from the control device to an external entity such as a user terminal, in particular to a mobile user terminal such as a smartphone P or tablet, etc., and / or to a network server N or cloud computer. The communications module 9 can comprise, for example, an Ethernet module, a WLAN module, a Bluetooth module, etc.

[0059] Fig.2 shows, as a timeline over a time t, possible image recording times, which can be generated, for example, using the cooking chamber camera 6 from Fig.1. As an example, images are to be generated here for three different purposes Zj with i = 1, ..., 3, e.g. for

[0060] Purpose Z Detecting the contamination of the cooking chamber 2 by taking an image before or at the start of a cooking process with a specified resolution, a specified compression level, an automatically set exposure time and with a reduced color depth.

[0061] Purpose Z2: Creating an image sequence by generating a plurality of images with a high resolution, a predetermined degree of compression and with an automatically set exposure time at the times ti , ts , ts and t?, etc., which are in particular temporally equidistant here. These images can be stored, for example, in a data memory of the oven 1 (e.g. in a data memory of the control device 4) and, at the user's request, played back on the display of the control panel 8 and / or after transmission via the communication module 9 on the smartphone P. Purpose Z3: Control of the cooking process through browning detection, in particular for determining the end of cooking time and ending the cooking process when the end of cooking time is reached by generating a plurality of images with a low resolution, a predetermined degree of compression, a predetermined exposure time and with a predetermined white balance at the times t2 and t6, etc., which are in particular temporally equidistant here.

[0062] Fig.3 shows a possible embodiment of the method according to the invention, which is explained in more detail using the baking oven 1 as an example.

[0063] In the method, in a step S1, the control device 4 decides to generate an image for an associated purpose Zj by means of the cooking chamber camera 6 at a time tj, e.g. for controlling the cooking process according to purpose Z3.

[0064] In a step S2, the control device 4 reads out the associated camera settings provided for this purpose Zj in a, possibly internal, data memory (not shown) and transmits them to the cooking chamber camera 6. For purpose Z3, the camera settings can be, for example, a specific image resolution that is the same for all images for purpose Z3 and is lower than a highest possible image resolution, a predetermined compression ratio greater than zero that is the same for all images for purpose Z3 and / or a predetermined white balance that is the same for all images for purpose Z3.

[0065] In a step S3, the cooking chamber camera 6 is configured according to these camera settings.

[0066] In a step S4, an image is generated using the cooking chamber camera 6. Image generation may, in particular, comprise image capture using an image sensor of the cooking chamber camera 6 and image post-processing of the raw image captured by the image sensor. The image post-processing may be performed in the cooking chamber camera 6, e.g., in a data processing device thereof, or may be performed in another component of the oven, such as the control device 4.

[0067] In a step S5, the generated image is provided to the control device 4 for use for the specific purpose Zj by transmitting the generated image from the cooking chamber camera 6 to the control device 4, e.g. for controlling the cooking process by browning detection based only on the images taken at times t2 and te.

[0068] The process then branches back to step S1.

[0069] This procedure corresponds to a "multiplexing" of the camera settings of the cooking chamber camera 6 depending on the purposes for which the images are used.

[0070] Of course, the present invention is not limited to the embodiment shown.

[0071] For example, an image can be taken before or at the start of the cooking process to determine the shelf level E1 - E3 into which the food carrier T has been inserted. The control device 4 can then, for example, vary the heat output applied to the heating elements of the cooking chamber 2, such as the bottom heat element 4, at least one top heat element, a hot air heater, etc., depending on the selected shelf level E1 - E3.

[0072] For example, an image can be taken before or at the start of the cooking process to determine the position of the food item 6 in the image. Subsequently, in a variant for determining the state of the food item G, only those image points that show the food item G can be evaluated.

[0073] In addition, an image can be taken, for example, before or at the start of the cooking process to determine the type of food 6 being cooked, e.g., chicken, bread dough, pizza, etc. The control device 4 can then be configured, for example, to automatically adapt the cooking process to the detected type of food G, e.g., to adjust a cooking chamber temperature and its progression, etc. The cooking product can be detected using artificial intelligence (K1) methods, e.g., by using a neural network. For this purpose, it is particularly advantageous if the position of the food 6 in the image has been determined beforehand.

[0074] In addition to or as an alternative to the degree of browning, images can be taken to determine the volume or volume change of the food G. This is particularly advantageous with dough, e.g., bread dough, to distinguish between a rising phase and a browning phase. The control device 4 can then, for example, be configured to automatically adapt the cooking process to the detected phases or phase ends. In general, "a," "an," etc., can be understood as a singular or a plural, in particular in the sense of "at least one" or "one or more," etc., as long as this is not explicitly excluded, e.g., by the expression "exactly one," etc.

[0075] A numerical value may also include the exact number stated as well as a usual tolerance range, as long as this is not explicitly excluded.

[0076] List of reference symbols

[0077] 1 oven

[0078] 2 cooking chamber

[0079] 3 Door

[0080] 4 Control device

[0081] 5 bottom heat radiators

[0082] 6 Cooking chamber camera

[0083] 7 lens

[0084] 8 Control panel with display

[0085] 9 Communication module

[0086] E1-E3 shelf levels

[0087] G Food to be cooked

[0088] N network server

[0089] P Smartphone

[0090] S1-S5 Process steps t Time tj Times for generating images

[0091] T Food carrier

[0092] Zj purposes

Claims

Patent claims Method (S1-S5) for operating a household appliance (1) with a treatment room (2) and at least one treatment room camera (6) with at least one variable camera setting, in which - several images are generated (S4) by means of the at least one treatment room camera (6) at a time offset (to - 1?) and - at least two of the images are used for at least two different purposes (Zi - Z3), where - at least one camera setting of the at least one treatment room camera (6) is varied (S2, S3) depending on the purpose. Method (S1-S5) according to claim 1, wherein the at least one camera setting is at least one camera setting from the group - Image resolution; - degree of compression; - Color depth; - White balance; - Sensor gain; - exposure time; - Gamma correction; - Image file format; - Use of a flash; - lens adjustment, if the treatment room camera (6) has an adjustable lens (7). Method (S1-S5) according to one of the preceding claims, wherein the household appliance (1) is a household cooking appliance, in particular an oven, the treatment room (2) is a cooking chamber, and the treatment room camera (6) is a cooking chamber camera.

4. Method (S1-S5) according to claim 3, wherein the different purposes (Zi - Z3) comprise at least two purposes (Z1 - Z3) from the group: - Geometric detection of food in the cooking chamber (G); - Detection of the contents of the food in the cooking chamber (G); - Detecting a state of food (G) (Z3) located in the cooking chamber (2); - Detection of contamination of the cooking chamber (Z1); - Detection of a shelf level (E1-E3) on which the food (G) is located; - Displaying at least one image to a user (Z2); - include provisions for data analysis.

5. Method (S1-S5) according to claim 4, wherein the state of the food (G) located in the cooking chamber (2) comprises at least one state from the group - Degree of browning on a surface of the food (G) (Z3); - Volume or change in volume of the food being cooked (G).

6. Method (S1-S5) according to claims 2 and 3 and one of claims 4 and 5, in which for one purpose at least one image with - a high resolution, - without image compression and with - an automatically set exposure time.

7. Method (S1-S5) according to claims 2 and 3 and one of claims 4 to 6, wherein to create an image sequence (Z2) several images with - a high resolution, - a given compression level and with - an automatically set exposure time (S2, S3).

8. Method (S1-S5) according to claims 2 and 3 and one of claims 4 to 7, in which to recognize the state of the food (G) (Z3) at least one image with - a low resolution, - a given degree of compression, - a given exposure time and with - a predefined white balance is generated (S2, S3).

9. Method (S1-S5) according to claims 2 and 3 and one of claims 4 to 8, in which, in order to recognize the shelf level on which the food to be cooked is located, at least one image with - a low resolution, - a given degree of compression and - a reduced color depth is created.

10. Method (S1-S5) according to claim 9, wherein the image generated for detecting the insertion plane is stored as a grayscale image.

11. Method (S1-S5) according to claims 2 and 3 and one of claims 4 to 10, in which to detect the contamination of the cooking chamber (G) (Zi) at least one image with - a given resolution, - a given degree of compression and - an automatically set exposure time and with - a reduced color depth is generated (S2, S3).

12. Method (S1-S5) according to claims 2 and 3 and one of claims 4 to 11, in which, for the purpose of retaining for later data analysis, at least one image with - a low resolution, - a given degree of compression and - a given exposure time.

13. The method (S1-S5) according to claim 12, wherein the at least one generated image is transmitted to an external instance (P, N) for later data analysis.

14. The method (S1-S5) according to one of the preceding claims, wherein - It is decided to create an image for a specific purpose (S1), - the camera settings associated with this specific purpose are provided (S2), - at least one camera is configured according to the provided camera settings (S3), - the image is generated (S4) and - the generated image is provided for use for the specific purpose (S5).

15. A household appliance (1), in particular a household cooking appliance, comprising a treatment chamber (2) and at least one treatment chamber camera (6) with at least one variable camera setting, wherein the household appliance (1) is configured to carry out the method (S1-S5) according to one of the preceding claims.

Images