Identifying food to be cooked in a thermal image

Abstract

The invention relates to a method (S1-S9) for identifying food to be cooked (G) in at least one pixel-based thermal image (W1, W2) captured from a cooking chamber (2) of a microwave oven (1), in which method microwave energy is emitted into the cooking chamber (2), a thermal image (W2) is captured, pixels which have temperature values that are increased by a specified temperature level in relation to the set of all pixels are identified, and a fill algorithm is performed which assigns pixels that belong to an image area which is framed by at least some of the identified pixels to the food to be cooked (G). The invention also relates to a microwave oven (1) comprising a cooking chamber (2) which can be subjected to microwaves, a thermal imaging camera (11) for capturing thermal images (W1, W2) from the cooking chamber (2), and a control device (13), the microwave oven (1) being designed to carry out the method (S1-S9; S1-S10). The invention can be applied particularly advantageously to domestic microwave ovens.

Description

[0001] Identifying food in a thermal image

[0002] The invention relates to a method for identifying food to be cooked in at least one pixel-based thermal image recorded from a cooking chamber of a microwave cooking appliance, in which microwave energy is radiated into the cooking chamber, a thermal image is recorded, and pixels are identified which have temperature values ​​increased by a predetermined temperature level in relation to the set of all pixels. The invention also relates to a method for operating a microwave cooking appliance, in which a position of the food to be cooked in a thermal image is determined using the method, and a cooking process is controlled based on an evaluation of the pixels assigned to the food. The invention further relates to a microwave cooking appliance, comprising a cooking chamber to which microwaves can be applied, a thermal imaging camera provided for recording thermal images from the cooking chamber, and a control device, wherein the microwave cooking appliance is configured to carry out the method.The invention is particularly advantageously applicable to household microwave cooking appliances.

[0003] For advanced control systems for cooking processes in microwave ovens to improve cooking results, which operate based on image-based infrared (IR) or thermal radiation measurements, precise identification of the food being cooked is important. This means, in particular, that a precise distinction must be made as to whether a pixel in the image of the field of view of a thermal imaging camera is assigned to or belongs to a sub-area of ​​the food being cooked or—irrelevant for the control of the cooking process—is assigned to an area surrounding the food being cooked, such as a food support (e.g., plate, grill rack, baking tray, etc.) or a cooking chamber wall.

[0004] The identification of an area assigned to a food item in an image of the cooking chamber can be performed by a user by manually marking it on a touchscreen. However, this is not very user-friendly and is often inaccurate. Complex image analysis methods, in particular based on artificial intelligence ("AI") methods, can also be used to automatically recognize the food item area in an image. However, this is very computationally and training-intensive. WO 2020 / 156928 A1 discloses a household cooking appliance. The household cooking appliance comprises a cooking chamber heating device configured for localized heating of a cooking chamber and operable with at least two configurations that generate different energy distributions in the cooking chamber, a temperature detection device configured for contactless detection of a heat distribution in the cooking chamber, a data processing device,which is configured to distinguish a non-cooked food area from at least one area of ​​the cooking chamber occupied by food from the detected heat distribution, and a control device configured to set a current configuration of the cooking chamber heating device with a view to increasing energy output into the detected food and to control the cooking chamber heating device, the temperature detection device, and the data processing device, wherein the data processing device is configured to distinguish the non-cooked food area from the food in the detected heat distribution based on a temperature difference between the non-cooked food area and the food.

[0005] WO 2020 / 156929 A1 discloses a microwave oven.The microwave appliance comprises a microwave device configured to generate microwaves and introduce the microwaves into a cooking chamber and operable with at least two configurations that generate different field distributions of the microwaves in the cooking chamber, a temperature detection device configured to contactlessly detect a heat distribution in the cooking chamber, a data processing device configured to detect a non-cooked food area in the cooking chamber from the detected heat distribution, and a control device configured to set a current configuration of the microwave device and to operate the microwave device, wherein the control device is configured to select or set at least one configuration of the microwave device with a view to reducing a power of the microwaves in the detected non-cooked food area.

[0006] EP 3 767 580 A1 discloses a control unit for a household appliance having at least one reference mark in an interior space, wherein the reference mark has reference values ​​for one or more different properties. The control unit is configured to capture image data relating to the interior space of the household appliance using a camera of the household appliance and to identify the reference mark in the image data. Furthermore, the control unit is configured to determine actual values ​​for the one or more properties of the reference mark based on the image data. The control unit is further configured to process the image data depending on the actual values ​​and depending on the reference values ​​in order to determine object information relating to an object in the interior space of the household appliance and / or to provide an enhanced image relating to the interior space of the household appliance.

[0007] DE 10 2017 101 183 A1 discloses a method for operating a cooking appliance and a cooking appliance in which food is heated in a cooking chamber using a heating device. The food is detected using a camera device. Based on the detection of the food, at least one cooking parameter is determined. The heating device comprises a heat source with a plurality of separately controllable heating elements. Each of a plurality of spatial segments in the cooking chamber is specifically heated using at least one heating element. The individual heating elements are controlled depending on the cooking parameter.

[0008] WO 2016 170 734 A1 discloses a cooking appliance comprising: a microwave generating unit that generates microwaves; a heating chamber for accommodating an object to be heated; an infrared sensor installed within the heating chamber; a scanning unit that moves the infrared sensor for scanning; and a control unit that controls the microwave generating unit based on the output of the infrared sensor. The infrared sensor obtains a plurality of temperature distributions by obtaining a temperature distribution each time the infrared sensor is scanned over a predetermined distance. The control unit controls the microwave generating unit according to a temperature distribution obtained by summing the plurality of temperature distributions.

[0009] US 10 219 330 B2 discloses an electronic oven and an accompanying control system that prevent boiling or splashing in a heating chamber of the oven while heating an object in the chamber. A method that can be performed by the control system includes evaluating sensor data from a visible light sensor and sensor data from an infrared light sensor. The controller is communicatively coupled to the visible light sensor and the infrared light sensor. The method is directed to generating a splash prediction in response to the evaluation of the sensor data from the visible light sensor and the sensor data from the infrared light sensor. The method is further directed to reducing a power level of the microwave energy source in response to the splash prediction. The controller is also communicatively coupled to the microwave energy source.

[0010] EP 2 618634 A1 discloses a microwave heating device and a method for heating a load using microwaves. The microwave heating device comprises a cavity arranged to receive a load, a plurality of supply openings for supplying microwaves from a plurality of microwave generators to the cavity, and a control unit. The control unit is configured to obtain a desired temperature pattern within the cavity based on information about a plurality of regions of the load, determine a heating pattern comprising corresponding zones of different intensity to adjust to the desired temperature pattern, and control at least some of the plurality of microwave generators to provide the heating pattern within the cavity.

[0011] WO 2012 / 109634 A1 discloses an apparatus for processing objects with RF energy. The apparatus may comprise a display for displaying an image of an object to be processed to a user, wherein the image includes at least a first portion and a second portion of the object. The apparatus may also comprise an input unit and at least one processor configured to: receive information based on an input provided to the input unit; and, based on the received information, generate processing information for use in processing the object to achieve a first processing result in the first portion of the object and a second processing result in the second portion of the object.

[0012] EP 1 997 349 B1 discloses an electromagnetic heating device for heating an irregularly shaped object, comprising: a cavity in which an object is to be placed; at least one feeder that feeds UHF or microwave energy into the cavity; and a controller that controls one or more properties of the cavity or the energy to ensure that the UHF or microwave energy is fed into the object evenly within ±30% over at least 80% of the volume of the object. WO 2020 / 200913 A1 discloses a household appliance.The household appliance comprises a treatment chamber for treating goods, in particular goods to be cooked, at least one sample light configured to irradiate at least one light pattern into the treatment chamber, and at least one image sensor directed into the treatment chamber for recording the at least one light pattern reflected from the treatment chamber. The sample light is rotatable by means of a motor, and the household appliance is configured to determine at least one piece of contour information of goods irradiated by the light pattern from at least two reflected light patterns associated with different angles of rotation of the at least one sample light. A method is used for determining contour information of goods located in a treatment chamber of a household appliance. WO 2020 / 200913 A1 is particularly advantageously applicable to determining contour information of goods to be cooked in an oven.

[0013] EP 3 574 711 A1 discloses methods and systems relating to improved human-machine interfaces for electronic ovens. Various methods for displaying information to the user are disclosed. Various methods for distributing segmentation and identification tasks between a user and a control system are disclosed. In one example, an electronic oven includes a touch display, a heating chamber for heating an article, a light sensor having a field of view of at least a portion of the heating chamber, and a microwave energy source coupled to the heating chamber. The oven also includes a computer-readable medium storing instructions for displaying the portion of the heating chamber as an image on the touch display using information from the light sensor and processing a touch input on the image.

[0014] EP 3 767 183 A1 discloses a method for detecting contamination of a household appliance. The method comprises the steps of providing a reference image of a cavity or section of the household appliance, acquiring a current image of the same cavity or section, comparing the current image with the reference image, generating a difference image of the same cavity or section, and checking whether a pixel and / or a group of neighboring pixels of the difference image exceeds a predefined threshold. EP 2 055 146 B1 discloses, as shown in FIG. 1, how the geometric shape of an object can be determined by measuring with RF energy.

[0015] DE 10 2020 215 681 A1 discloses a household microwave appliance which is operated successively under a plurality of parameter configurations which treat the food to be cooked differently locally in order to carry out an initial scan by means of a thermal imaging sensor directed into the cooking chamber for determining temperature distributions on a surface of the food to be cooked in order to obtain change patterns from differences between different temperature distributions, for which an evaluation value is calculated which, based on a target temperature distribution which results from a standardized target state and a current temperature distribution, determines the heating pattern which best approximates the current temperature distribution to the target temperature distribution and subsequently the food to be cooked is subjected to microwave power with the parameter configuration belonging to the heating pattern.

[0016] It is the object of the present invention to at least partially overcome the disadvantages of the prior art and in particular to provide an improved possibility of recognizing food in a pixel-based image taken from a cooking chamber of a microwave cooking appliance.

[0017] This object is achieved according to the features of the independent claims. Preferred embodiments can be found in particular in the dependent claims.

[0018] The object is achieved by a method for identifying food in at least one pixel-based thermal image taken from a treatment chamber ("cooking chamber") of a microwave cooking appliance, in which

[0019] - microwave energy is radiated into the cooking chamber,

[0020] - a thermal image is taken,

[0021] - pixels are identified which have temperature values ​​increased by a given temperature level in relation to the set of all pixels,

[0022] - a filling algorithm is executed that assigns pixels to the food that belong to an image area bordered by at least some of the identified pixels. This process offers the advantage that it is very precise, user-friendly, requires little computational effort, and thus very fast, and also runs automatically—in contrast to manual marking on a touchscreen or automated methods such as image recognition.

[0023] The method exploits the so-called "edge overheating effect," in which a heating image frequently shows the behavior that the edges of the food are heated significantly in some places when exposed to microwaves, especially initially, while the inner area of ​​the food surface is heated significantly less or barely noticeable. The "edge overheating effect" occurs in virtually any food containing water. While this effect fundamentally complicates the cooking process, as increased energy input occurs at the edges, while the center heats up only slowly, this phenomenon now enables a method with minimal computational effort to capture the outer contour of the food and thereby distinguish between pixels belonging to the food and pixels not belonging to it. This assignment can also be referred to as "areal masking."

[0024] The identification of food in a thermal image corresponds in particular to the allocation of the pixels as to whether a temperature is recorded on a surface of the food or - conversely - a temperature not on a surface of the food, but e.g. a temperature on a surface of another object such as a food carrier, a cooking chamber wall, etc. The identification thus corresponds in particular to an identification of the position of the food, not its type.

[0025] The cooking chamber serves to house items to be treated, particularly cooked, with microwaves. The cooking chamber is designed to be exposed to microwaves. It often has a front loading opening that can be sealed with a microwave-tight door.

[0026] The microwaves are generated by at least one microwave generator, for example, a magnetron or a semiconductor-based microwave generator. The microwaves can be introduced into the cooking chamber directly from the microwave generator or via a respective microwave guide. A further development is for microwaves to be introduced into the cooking chamber via a microwave feed point ("microwave port") or via multiple microwave ports, possibly phase-shifted in the case of multiple microwave ports, in particular with a variably adjustable phase shift. Rotatable antennas and / or stirrers can also be present at the microwave port in order to vary the mode pattern of the microwaves in the cooking chamber. Furthermore, the mode pattern in the area of ​​the food being cooked can be varied by providing a turntable, if present. The frequency of the microwaves can in particular be in the range between 2.4 GHz and 2.5 GHz, in particular approximately 100 MHz.2.45 GHz, or in the range between 902 MHz and 928 MHz, especially around 915 MHz. A semiconductor-based microwave generator, in particular, makes it possible to specifically vary the frequency of the microwaves.

[0027] The pixel-based thermal image is constructed in particular from a matrix-like arrangement of pixels, whereby the values ​​associated with the pixels correspond to a temperature in the surface area measured by the respective pixels.

[0028] The pixel-based thermal image is typically captured by a digital thermal imaging camera. To capture a large area of ​​the food being cooked, the thermal imaging camera is advantageously positioned near the ceiling of the cooking chamber wall and, in particular, is directed downwards or diagonally downwards. The field of view of the thermal imaging camera advantageously includes at least the floor of the cooking chamber wall and, if necessary, also parts of the sides of the cooking chamber wall.

[0029] The microwave cooking appliance can also have several thermal imaging cameras, which advantageously take images of the cooking chamber from different angles.

[0030] In one embodiment, the microwave cooking appliance comprises, in addition to the at least one digital thermal imaging camera, at least one digital "optical" cooking chamber camera that captures images in the optical spectral range. This is advantageous for controlling a cooking process based on an optical property of the surface of the food, such as a degree of browning, a color change, a change in volume, etc. The images captured by this camera can also be displayed to a user. The optical cooking chamber camera can, for example, be a digital RGB color camera. In one embodiment, advantageous for capturing large-area food, the optical cooking chamber camera is arranged in the region of a ceiling of the cooking chamber wall and, in particular, is directed downwards or diagonally downwards.It is a further development that the thermal imaging camera and the optical cooking chamber camera are arranged close to each other, which has the advantage that the thermal images and the optical images show particularly similar areas of the cooking chamber and are particularly easy to compare.

[0031] By radiating microwave energy into the cooking chamber, the food inside is heated. The edges of the food are not only heated more than the outer "non-food" area, but due to the "edge overheating effect," they are also typically heated more than the center of the food. After a certain exposure time, a thermal image of the cooking chamber is recorded, typically depicting the temperature distribution of the surface of the food.

[0032] All pixels in the thermal image exhibit corresponding temperature values, with at least some pixels associated with the edge of the food being cooked being identifiable by having a significantly higher temperature level or displaying a significantly higher temperature than the other pixels. The pixels with higher temperature levels can be differentiated from other pixels, for example, by threshold comparison and thus identified. These pixels with higher temperature levels show at least sections of the edge or outer contour of the food being cooked from the perspective of the thermal imaging camera.

[0033] The fill algorithm then fills the area in the image plane bounded by the pixels assigned to the edge of the food, or identifies the pixels that lie within such an area. After the fill algorithm has been executed, those pixels that lie within the area, including its edge, are assigned to the food. Conversely, those pixels that lie outside the area, including its edge, are not assigned to the food.

[0034] It is a further development that the filling algorithm is designed in such a way that pixels previously assigned to the edge of the food are not assigned to the area corresponding to the food after the filling algorithm has been carried out, but to the non-food area because they did not satisfy certain criteria of the filling algorithm for assignment to the area.

[0035] In one embodiment, a first thermal image is recorded before the microwave energy is radiated into the cooking chamber; a second thermal image is recorded after the microwave energy is radiated into the cooking chamber; a differential thermal image is created in which, for each pixel, a temperature difference between the temperature value recorded with the second thermal image and the temperature value recorded with the first thermal image is calculated; and the pixels are identified in the differential thermal image. This offers the advantage that, after the cooking chamber has been loaded with the food, existing temperature differences between the food and the surroundings of the food, or even within the food, are taken into account, and their influences are neutralized.

[0036] One embodiment involves mapping or "normalizing" the temperature differences to a value range [0; 1] (or other predefined value range). This provides the advantage that the process can be more easily adapted to different types of food and / or to different microwave operating parameters during microwave feed.

[0037] One embodiment identifies those pixels as being associated with the edge of the food that reach or exceed a specified temperature threshold. This is advantageously particularly easy to implement.

[0038] One embodiment provides that when the temperature differences are mapped to a value range [0; 1], the temperature threshold lies in a range [0.2; 0.5]. This has proven to be a particularly good compromise between precise detection of the edge of the food and a sufficiently closed edge. If the temperature threshold is set too low, e.g. 0.1, the food G is usually completely detected; however, large areas of the food support or the cooking chamber walls may then be incorrectly identified as belonging to the food. If the temperature threshold is set too high, e.g. 0.7, only individual, particularly strongly heated edge segments of the food G are detected, which may be too few to reliably define an edge that is filled in by the Starfill algorithm.Furthermore, it is advantageous that by mapping the value range to a fixed range, the same threshold value can be suitably applied to many different cooking items and / or different microwave operating parameters.

[0039] One embodiment uses an algorithm (hereinafter referred to as the "Starfill" algorithm without loss of generality) in which a pixel is assigned to the food if the number of directions radiating from this pixel, in which identified pixels are located, reaches or exceeds a minimum number. In particular, this prevents gaps in the identified edge from causing areas outside the edge to also be assigned to the food, for example, in contrast to the so-called "Floodfill" algorithm (which is often used in graphics programs to fill bordered areas with color). Furthermore, the "FloodfiH" algorithm requires one of the interior points of the bordered area as a starting point, which, by definition, are unknown at the beginning of the evaluation.

[0040] The Starfill algorithm checks whether a given pixel is associated with the food item in various directions within the image plane of the thermal image, particularly the differential thermal image, radiating from this pixel in a star-shaped pattern. The given pixel is then associated with the food item if a predefined minimum number of directions meet this condition. The Starfill algorithm can perform this check and subsequent association for all pixels of the thermal image, particularly the differential thermal image, for example, row- and column-wise.

[0041] In an (x, y) matrix-shaped thermal image, directions radiating "star-shaped" from a pixel are the symmetrical directions along the x-extension and the y-extension as well as, if applicable, the oblique directions lying evenly between these (main) extensions.

[0042] The smallest number of star-shaped directions for an interior point lying within the (x, y) matrix is ​​four, namely starting from this image point

[0043] - in x-direction,

[0044] - in (-x) direction,

[0045] - in y-direction and - in (-y)-direction.

[0046] The next largest number of star-shaped directions is eight, namely, starting from this pixel, the additional four directions through the nearest neighboring pixels, which are oblique to the x- and y-extensions. This can, in principle, be extended to 12, 16, 20, etc. directions as desired. The higher the number of directions, the higher the accuracy of the assignment at oblique edges of the boundary.

[0047] Edge points and vertices have a reduced number of possible directions available, namely two directions for vertices and three directions for edge points, when generally - for interior points - four directions are available, three directions for vertices and five directions for edge points, when generally - for interior points - eight directions are available, etc.

[0048] One embodiment is that for a total of eight star-shaped directions, at least one of the following boundary conditions of the Starfill algorithm is set: a minimum of two corner points applies; a minimum of three edge points applies; and a minimum of six interior points applies. This has proven to be a particularly good compromise between completely filling the volume bounded by the edge and avoiding assigning pixels outside the edge to the food. With a total of eight star-shaped directions, a maximum of three directions can emanate from a corner point, a maximum of five directions from an edge point, etc.

[0049] A further development is that the starfill algorithm is applied to a specific pixel and then applied to the next pixel, including the result for all previous pixels. The starfill algorithm can be performed, in particular, row- and column-wise.

[0050] It is a further development that already identified pixels remain unchanged, i.e., an assignment to the food item once made is not resolved, e.g., even if the specified minimum number of directions is no longer met for this pixel or the conditions of the Starfill algorithm are no longer met. It is a further development that an identified pixel or one assigned to the food item is no longer assigned to the food item (i.e., the assignment is resolved) if the specified minimum number of directions is no longer met.

[0051] One embodiment involves executing the filling algorithm multiple times ("recursively") for all pixels until no further changes occur in the identified pixels. The method can thus perform the assignment of pixels iteratively. This is particularly advantageous for achieving the most complete assignment of pixels to the food or not to the food.

[0052] Alternatively or additionally, the filling algorithm can be executed multiple times until a maximum number of passes or iterations has been reached.

[0053] In one embodiment, the pixels assigned to the food being cooked are additionally converted to pixels from an optical digital camera of the microwave cooking appliance. This is particularly advantageous for improving optical monitoring (in the visible spectral range) of food being cooked, e.g. with regard to the degree of browning, a change in volume, etc., by comparing it with the present method, since this makes information about the position of the food being cooked usable for optical monitoring. This exploits the fact that pixels from the thermal imaging camera can be correlated with pixels from an optical digital camera, so that if a pixel from the thermal imaging camera is assigned to the food being cooked, one or more pixels from the optical digital camera that show the same area of ​​the food being cooked can also be assigned to the food being cooked.It is a further development that pixels of the thermal imaging camera that are not assigned to the food being cooked are converted into pixels of an optical digital camera of the microwave cooking appliance.

[0054] A further development allows this to be carried out in reverse, i.e., pixels from an optical digital camera assigned to the food and / or pixels from an optical digital camera not assigned to the food are converted to corresponding pixels from the thermal imaging camera. This means that if at least one pixel from the optical digital camera is not assigned to the food, a pixel from the optical thermal imaging camera that shows the same area of ​​the food will also not be assigned to the food. One embodiment allows the position of the food to be identified in an initial phase of a cooking process. This is particularly advantageous because the food is not yet thoroughly heated and the "edge overheating effect" is particularly pronounced. This, in turn, increases the reliability of the method.

[0055] One embodiment is that, while the microwave energy is radiated into the cooking chamber for the purpose of carrying out the method, at least one microwave operating parameter that changes a microwave mode pattern in the cooking chamber is varied. This advantageously avoids narrowly localized spatial areas with high microwave energy (so-called "hot spots") or at least varies them sufficiently to achieve a larger-area heating of the food. This, in turn, is advantageous for being able to detect the increased heating of the edge of the food caused by the "edge overheating effect" particularly well using the thermal imaging camera, especially over the entire edge if possible, and for keeping the number and / or length of edge sections that do not heat up particularly strongly to a minimum.This configuration can be implemented particularly advantageously in connection with DE 102020215681 A1, specifically within the framework of the "initial scan" mentioned in DE 102020215681 A1, in which microwaves are fed into the cooking chamber under different parameter configurations, temperature distributions on the surface of the food corresponding to the parameter configurations are measured using the thermal imaging camera, and heating patterns are determined from the differences between the different temperature distributions. The parameter configurations can then correspond, in particular, to the above-mentioned microwave operating parameters.

[0056] It is a further development that a microwave operating parameter contains at least one parameter from the group

[0057] - if a rotating antenna is present: angle of rotation of the rotating antenna, e.g. in the range [0°; 180°] or in the range [0°; 360°], e.g. in steps of 1°, 5° or 10°;

[0058] - in the presence of a mode stirrer: angle of rotation of the mode stirrer, e.g. in the range [0°; 180°] or in the range [0°; 360°], e.g. in steps of 1°, 5° or 10°; - in the case of a microwave generator with variable microwave frequency, in particular a semiconductor-based microwave generator: microwave frequency, e.g. in the range [2.4 GHz;

[0059] 2.5 GHz] e.g. in steps of 10 MHz;

[0060] - in the case of multiple microwave ports with the possibility of varying a phase difference between the microwaves emitted by these microwave ports: phase difference, e.g. in the range [0°, 360°]. It is advantageous if the turntable (if present) has rotated at least once, the rotating antenna (if present) has rotated at least once, the microwave frequency has been varied completely once (if possible), the phase differences have been traversed once (if possible), etc. When using a turntable, the second thermal image is preferably recorded after an integer number of complete revolutions in order to ensure that the food being cooked is in the identical position when the first and second thermal images are recorded. Otherwise, the thermal images would have to be aligned with one another using a suitable rotation matrix, which is also possible in principle.

[0061] The object is also achieved by a method for operating a microwave cooking appliance, in which the position of the food to be cooked is determined using the method as described above, and a cooking process is controlled based on an evaluation of at least the pixels assigned to the food to be cooked, possibly also based on a combination of an evaluation of the pixels assigned to the food to be cooked and an evaluation of the pixels not assigned to the food to be cooked. The method for operating the microwave cooking appliance can be designed analogously to the method for identifying food to be cooked described above, and vice versa, and has the same advantages. For example, determining the position of the food to be cooked can be used to more accurately detect a target cooking state and thus increase the reliability of success and improve the cooking result. Upon reaching the target cooking state, at least one action can be triggered, e.g.the feeding of microwaves is stopped and / or a notice is given to a user, etc.

[0062] The object is also achieved by a microwave cooking appliance comprising a cooking chamber capable of being exposed to microwaves, a thermal imaging camera provided for recording thermal images from the cooking chamber, and a control device. The microwave cooking appliance, in particular its control device, is configured to carry out the method(s) described above. The microwave cooking appliance can be designed analogously to the methods, and vice versa, and has the same advantages.

[0063] The microwave cooking appliance is, in particular, a household appliance. The microwave cooking appliance can be a standalone microwave appliance or a combination appliance, e.g., a microwave appliance with an additional heat radiator, e.g., an electrical resistance heating element, particularly in the form of a tabletop appliance, or an oven with microwave functionality.

[0064] In particular, the microwave cooking appliance may comprise one or more of the following devices:

[0065] - a microwave-tight cooking chamber door for microwave-tight closure of a front loading opening of the cooking chamber;

[0066] - at least one microwave generator;

[0067] - at least one microwave feed point;

[0068] - a rotating antenna;

[0069] - a fashion stirrer;

[0070] - a turntable.

[0071] The above-described properties, features and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more clearly understood in connection with the following schematic description of an embodiment, which is explained in more detail in connection with the drawings.

[0072] Fig.1 shows a sectional side view of a sketch of a microwave cooking appliance which is set up to carry out a method for identifying food in a pixel-based thermal image taken from its cooking chamber;

[0073] Fig.2 shows a possible sequence of the procedure;

[0074] Fig. 3 shows a pixel-based thermal image of a food placed on a plate before microwaves were applied; Fig. 4 shows a pixel-based thermal image of the food placed on the plate from Fig. 2 after microwaves were applied.

[0075] Fig.5 shows a temperature difference image generated from the thermal images according to Fig.3 and Fig.4;

[0076] Fig.6 shows an assignment of pixels to the food derived from the temperature difference image before application of the Starfill algorithm; and

[0077] Fig.7A shows an application of the Starfill algorithm for an interior point of the temperature difference image;

[0078] Fig.7B shows an application of the Starfill algorithm for an edge point of the temperature difference image;

[0079] Fig.7C shows an application of the Starfill algorithm for a corner point of the temperature difference image;

[0080] Fig.8 shows an assignment of pixels to the food derived from the temperature difference image after the first application of the Starfill algorithm; and

[0081] Fig.9 shows an assignment of pixels to the food derived from the temperature difference image after further application of the Starfill algorithm.

[0082] Fig. 1 shows a sectional side view of a sketch of a microwave cooking appliance 1 with a cooking chamber 2, which is set up to run a method for identifying food G in a pixel-based thermal image W1, W2 taken from a cooking chamber 2 (see Fig. 3 and Fig. 4). The cooking chamber 2 can be closed at its front loading opening by a cooking chamber door 3. In the present case, the microwave cooking appliance 1 is designed purely as an example as an oven with microwave functionality, in which the food G can typically be placed on a food carrier 4 located at a specific insertion level, such as a baking tray or wire rack, etc., e.g. lying on a cooking utensil such as a plate 5.

[0083] The microwave cooking appliance 1 has a microwave generator 6, in particular a semiconductor-based one, for generating microwaves, in particular in a range [2.4 GHz; 2.5 GHz]. The generated microwaves are guided via a microwave guide 7, in particular in the form of a waveguide, to a microwave feed point 8 and fed there into the cooking chamber 2. A rotatable antenna 9 is located at the microwave feed point 8 for the purpose of mode variation. Furthermore, a pixel-based thermal imaging camera 11 for recording thermal images W1, W2 is located on a ceiling 10 of the cooking chamber 2. The camera is directed into the cooking chamber 2 from above and has the cooking chamber 2 with the food G to be cooked therein in its field of view. Optionally, the microwave cooking appliance 1 can also have an optical digital camera 12, in particular also arranged on the ceiling 10 of the cooking chamber 2, specifically close to the thermal imaging camera 11.

[0084] The microwave generator 6, a motor of the rotating antenna 9, as well as the thermal imaging camera 11 and, if applicable, the optical digital camera 12, can be controlled by a control device 13. The control device 13 can also be configured to execute the method described below.

[0085] Fig.2 shows a possible sequence of the method for identifying the position of the food G in a thermal image W1, W2.

[0086] In a step S1, the food G to be cooked is placed on the food support 4, possibly on or in a dish such as a plate 5, a bowl, etc. In addition, a microwave power level is set.

[0087] In a step S2, a first thermal image W1 of the not yet heated food G is taken from the cooking chamber 2, which is shown in Fig. 3 with a temperature scale on the right edge. The thermal image W1 shows, for example, a plate with food G at approximately room temperature (here: ravioli with sauce in a round soup plate). The thermal image records the IR pixels arranged in the form of an (x, y) matrix, the values ​​of which correspond to the local surface temperatures of the objects in the field of view of the thermal imaging camera 11 or at least correlate with the surface temperatures. Consequently, a temperature distribution in the cooking chamber 2 is mapped onto a matrix of (x ■ y) IR pixels, wherein the appropriately used thermal imaging cameras 11 can have several hundred to several thousand individual pixels. The resolution of the displayed thermal image W1 contributes (24 ■ 32) = 768 pixels.The method is also applicable to spatially distributed and separated food items, e.g., individual potatoes with a piece of meat next to them. In a step S3, microwaves are radiated into the cooking chamber 2 at the set microwave power, wherein during the radiating process, at least one microwave operating parameter that changes a mode pattern of the microwaves in the cooking chamber 2 is varied, e.g., a rotation angle of the rotating antenna 9 and / or a microwave frequency. For example, the rotating antenna 9 can be rotated continuously during this initial phase or "initial scan." It is advantageous to pass through as many different field distributions or mode patterns as possible in order to effect a temperature increase in as many areas of the food item G as possible.

[0088] Step S3 can be carried out for a predetermined duration, e.g., between 10 s and 30 s, although this duration (e.g., analogous to the initial scan described in DE 10 2020 215 681 A1) can be dependent, for example, on the set power, a thermal mass of the food G, and the absorption capacity of the food G. It is advantageous if, in step S3, the rotating antenna 9 has rotated at least once around itself, the microwave frequency has been varied completely once (if possible), the phase differences have been passed through once (if possible), etc.

[0089] After step S3, a second thermal image W2 is taken in step S4, which is shown in Fig. 4. It shows that the food G heats up significantly more than the cooking chamber walls or visible parts of the plate on which the food G is positioned. In particular, the "edge overheating effect" is also evident, in which an edge of the food G is heated particularly strongly, while the inner area of ​​the surface of the food G usually heats up significantly less or barely noticeably.

[0090] In step S5, a differential thermal image DW is created (see Fig.5), in which a temperature difference ("temperature lift") between the temperature value recorded with the second thermal image W2 and the temperature value recorded with the first thermal image W1 is calculated for each of the pixels.

[0091] In step S6, pixels are identified in the differential thermal image DW that have temperature values ​​that are higher by a predetermined temperature level than the set of all pixels in the differential thermal image DW. These pixels form a corresponding image pattern BM0 in the (x, y) matrix M, as shown in Fig. 6. Step S6 can, for example, be implemented such that in a first sub-step S6A the temperature differences are mapped to a value range [0; 1]. The lowest value of the temperature differences of all 768 pixels is set to the value zero, and the highest value of the temperature differences of all pixels is set to the value one. For example, if the lowest value were 3 °C and the highest value were 13 °C, the set of temperature differences {3; 5; 10; 13} would be mapped to the set {0; 0.2; 0.7; 1}.

[0092] In a subsequent sub-step S6B, those pixels that reach or exceed a predetermined temperature threshold, here, for example, 0.27, are identified as belonging to the edge of the food G. These pixels form the initial image pattern BM0 shown in Fig. 6 before the filling algorithm is executed.

[0093] The pixels identified as belonging to the food item G in Fig. 6 already clearly show the – here approximately circular – edge or outer contour of the food item G. The inner area of ​​the food item G is not filled, and there are still openings in the edge at points that have remained relatively cold. This cannot be resolved by a universally valid temperature threshold.

[0094] In step S7, a filling algorithm in the form of a starfill algorithm is applied to the initial image pattern BM0. In this algorithm, a pixel is assigned to the food item G if the number of directions radiating from this pixel, in which identified pixels are located, reaches or exceeds a minimum number. In the following, a starfill algorithm with a total of eight star-shaped directions is considered as an example.

[0095] This is indicated in Fig. 7A for a pixel located as a corner point in the matrix M and marked by a filled circle: the corner point has, to a limited extent, three possible directions radiating from it in a star shape, namely the x-direction, the y-direction and an (x, y) oblique direction. Here, for example, pixels identified as belonging to the food (filled) lie in the x-direction and in the y-direction, as indicated by the solid arrows. The distance of the identified pixels to the corner point is irrelevant in the present embodiment, but can generally be taken into account. In contrast, there is no identified pixel in the (x, y) oblique direction, as indicated by the dotted arrow. If the corner point is assigned to the food G if there is at least one identified pixel in at least two of the directions, this would be the case for the corner point shown.

[0096] Fig. 7B shows the starfill algorithm for an edge point in the matrix M, marked by a filled circle: the edge point has a limited number of five possible directions radiating out from it in a star shape, namely the x-direction, the (-x)-direction, the y-direction, the (x, y)-slant direction, and the (-x, y)-slant direction. Here, for example, image points identified as belonging to the food to be cooked lie in the (-x)-direction, the y-direction, and the (x, y)-slant direction, as indicated by the solid arrows. In contrast, there are no identified image points in the x-direction or the (-x, y)-slant direction, as indicated by the dotted arrow. If the edge point is assigned to the food G if there is at least one identified image point in at least three of the directions, this would be the case for the edge point shown.

[0097] Fig. 7C shows the starfill algorithm for an image point located as an interior point in the matrix M and marked by a filled circle: the interior point has all eight possible directions radiating from it in a star shape, namely the x-direction, the (-x)-direction, the y-direction, the (-y)-direction, the (x, y)-slant direction, the (-x, y)-slant direction, the (x, -y)-slant direction, and the (-x, -y)-slant direction. Here, for example, image points identified as belonging to the food to be cooked lie in the x-direction, the (-x)-direction, the y-direction, the (-y)-direction, the (-x, y)-slant direction, and the (x, -y)-slant direction, as indicated by the solid arrows. In the (x, y) oblique direction and in the (-x, -y) oblique direction, however, there are no identified pixels, as indicated by the dotted arrow.If the interior point is assigned to the food G if there is at least one identified image point in at least six of the directions, this would be the case for the interior point shown.

[0098] The Starfill algorithm can be applied to all pixels of the matrix M in a single pass. In particular, the Starfill algorithm can be applied to a specific pixel (e.g., first to the pixel x = 0, y = 0) and then applied to the next pixel, including the result for all previous pixels. Existing identified pixels remain unchanged, even if they subsequently fulfill the conditions of the Starfill algorithm. In particular, the Starfill algorithm can be performed row- and column-wise.

[0099] The Starfill algorithm is particularly suitable for filling edge contours with openings. Simple fill algorithms, such as "Floodfill," which captures areas of contiguous pixels of one color, are unsuitable for non-closed contours. The Starfill algorithm is particularly suitable for images or pixel matrices with relatively low resolution.

[0100] Fig.8 shows the (x, y) matrix M with the image pattern BM1 after a single application of the Starfill algorithm to the image pattern BMO from Fig.6, where starting from the image point x = 0, y = 0 the Starfill algorithm was first applied column by column in the x-direction and then row by row in the y-direction.

[0101] In a step S8, it is checked whether (a) a maximum number ("recursion number") of applications or passes of the Starfill algorithm has been reached or whether (b) no further changes occur during successive passes of the Starfill algorithm, depending on which condition occurs first.

[0102] If this is not the case ("N"), the process branches back to step S7; otherwise ("Y"), the process continues to step S9. Figure 9 shows the (x, y) matrix M with an image pattern BMn, for which the Starfill algorithm has been applied n times with n > 1.

[0103] In step S9, the method is terminated, wherein the pixels then identified as belonging to the food to be cooked can be used as a basis for the following method steps S10 of at least one method for operating the microwave cooking appliance 1, e.g. a method for detecting a desired cooking state of the food G. For this purpose, the identified pixels can serve, for example, as a "mask", which is further used in the following method(s), while all pixels that do not belong to the mask are disregarded or "masked out".

[0104] The described method allows for extremely precise masking of the food G. The method particularly avoids surfaces that heat up only slightly, such as the wide rim of the soup plate in the exemplary embodiment or other tableware items.

[0105] Of course, the present invention is not limited to the embodiment shown.

[0106] In general, "a", "an", etc., can be understood as a singular or a plural, in particular in the sense of "at least one" or "one or more", etc., as long as this is not explicitly excluded, e.g. by the expression "exactly one", etc. A numerical specification can also include exactly the specified number as well as a usual tolerance range, as long as this is not explicitly excluded.

[0107] List of reference symbols

[0108] 1 microwave oven

[0109] 2 cooking chamber

[0110] 3 Cooking chamber door

[0111] 4 food carriers

[0112] 5 plates

[0113] 6 microwave generator

[0114] 7 Microwave guide

[0115] 8 Microwave feed point

[0116] 9 Rotating antenna

[0117] 10 Ceiling of the cooking chamber

[0118] 11 Thermal imaging camera

[0119] 12 Optical digital camera

[0120] 13 Control device

[0121] BMO Initial Image Pattern

[0122] BM1 image pattern

[0123] BMn image pattern

[0124] DW differential thermal image

[0125] G Food to be cooked

[0126] M Matrix

[0127] S1-S10 process steps

[0128] T Temperature

[0129] W1 First thermal image

[0130] W2 Second thermal image x x-direction y y-direction

Claims

Method (S1-S9) for identifying food (G) in at least one pixel-based thermal image (W1, W2) taken from a cooking chamber (2) of a microwave cooking appliance (1), in which - microwave energy is radiated into the cooking chamber (2) (S3), - at least one thermal image (W1, W2) is taken (S2, S4), - pixels are identified (S6) which have temperature values ​​increased by a predetermined temperature level in relation to the set of all pixels, and - a filling algorithm is executed which assigns pixels to the food (G) that belong to an image area bordered by at least some of the identified pixels. Method (S1-S9) according to claim 1, in which - a first thermal image (W1) is taken before the microwave energy is radiated into the cooking chamber (2), - after the microwave energy has been radiated into the cooking chamber (2), a second thermal image (W2) is taken, - a differential thermal image (DW) is created, in which a temperature difference between the temperature value recorded with the second thermal image (W2) and the temperature value recorded with the first thermal image (W1) is calculated for each of the image points, and - the pixels in the differential thermal image (DW) are identified. The method (S1-S9) according to claim 2, wherein the temperature differences are mapped to a value range [0; 1]. The method (S1-S9) according to any one of the preceding claims, wherein those pixels are identified as belonging to the edge of the food (G) that reach or exceed a predetermined temperature threshold. The method (S1-S9) according to claims 3 and 4, wherein the temperature threshold lies in a range [0.2; 0.5].

6. Method (S1-S9) according to one of the preceding claims, in which a starfill algorithm is used as the filling algorithm, in which a pixel is assigned to the food (G) to be cooked if a number of directions radiating from this pixel in a star shape, in which identified pixels are located, reaches or exceeds a minimum number.

7. Method (S1-S9) according to claim 5, wherein at least one of the following boundary conditions of the Starfill algorithm is set: - a minimum number of two applies to corner points; - for edge points a minimum of three applies: - a minimum number of six applies to interior points.

8. Method (S1-S9) according to one of the preceding claims, in which the filling algorithm is executed several times until there is no further change in the identified pixels.

9. Method (S1-S9) according to one of the preceding claims, in which the pixels assigned to the food (G) are additionally converted to pixels of an optical digital camera (12) of the microwave cooking appliance (1).

10. Method (S1-S9) according to one of the preceding claims, wherein the identification of the position of the food (G) is carried out in an initial phase of a cooking process.

11. Method (S1-S9) according to claim 10, wherein during the irradiation of the microwave energy into the cooking chamber (2) for the purpose of carrying out the method, at least one microwave operating parameter which changes a mode image of the microwaves in the cooking chamber (2) is varied.

12. Method (S1-S10) for operating a microwave cooking appliance (1) according to one of the preceding claims, in which a position of the cooking product (G) in a heat- thermal image (W1, W2) is carried out using the method (S1-S9) according to one of the preceding claims, and a cooking process is controlled based on an evaluation of the pixels assigned to the food to be cooked (G).

13. Microwave cooking appliance (1), comprising a cooking chamber (2) to which microwaves can be applied, a thermal imaging camera (11) provided for recording thermal images (W1, W2) from the cooking chamber (2), and a control device (13), wherein the microwave cooking appliance (1) is set up to carry out the method according to one of the preceding claims (S1-S9; S1-S10).

Images