VISUAL DETECTION OF A GARGUT

DE502020009747D1Active Publication Date: 2024-11-28BOSCH SIEMENS HAUSGERATE GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502020009747
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-11
Filing Date
2020-03-06
Publication Date
2024-11-28
Estimated Expiration
2040-03-06

AI Technical Summary

Technical Problem

Existing techniques for optically scanning food items to determine their doneness or recognition during cooking are limited in their ability to provide meaningful analysis, lacking the precision and selectivity needed for effective cooking control.

Method used

A device and method utilizing multiple colored light sources, including red, green, blue, and infrared, to illuminate food items sequentially, allowing for improved recognition through the capture of distinct images under different color conditions, enabling better visualization of cooking properties and chemical changes.

Benefits of technology

This approach enhances the recognition and control of cooking processes by providing more selective and accurate optical information, allowing for better determination of food doneness and issuing precise instructions for cooking, even when multiple items are cooked simultaneously.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to the detection of a food item. In particular, the invention relates to the detection of the food item based on optical scanning.

[0002] Ingredients of a dish can be cooked on a stove or in an oven; this is then referred to as a cooked item. The cooked item can include meat, vegetables, or grains, for example. The cooked item can be visually scanned, for example, to determine its degree of doneness.

[0003] WO 2015 / 185608 A1 proposes illuminating the food using a predetermined light pattern, providing an image of the illuminated food, and determining a three-dimensional structure of the food. DE 10 2017 206 056 A1 describes methods for operating a cooking appliance in which a cooking chamber is irradiated with light in different wavelength ranges. DE 10 2013 102 295 A1 describes a method for adapting a cooking process in which food is captured by a camera at the beginning and after completion of the cooking process. DE 10 2010 039 371 A1 discloses a control device for a household appliance with a detector for detecting operating actions of an operator.

[0004] DE 10 2013 211 097 A1 concerns a refrigerator whose contents can be visually scanned at different times in order to determine the freshness of the contents.

[0005] However, known techniques only allow for a limited, meaningful analysis of a cooked item. One object underlying the present invention is to provide an improved technique for optically scanning a cooked item. The invention solves this problem by means of the subject matter of the independent claims. Subordinate claims specify preferred embodiments.

[0006] A device for recognizing a food item comprises a first light source configured to emit light of a first color onto the food item; a second light source configured to emit light of a second color onto the food item; a light sensor configured to scan images of the food item in light of the first and second colors; a control device, and a processing device. The control device is configured to provide a first image of the food item while the first light source is activated and the second light source is deactivated; and to provide a second image of the food item while the first light source is deactivated and the second light source is activated. The processing device is configured to recognize the food item based on the first and second images.

[0007] By illuminating the food to be cooked with colored light, i.e. light with a more limited light wavelength spectrum than white light, images can be created that make certain aspects of the food particularly easy to recognize. The food to be cooked can in particular comprise a dish or ingredients of a dish, and the described technology can be applied while the food is being cooked. For this purpose, the food can be held in a cooking vessel such as a pot, roasting pan, or frying pan. The cooking vessel can be on a stovetop or in an oven. The detection of the food can thus be improved, and a cooking process can be controlled more effectively based on the detection. The aforementioned light sensor is in particular designed as a camera.

[0008] The control device can be configured to carry out a method described herein in whole or in part. For this purpose, the control device can comprise a programmable microcomputer or microcontroller. The processing device can comprise a further microcomputer or microcontroller or be integrated with the control device. The method can be in the form of a computer program product with program code means. The computer program product can also be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the device, or vice versa.

[0009] The colors preferably have different distributions of their intensities across the wavelength of the respective emitted light. In one embodiment, a light color comprises only wavelengths within a predetermined range, whereby the ranges of different colors can be disjoint, i.e., non-overlapping. A slight overlap at intensities of, for example, approximately 10% or approximately 5% of the maximum intensity can also be disregarded.

[0010] Light that exhibits a perceptible color has a spectrum of light waves that essentially covers only a portion of the visible range. For example, light dominated by wavelengths of approximately 600–680 nm can be perceived as red; light that consists almost exclusively of wavelengths between 520 and 565 nm can be perceived as green; and light that is dominated by wavelengths in the interval between 460 and 480 nm can be perceived by the human eye as blue. A small portion of another wavelength range cannot significantly change the color impression.

[0011] The colors are preferably taken from the primary colors red, green, and blue. If more than two light sources are used, it is preferred that at least one pair of light sources have disjoint light wave spectra.

[0012] In particular, N light sources can be used, where N > 2, and the light sources are configured to emit light of different colors. Preferably, each light source is configured to emit light of a different color. Further preferably, the control device is configured to provide N images, each of which is provided by illuminating the food item using only one of the N light sources; and the processing device is configured to recognize the food item based on the N provided images.

[0013] This allows for more selective color images of the food to be provided, allowing for improved visual recognition of specific properties or chemical effects. The N images provided can better isolate optical information, making it easier or even better to identify the food.

[0014] In a particularly preferred embodiment, three light sources are provided, one each for emitting red, green, and blue light. The light sources can be comprised of a common illumination device. The illumination device can be configured to mix red, green, and blue light on the food to enable predetermined illumination. The illumination device can, in particular, be part of a projector configured to project a color image onto the food. According to the invention, such a projector can be controlled to sequentially illuminate the food with light of different colors. The projector and the camera can be parts of an interaction device configured to recognize a user's gesture in a projection area. A household appliance can be controlled based on the gesture.The household appliance can be connected to a cooking process for the food and can include, for example, a stove, an oven or an extractor.

[0015] At least one of the light sources can comprise a light-emitting diode. The light-emitting diode can be constructed as a III-V semiconductor and emit only light within a predetermined wavelength range or a predetermined intensity distribution. The emitted light can thus be more effectively concentrated on an area in which a predetermined feature of the food can be particularly well optically discriminated.

[0016] At least one of the light sources comprises a laser. The laser can be implemented, particularly cost-effectively, as a laser diode or laser LED. The laser light can cover a particularly narrow range of wavelengths. The light emitted by the laser can also be coherent, allowing for improved optical scanning of a specific feature of the food.

[0017] At least one of the colors can be in an infrared range invisible to the human eye. In particular, the color can be in the near-infrared range, which extends approximately from 0.75 to 1.4 µm. This allows additional optical information from the food to be isolated. In the case of the interaction device described above, a dedicated light source for infrared light, separate from the projector, can be provided.

[0018] The control device is preferably configured to provide a further image of the food to be cooked while several of the light sources are activated, wherein the processing device is configured to recognize the food to be cooked on the additional basis of the further image. The activated light sources together preferably produce essentially white light. The light sources can be activated simultaneously or individually one after the other while the camera exposes a light-sensitive area. By taking into account an image that is usually perceivable by a person, further processing can be further facilitated. In one embodiment, an initial evaluation can be carried out under illumination with light of several colors (in particular with white light), on the basis of which initial evaluation a scanning of the food to be cooked with monochromatic light can then be controlled.This initial evaluation can also detect multiple items that are spatially separated from one another on a surface and / or within a cooking device. Likewise, multiple items that are arranged separately within an open cooking vessel can be detected. The multiple detected items can be scanned together with a single-color light, with the final evaluation performed individually for each item. Accordingly, different instructions can be issued to a user for each of the detected items.

[0019] In a further embodiment, the light sources are each configured to project a colored image onto a surface on which the food to be cooked is located, wherein the control device is configured to illuminate only a predetermined area of ​​the surface one after the other using the light sources in order to provide the images. The projected images can be divided into individual image points (pixels), which are usually arranged in a matrix-like manner. By activating only predetermined image points, virtually any predetermined area can be illuminated. For example, the colored illumination can be limited to an area of ​​the food to be cooked that is of interest for analysis. A remaining area can be illuminated in a different way, for example in white, another predetermined color, or by projecting a predetermined image.A user can remain largely unaffected by color-changing lighting that is recognizable to him.

[0020] In particular, the food can be detected and the area determined based on the detected food. In other words, the food can be optically scanned and detected, and based on the detection result, an area can be determined, which is then sequentially scanned using different monochrome lights. The first optical scan can be performed, for example, using white light or using a depth camera configured to provide a three-dimensional image of the food. The three-dimensional image typically initially contains no color information.

[0021] According to a second aspect of the invention, a method for recognizing a cooking item comprises steps of activating a first light source configured to emit light of a first color onto the cooking item; deactivating a second light source configured to emit light of a second color onto the cooking item; providing a first image of the cooking item in light of the first color; deactivating the first light source and activating the second light source; providing a second image of the cooking item in light of the second color; and recognizing the cooking item on the basis of the first and second images.

[0022] The method is carried out using a device described herein. In one embodiment, a conventional interaction device, for example, for a cooking area, can carry out the method according to the invention. The interaction device can thus be cost-effectively expanded or retrofitted according to the invention.

[0023] In another embodiment, two disjoint areas of the food are simultaneously illuminated with light of different colors, and two partial images of the areas are provided. The areas can be moved relative to the food, so that an observer can see the impression of a series of differently colored light bars. The optical scanning process can thus be displayed more effectively to a human observer.

[0024] Based on the images, the cooking status of the food can be determined. In particular, coloring or bleaching of the food can be sensed and the degree of cooking determined on this basis. The coloring can be the result of an enzymatic reaction, which may be part of the cooking process. The color change can occur, for example, due to the Maillard reaction when roasting or frying meat, or due to the caramelization of sugar. The color change can also indicate an enzymatic reaction, for example if the food contains animal fat, for example in the form of a dairy product such as cheese. Corresponding actions derived from this in the form of instructions to a user and / or control commands to a cooking appliance or extractor hood can be output by the control device.

[0025] The invention will now be described in more detail with reference to the accompanying figures, in which: Figure 1 shows an exemplary system with an interaction device; Figure 2 shows a system in a further embodiment; Figure 3 shows a flowchart of an exemplary method for detecting a cooking item; Figure 4 shows an exemplary representation of a cooking item from the perspective of an interaction device; and Figure 5 shows different intensity distributions of exemplary colored light.

[0026] Figure 1 shows a system 100 with a cooking device 105 and a device 110. The cooking device 105 is preferably configured to provide heat to cook a food item 115 and can, for example, comprise a hob or a stove. The device 110 is arranged, for example, above a surface 120 into which the cooking device 105 can be embedded, and includes a camera 125, a plurality of light sources 130, a control device 135, and a processing device 140.

[0027] The light sources 130 are each configured to illuminate the food 115 being cooked with light of a different color. For example, a first light source 130.1 illuminates red, a second light source 130.2 illuminates green, a third light source 130.3 illuminates blue, and a fourth light source 130.4 illuminates infrared. The technology presented herein can also be implemented using other colors and with two or more light sources 130. Two or more of the light sources 130 can be comprised by a projector 145 configured to project light—in particular, a colored image—onto the surface 120. For this purpose, the projector 145 can comprise a light valve 150, which, for example, comprises a matrix of micromirrors to direct individual pixels of light onto the surface 120. If a pixel is illuminated sufficiently quickly one after the other using differently colored light, the colors can blend for a viewer.In another embodiment, a light source for white light is provided, from which different colors are filtered one after the other using a color wheel. In yet another embodiment, a light valve 150, for example in the form of a liquid crystal element, is assigned to each of several light sources 130, so that light of different colors is continuously emitted, which can mix on the surface 120.

[0028] The camera 125 is preferably configured to scan an image in an area that is at least partially illuminated by the light sources 130. In one embodiment, the camera 125 is configured to scan an object located in an area between the device 110 and the surface 120. The object can comprise a user's hand, for example, and a position or gesture of the hand can be optically detected and determined by one of the devices 135, 140. The hand can be in the context of a projection projected onto the surface 120 by the projector 145. The position or gesture can be evaluated to control a household appliance, in particular the cooking device 105. Thus, the projector 145 and the camera 125 together can form an interaction device 155.

[0029] In a further embodiment, a depth camera 160 can be included in the interaction device 155. The depth camera 160 is configured to optically scan an object between it and the surface 120 and can provide distance information to the object. The depth camera 160 can operate based on stereoscopic scanning. A built-in light source 130 can be configured to emit structured light. The light source 130 can operate in particular in the infrared range, so that it can also be used as light source 130.4 for the device 110. In another embodiment, the depth camera 160 operates based on distance measurement based on the speed of propagation of light (TOF: time of flight).

[0030] For the purposes of the interaction device 155, an image provided by the camera 125 can be continuously recorded while the food 115 is illuminated with light of two or more colors. If the light sources 130 operate sequentially, so that only one of the light sources 130 is active at a time, a shutter speed of the camera 125 can be one cycle of all colors or longer. The provided image contains color information and can be further processed in this form.

[0031] For the purposes of the present invention, the light sources 130 are controlled such that the food 115 is illuminated successively with light of different colors. During the illumination with single-color light, an image of the food 115 can be created by means of the camera 125, so that for N light sources 130 (in the illustration of Figure 1is N = 4) the same number of images can be created, each scanned under differently colored lighting. Optionally, a further image can be created using a mixture of several light colors. The mixture can, as is usual for the purposes of the interaction device 155, add up to white light. In a further embodiment, on the basis of an image created with multi-colored (in particular white) illumination of the food item 115, an initial processing can take place, which can lead to the determination of an illumination parameter for the sequential single-color illumination. In this case, several different food items 115 can be detected and an independent evaluation of the degree of doneness of the individual food items 115 can be initiated by the processing device 140. The determined degrees of doneness of the different food items 115 can lead to correspondingly different instructions for a user.

[0032] For scanning an image, the duration for which one of the light sources 130 is activated (and the other light sources 130 are deactivated) can be at least as long as the exposure or shutter time of the camera 125. In the case of a color wheel, its rotation speed can be controlled accordingly to achieve the same effect. The shutter speed of the camera 125 can be reduced for monochrome scanning compared to normal operation to meet this condition.

[0033] The images created under different light colors can then be processed by the processing device 140, wherein, in particular, a recognition of the food 115 can be carried out. The recognition can, in particular, comprise a determination of the food 115 or at least one of its components. Furthermore, a color of the food 115 in at least one area or a visually perceptible piece of information can be determined, which can be important for controlling a cooking process of the food 115. The determined information can be provided externally via an interface 165. Using the provided information, for example, a user can be informed about a status or progress of the food 115, or the cooking device 105 can be controlled depending on the information.This process can be carried out in parallel for several different cooking items 115 that are positioned on the cooking device 105 and / or the surface 120.

[0034] Figure 2shows a system 100 in a further embodiment. Here, the interaction device 155 is, for example, attached to an extractor hood 205 or integrated with it. An additional lighting device 210 is preferably configured to emit white light and can illuminate a larger area than the projector 145. Purely by way of example, no depth camera 160 is provided here. The projector 145 and the camera 125 have optical axes that enclose substantially equal angles with a vertical. In other embodiments, the optical axes can also coincide or enclose other angles with the vertical. In general, it is preferred that the optical axes of the projector 145 and the camera 125 enclose the smallest possible angle with one another.

[0035] Figure 3shows a flowchart of an exemplary method for detecting a cooking product 115. The method 300 can be carried out in particular using the device 110. To explain the method 300, N = 3 differently colored light sources 130 are assumed as an example.

[0036] In a step 305, the surface 120 can be scanned. This can be done, for example, using the depth camera 160 or using the camera 125 while illuminating the surface 120 with white light. In a step 310, based on the scanning, it can be determined where on the surface a cooking item 115 is located or which area of ​​the cooking item 115 is of particular interest for optical scanning. In this step, several different cooking items 115 can also be detected, which are then optically scanned together and subsequently evaluated separately by the processing device 140.

[0037] In a step 315, the food 115 is illuminated successively with light from only one of the light sources 130, and simultaneously an image is scanned using the camera 125. This step 315 is typically repeated as many times as the number of differently colored, individually activatable light sources 130 provided. In the present example, in a first run of step 315, for example, the first light source 130.1 is activated and the light sources 130.2 and 130.3 are deactivated, and a first image is scanned under red illumination. In a second run, the first and third light sources 130.1, 130.3 are deactivated and the second light source 130.2 is activated, and a second image is scanned under green illumination. In a third run, the first and second light sources 130.1, 130.2 are deactivated and the third light source 130.3 is activated, and a third image is scanned under blue illumination.The illumination can be limited to the area determined in step 310. In a surrounding area, the illumination can be switched off, or a portion of a predetermined display can be output, which is preferably displayed before and / or after the entire illuminated area.

[0038] In a step 320, the food 115 can be recognized based on the individual images created. Optionally, one or more images taken under illumination from multiple colored or one white light source 130, 210 can be included. It is preferred that the images used for recognition were acquired close to one another in time to avoid decorrelation of their image content. For example, all images can originate from a time range of no more than one second or less.

[0039] In a step 325, one of the properties of the detected food 115 can be determined. For this purpose, color information at one or more locations on the food 115 can be evaluated. The color information can be related to the type of detected food 115. For example, the color brown for a vegetable can lead to a different evaluation than for meat, dough, or fruit.

[0040] In a step 330, the determined information can be output. Furthermore, a household appliance can be controlled, which is particularly related to the cooking process of the food 115. This can include, for example, the cooking device 105 or the extractor hood 205. For example, the cooking device 105 can be throttled in its heat supply when a desired cooking point is reached, or the extractor hood 205 can be set to a higher level if boiling liquid, rising vapors, or smoke is detected in the area of ​​the food 115.

[0041] Figure 4shows an exemplary representation of a cooking item 115 from the perspective of an interaction device 155. The cooking item 115 is located, for example, in a cooking container 405, in this case in a pan. A region 410 is shown on the cooking item 115, which region is of interest, for example, for assessing the cooking point or cooking process of the cooking item 115. The region 410 can be selectively illuminated one after the other using different colored light, while an image is scanned in each case. The region 410 can also be divided into sub-regions 410.1, 410.2, and 410.3, and the sub-regions 410.1, 410.2, and 410.3 can be illuminated simultaneously with different colors, while an image is captured in each case. The number of sub-regions 410.1, 410.2, and 410.3 can correspond to the number N of light sources 130 or can be selected independently.The cooking container 405 can also contain several different cooking items, which are recognized accordingly and evaluated individually by the processing device 140.

[0042] In yet another embodiment, for example, a bar of colored light can be moved across area 410 while an image is being captured. After passing through N bars, all the necessary information can be captured. Different areas of the recorded images can be combined to create three single-color images, which can be evaluated in the manner described.

[0043] Figure 5shows different intensity distributions of exemplary colored light. In the horizontal direction, an approximate wavelength in nm is indicated, and in the vertical direction, an intensity in percent. A first intensity distribution 505 corresponds to red light, a second intensity distribution 510 to green light, a third intensity distribution 515 to blue light, and a fourth intensity distribution 520 to infrared light. The intensity distributions have different range limits. The range limit can be assumed to be a wavelength beyond which the intensity of the intensity distribution 505-520 lies below a predetermined intensity value, for example, approximately 10%, preferably approximately 5%, more preferably approximately 2% or even less. The intensity distributions 505-520 shown are disjoint in that they do not cover any overlapping ranges in pairs. Reference symbol

[0044] 100System 105Cooking device 110Device 115Food 120Surface 125Camera / Light sensor 130Light source 135Control device 140Processing device 145Projector 150Light valve 155Interaction device 160Depth camera 165Interface 205Extractor hood 210Lighting system 300Procedure 305Scanning the work surface 310Determining the area 315Illuminating the area in color, capturing an image 320Detecting the food 325Determining the property 330Controlling the cooking appliance / displaying information 405Cooking container 410Area 410.1First sub-area 410.2Second sub-area 410.3Third sub-area 505First intensity distribution (Red) 510Second intensity distribution (Green) 515Third intensity distribution (Blue) 520Fourth intensity distribution (Infrared)

Claims

1. Apparatus (110) for recognition of food to be cooked (115), wherein the apparatus (110) comprises the following: - a first light source (130.1), which is configured to emit light of a first colour onto the food to be cooked (115); - a second light source (130.2), which is configured to emit light of a second colour onto the food to be cooked (115); - a light sensor (125), which is configured to scan images of the food to be cooked (115) in the light of the first and of the second colour; - a control apparatus (135), which is configured to: ∘ provide a first image of the food to be cooked (115), with the first light source (130.1) being activated and the second light source (130.2) being deactivated; ∘ provide a second image of the food to be cooked (115), with the first light source (130.1) being deactivated and the second light source (130.2) being activated - a processing apparatus (140), which is configured to recognise the food to be cooked (115) based on the first and the second image; characterised in that one of the light sources (130) comprises a laser.

2. Apparatus (110) according to claim 1, wherein the colours have different intensity distributions (505-520) over the wavelengths of the light emitted in each case.

3. Apparatus (110) according to claim 1 or 2, wherein N>2 light sources (130) are provided which are configured to emit light of different colours; the control apparatus (135) is configured to provide N images, which are provided in each case when the food to be cooked (115) is illuminated by means of just one of the N light sources (130); and the processing apparatus (140) is configured to recognise the food to be cooked (115) based on the N provided images.

4. Apparatus (100) according to claim 3, wherein three light sources (130) are provided, of which one is provided to emit red, green and blue light in each case.

5. Apparatus (100) according to one of the preceding claims, wherein at least one of the colours lies in an infrared range which is not visible to the human eye.

6. Apparatus (100) according to one of the preceding claims, wherein the control apparatus (135) is configured to provide a further image of the food to be cooked (115) when a number of the light sources (130) are activated, and the processing apparatus (140) is configured to recognise the food to be cooked (115) based additionally on the further image.

7. Apparatus (110) according to one of the preceding claims, wherein the light sources (130) are each configured to project a coloured image onto a surface (120) on which the food to be cooked (115) is situated, and the control apparatus (135) is configured to successively illuminate only a prespecified region (410) of the surface (120) by means of the light sources (130) in order to provide the images.

8. Apparatus (110) according to claim 7, wherein the food to be cooked (115) is recognised and the region (410) is specified depending on the food to be cooked (115) recognised.

9. Method (300) for recognition of food to be cooked (115), wherein the method (300) is carried out by means of an apparatus according to one of claims 1 to 8 and comprises the following steps: - activate (315) a first light source (130.1), which is configured to emit light of a first colour onto the food to be cooked (115); - deactivate (315) a second light source (130.2), which is configured to emit light of a second colour onto the food to be cooked (115); - provide (315) a first image of the food to be cooked (115) in light of the first colour; - deactivate (315) the first light source (130.1) and activate the second light source (130.2); - provide (315) a second image of the food to be cooked (115) in light of the second colour; and - recognise (320) the food to be cooked (115) based on the first and the second image.

10. Method (300) according to claim 9, wherein two disjointed regions (410.1, 410.2, 410.3) of the food to be cooked (115) are simultaneously illuminated with light of different colours and two partial images of the regions (410.1, 410.2, 410.3) are provided.

11. Method (300) according to one of claims 9 or 10, wherein a cooking state of the food to be cooked (115) is determined based on the images.