Method and device for detecting an interior of a cleaning device and cleaning device

Abstract

A method for detecting an interior (105) of a cleaning device (100) comprises a step of outputting a first illumination signal to an interface to a first illumination device (115) to bring the first illumination device (115) into a first state, a step of reading a first image signal via an interface to a camera (110), wherein the first image signal represents a first image of the interior (105) captured by the camera (110) while the first illumination device (115) is in the first state, a step of outputting a second illumination signal to the interface to the first illumination device (115) to bring the first illumination device (115) into a second state, and a step of reading a second image signal via the interface to the camera (110), wherein the image signal represents a first image of the interior (105) captured by the camera (110).while the first lighting device (115) is in the second state, the recorded second image represents a step of determining a difference image signal using the first image signal and the second image signal, wherein the difference image signal represents depth information of the interior (105).

Description

[0001] The invention relates to a method and a device for detecting an interior of a cleaning device and to a cleaning device.

[0002] 102021200715 A1 discloses a system with a dishwasher.

[0003] The invention has for its object to provide an improved method and an improved device for detecting an interior of a cleaning device as well as an improved cleaning device.

[0004] According to the invention, this object is achieved by a method and a device for detecting the interior of a cleaning device and by a cleaning device having the features of the main claims. Advantageous embodiments and further developments of the invention are set forth in the following subclaims.

[0005] The advantages achievable with the invention consist in particular in the fact that a method is created that can enable rapid and reliable detection of an interior of the cleaning device.

[0006] The invention relates to a method for detecting an interior of a cleaning device.

[0007] The cleaning device has a camera for capturing the interior of the cleaning device and at least one first illumination device and preferably a second illumination device for illuminating the interior. The cleaning device also has a control device. The method comprises a step of outputting a first illumination signal to an interface to the first illumination device in order to place the first illumination device into a first state. The method also comprises a step of reading in a first image signal via an interface to the camera, wherein the first image signal represents a first image captured by the camera while the first illumination device is in the first state.The method further comprises a step of outputting a second illumination signal to the interface to the first illumination device in order to place the first illumination device into a second state. Furthermore, the method comprises a step of reading in a second image signal via the interface to the camera, wherein the image signal represents a second image captured by the camera while the first illumination device is in the second state. The method additionally comprises a step of determining a difference image signal using the first image signal and the second image signal, wherein the difference image signal represents depth information of the interior.

[0008] The cleaning device can be designed as a dishwasher for cleaning dishes.

[0009] The camera can be arranged in a door of the cleaning device, and the lighting devices can be arranged in the interior of the cleaning device, for example on walls of the interior. The lighting devices can capture the interior from different viewing directions. For example, the lighting devices can be arranged on different walls or spaced apart from one another on a wall. In the first state, the first lighting device can emit light with a first light intensity and / or first light color. In the second state, the first lighting device can emit light with a second light intensity and / or second light color that differs from the first state, or can be switched off. A difference image can be created from the images recorded by the camera. A difference image can be created, for example, by pixel-by-pixel subtraction of brightness values ​​of the images.The difference image can provide information about the degree of dryness, the degree of load, the presence of a wash shadow, a spray arm blockage, and / or residual water in the dishes. The approach presented here can enable improved detection algorithms for a dishwasher camera using controllable LEDs.

[0010] The method may include a step of determining a parameter signal for controlling a program sequence of the cleaning appliance. The parameter signal may be determined using the differential image signal. The parameter signal may represent a parameter of the interior, such as a degree of dryness, a load level, a rinse shadow, a spray arm blockage, and / or residual water. A degree of dryness may, for example, define the presence of residual moisture. The load level may, for example, define the number and distribution of dishes.

[0011] The method may include a step of preprocessing the first image signal and the second image signal, wherein the preprocessing step may be performed before the determining step. In the preprocessing step, the image signals may be preprocessed using a smoothing operator and / or a 2D median filter in order to reliably determine the depth information in the determining step.

[0012] The method may comprise a step of outputting a third illumination signal to an interface to the second illumination device in order to place the second illumination device in the first state, and a step of reading in a third image signal via the interface to the camera. The third image signal may represent a third image recorded by the camera while the second illumination device is in the first state. In the determining step, the difference image signal may be determined using the first image signal, the second image signal, and the third image signal. The second illumination device may be placed in the first state while the first illumination device is also in the first state or in the second state. In the first state, the second illumination device may emit light with a first light intensity and / or first light color.The first light intensity and first light color of the first and second lighting devices can correspond to or differ from each other in the respective first state.

[0013] The method may comprise a step of outputting a fourth illumination signal to an interface to the second illumination device in order to place the second illumination device in the second state, and a step of reading in a fourth image signal via the interface to the camera. The fourth image signal may represent a fourth image recorded by the camera while the second illumination device is in the second state. In the determining step, the difference image signal may be determined using the first image signal, the second image signal, the third image signal, and the fourth image signal. The second illumination device may be placed in the second state while the first illumination device is also in the second state or in the first state.In the second state, the second lighting device can emit light with a second light intensity and / or second light color that differs from the first state, or it can be switched off. The second light intensity and second light color of the first and second lighting devices can be the same or different in the respective second state, provided they are not deactivated in the second state.

[0014] In the dispensing step, the first state may represent a state in which the first lighting device and / or the second lighting device is activated to illuminate the interior of the cleaning device. In the first state, the interior may be fully illuminated. The lighting devices may be activated simultaneously or sequentially.

[0015] In the outputting step, the second state may represent a state in which the first lighting device and / or the second lighting device is deactivated or dimmed. The lighting devices may be switched to the second state simultaneously or sequentially.

[0016] The method may include a step of capturing a reference image signal. The reference image signal may represent a reference image of an empty interior of the cleaning appliance captured by the camera. In the determining step, the difference image signal may be determined using the reference image signal. The reference image may be captured with the lighting device activated. Dishware may then be placed in the interior. Reading in the reference image signal may be advantageous for geometrically complex dishes in order to achieve better separation between shadows of the dishes and load baskets.

[0017] Thus, according to one of the aforementioned embodiments, an improved method for detecting the interior of a cleaning device is created. The detection of the interior may, for example, include: The detection of one or more functional components of the cleaning device arranged in the interior, such as the spray arm of the cleaning device designed as a dishwasher, which is rotatably mounted in the interior; and / or the detection of items to be cleaned arranged in the interior, in particular the size, type and / or arrangement or orientation of individual items to be cleaned and, alternatively or additionally, their degree of soiling and / or dryness.

[0018] A control device for detecting an interior of a cleaning device with a camera for detecting an interior of the cleaning device and a first lighting device and a second lighting device for illuminating the interior of the cleaning device can carry out and / or control the steps of the said method in corresponding units.

[0019] The control device can be designed to read in input signals and to determine and provide output signals using the input signals. An input signal can, for example, represent a sensor signal that can be read in via an input interface of the control device. An output signal can represent a control signal or a data signal that can be provided at an output interface of the device. The control device can be designed to determine the output signals using a processing rule implemented in hardware or software. For example, the control device can comprise a logic circuit, an integrated circuit, or a software module and can, for example, be implemented as a discrete component or be comprised of a discrete component.

[0020] A corresponding cleaning device comprises a camera for capturing an interior of the cleaning device, a first lighting device and a second lighting device for illuminating the interior of the cleaning device, as well as an embodiment of a control device mentioned herein. The lighting devices can be configured as LED bars.

[0021] The cleaning device can have a door to close the interior of the cleaning device. The camera can be mounted on the door. This allows the entire interior to be captured by the camera.

[0022] Even if the described approach is described using a household appliance, the method and / or the device described here can be used accordingly in connection with a commercial or professional device, for example a medical device, such as a cleaning or disinfection device, a small-scale sterilizer, a large-scale disinfector or a container washing system.

[0023] An embodiment of the invention is shown purely schematically in the drawings and is described in more detail below. It shows Figure 1 shows an embodiment of a cleaning device; Figure 2 shows an embodiment of a cleaning device; Figure 3 shows an embodiment of a cleaning device; Figure 4 shows an embodiment of a cleaning device; Figure 5 shows an embodiment of a cleaning device; Figure 6 shows an embodiment of a cleaning device; Figure 7 shows an embodiment of a cleaning device; Figure 8 shows an embodiment of a cleaning device; Figure 9 shows an embodiment of a cleaning device; Figure 10 shows an embodiment of a cleaning device; Figure 11 shows an embodiment of a cleaning device; Figure 12 shows an embodiment of a cleaning device; Figure 13 shows an embodiment of a cleaning device;Figure 14 shows an embodiment of a cleaning device; Figure 15 shows an embodiment of a cleaning device; Figure 16 shows an embodiment of a cleaning device; Figure 17 shows an embodiment of a cleaning device; Figure 18 shows an embodiment of a cleaning device; Figure 19 shows an embodiment of a cleaning device; Figure 20 shows an embodiment of a cleaning device; Figure 21 shows an embodiment of a cleaning device; Figure 22 shows an embodiment of a cleaning device; Figure 23 shows an embodiment of a cleaning device; Figure 24 shows an embodiment of a cleaning device; Figure 25 shows an embodiment of a cleaning device; Figure 26 shows an embodiment of a cleaning device;Figure 27 shows an illustration of an interior of an exemplary embodiment of a cleaning device; Figure 28 shows an illustration of an interior of an exemplary embodiment of a cleaning device; Figure 29 shows a flowchart of an exemplary embodiment of a method for detecting an interior of a cleaning device; Figure 30 shows a flowchart of an exemplary embodiment of a method for detecting an interior of a cleaning device; and Figure 31 shows a block diagram of an exemplary embodiment of a control device.

[0024] Figure 1 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is designed merely as an example as a dishwasher.

[0025] The cleaning appliance 100 has an interior space 105 configured to accommodate, by way of example only, a first loading basket 140, a second loading basket 150, and a cutlery drawer 130. The loading baskets 140, 150 and the cutlery drawer 130 are arranged one above the other, for example, and configured to accommodate items to be cleaned.

[0026] In addition, the cleaning device 100 has, for example, three spray arms 155, 165, 175. A first spray arm 155 is arranged, for example, on a ceiling of the interior 105 above the cutlery drawer 130, and a second spray arm 165 is arranged on a floor of the interior 105 below the second loading basket 150. A third spray arm 175 is arranged, for example, between the first loading basket 140 and the second loading basket 150. The spray arms 155, 165, 175 are designed to spray a cleaning fluid into the interior 105.

[0027] The cleaning appliance 100 is shown, for example, in a loaded state, with cutlery being arranged in the cutlery drawer 155, drinking vessels and bowls as well as saucers being arranged in the first loading basket 140 merely as an example, and large items to be washed, such as plates, a pan and a pot, being arranged in the second loading basket 150.

[0028] The cleaning device 100 has a camera 110 and a plurality of lighting devices 115, 125, 135, 145. The camera 110 is arranged on a door of the cleaning device 100, so that the camera 110 captures the interior 105 of the cleaning device 100. The lighting devices 115, 125, 135, 145 are distributed within the interior 105. For example only, two lighting devices 115, 125 are arranged above the cutlery drawer 130 on the ceiling of the interior 105, and two lighting devices 135, 145 are arranged between the first loading basket 140 and the third spray arm 175 on the side walls of the interior 105. The lighting devices 115, 125, 135, 145 are designed to illuminate the interior 105 of the cleaning device 100.

[0029] The cleaning device 100 additionally has a control device 120, which is arranged adjacent to the camera 110 on the door of the cleaning device 100, merely by way of example.

[0030] The camera 110 is configured to capture the interior 105 of the cleaning appliance 100 under different lighting conditions. For example only, all lighting devices 115, 125, 135, 145 are activated while the camera 110 captures a first image and deactivated when the camera 110 captures a second image. Alternatively, one of the lighting devices 115, 125, 135, 145 is activated or deactivated during image capture. Thus, a property of the illumination of the interior 105 generated by the lighting devices 115, 125, 135, 145 is changed between the captures of the images. A difference image is created from the captured images, which is evaluated according to one exemplary embodiment to draw conclusions about a degree of drying, a load level, a wash shadow, a spray arm blockage, and / or residual water in the washware.

[0031] For example, the spray arms 155, 165, 175 do not rotate freely if the items to be washed, which can also be referred to as dishes, are poorly arranged in the loading baskets 140, 150 or the cutlery drawer 130, see for example Figure 6 The approach presented here enables the provision of 3D information based on the difference image to detect a possible collision between spray arms 155, 165, 175 and the washware.

[0032] With the help of camera 110, which can also be referred to as a dishwashing camera, as many abstract process parameters as possible are derived or situations or misalignments are detected. The approach presented here demonstrates a method for advantageously controlling the lighting devices 115, 125, 135, and 145, so that more information is available than with constant lighting. These process parameters include, for example, the load level, a dishwashing shadow, an upward-facing bowl without a drain (e.g., a cereal bowl), or spray arm blockages.

[0033] If the camera 110 is installed in the cleaning device 100, it is obvious to capture an image at a defined exposure. The approach presented here makes it possible to determine 3D information from this and thus generate information that ultimately targets the target parameters such as the degree of drying or spray arm blockage.

[0034] The technical implementation is based on a commercially available cleaning device 100, in which the camera 110 looks into the interior 105. The interior 105 is equipped with at least two lighting units 115, 125, which are individually controllable and installed at various locations within the cleaning device 100. The camera 110 and the lighting units 115, 125, 135, 145 can be controlled or read by a central processing unit. The connection is established via direct wiring, radio communication, or a device bus system and multiple circuit boards.

[0035] However, instead of just one image, multiple images are now captured for the scenes. The advantage of this is that the reflections, refractions, and shadows are different for each lighting device (115, 125, 135, 145). If the images are then analyzed using an analytical approach or a neural network, the target parameters can be derived more effectively than from a single image.

[0036] Now, there are various applications that utilize this setup: For blockage detection, a check is carried out before the wash cycle starts to determine whether at least one of the spray arms 155, 165, or 175 is blocked by washware. Potential blockage objects are detected via the shadow of the spray arm 155, 165, or 175, see, for example, Figure 7 to Figure 11There are various approaches to improving or facilitating detection using controllable light sources: If the spray arm 175, camera 110, and lighting devices 135, 145 are almost in line, this can be used to determine whether a blockage is present. Simply put, this creates a light barrier: If the lighting device 135, 145, which is positioned just below the spray arm 175, is still visible in the image, then no collision can be assumed. To avoid misinterpreting reflections, the lighting devices 115, 125, 135, 145 are switched on one after the other and analyzed image by image. The brightest spot in the image corresponds to the expected position, see, for example, Figure 12 to Figure 13 .

[0037] Alternatively, the shadow on the spray arm 175 can also be examined. If a part protrudes into the rotation area of ​​the spray arm 175, a characteristic shadow will appear on the spray arm 175. This phenomenon is particularly evident with the lighting devices 135, 145, which are located at the level of the spray arm 175. If an unexpected shadow is visible on the spray arm 175, a spray arm blockage can be diagnosed, see, for example, Figure 6 to Figure 11 .

[0038] Both variants can be evaluated using classical image processing, in particular difference images and threshold comparisons, or neural networks, for example trained for abnormality detection.

[0039] In addition, the approach presented here makes it possible to determine depth information from shadows for the detection of concave shapes. For many pieces of tableware, it is difficult to determine from an image whether they have a concave surface towards the top. Many pieces of tableware are patterned, so dark circles / ellipses alone can lead to misjudgments. However, if the lighting is varied, a different situation arises for each lighting situation. If a concave shape is present, in which dishwater in particular can remain, the shadow begins behind the edge and is no longer present on the opposite side. For a convex shape, this behavior is the opposite, see, for example, Figure 23 to Figure 26 .

[0040] Advantageously, the approach presented here makes it possible to exploit the shadows cast at different exposure settings to generate depth information. In the case of the cleaning device 100, this depth information is relevant in a somewhat more abstract form: "Can the rinse water flow unhindered along the surfaces?"

[0041] The cleaning device 100 comprises the interior 105, which can also be referred to as the interior rinsing chamber, the camera 110 looking into the interior 105, the control device 120, which can also be referred to as the processing computer, and several individually controllable lighting devices 115, 125, 135, 145. The lighting devices 115, 125, 135, 145 can be LED bars and / or brilliant light LEDs, which provide conventional interior lighting for the user. Thus, other components from the system that were originally installed in the cleaning device 100 for other purposes can be "shared." However, additional lighting devices can be used, or existing lighting devices can be split, for example, two partial lights instead of a single light.

[0042] The approach presented here enables the detection of clean dishes. For this purpose, the gloss of the dishes, or particularly glasses, is examined. When dirty, the dishes tend to appear dull or milky. After cleaning, an increased gloss can be detected. This behavior can also be captured algorithmically. For example, the reflection on a glass can be switched on and off by switching the lighting devices 115, 125, 135, and 145 on and off. If a reflection point is examined, a before-and-after comparison of the sharpness or contrast can be performed, for example, using MTF of the reflection. A similar process works with white porcelain: The dirty plate hardly reflects because, for example, spinach covers the plate. After cleaning, a reflection is visible on the shiny dishes, and the cleaning performance is evident, see, for example, Figure 27 to Figure 28 .

[0043] In other words, a specularity analysis is performed. This specularity analysis depends particularly on the reflections from the illumination devices 115, 125, 135, and 145. To make it easier to distinguish between all the multiple reflections, the illumination sources are switched on one after the other and several images are taken.

[0044] The before-after comparison is elementary, which means that there is a memory in the system that stores the before images, which are compared with the current images in the analysis phase.

[0045] However, there's also the possibility that only the viewing window of camera 110 is dirty. To account for this, the reflections on the side walls, which hardly become dirty during loading, are examined. This allows a reference value to be determined.

[0046] The approach presented here makes it possible to determine the degree of drying. When the wash liquor dries, droplets hang from the load basket 140, 150. These droplets can presumably be easily identified using difference images, neural networks, and pattern matching. Because the droplets are round, small, and transparent, they exhibit completely different refraction and diffraction behavior than the wash liquor. The edges of the droplets, in particular, are clearly visible due to reflections and diffraction. If the drying behavior is now analyzed based on these hanging droplets, or other droplets, for example, on the spray arm 175, a value for the degree of drying can be established. It may still be difficult to detect the droplets. The controllable illumination devices 115, 125, 135, 145 are helpful here. Because the droplets are so small and round, an illumination device 115, 125, 135, 145 can often be seen in the droplets via diffraction.By switching the lighting devices 115, 125, 135, and 145, the chances of distinguishing the transparent drops from other image objects and / or blending are increased. Especially the drops that are in the line of sight to a lighting device 115, 125, 135, and 145 are not recognizable without switchable lighting devices 115, 125, 135, and 145.

[0047] The approach presented here makes it possible to mask out extraneous light. Instead of just taking an image with the door slightly open, at least one image is taken with the lighting devices 115, 125, 135, 145 switched on and one image with the lighting devices 115, 125, 135, 145 switched off. A processing unit then calculates which portion of the light came from which light source and, using corresponding difference images, creates a synthetic image that appears as if no extraneous light penetrated the door gap. This takes advantage of the fact that the optoelectronic transfer function, or OETF for short, is deterministic, meaning that it is possible to model which portion of the light belongs to which light source. To keep the process as simple as possible, it is advantageous if the images are taken with the same exposure settings.It is also possible, however, to use multiple images with different exposure times, which essentially corresponds to a high dynamic range image, or HDR for short. Especially in the front areas, the lighting devices 115, 125, 135, and 145 contribute less to the brightness than the ambient light. In the rear corners, the situation is exactly the opposite. In order to achieve sufficient resolution and sufficiently low noise in these areas, it is advantageous to use multiple exposure times. Modeling the transfer functions is considerably more complex in this case, but it is not an insoluble mathematical problem. The outer ends of the value scale are important: pixels that are almost fully on are tilted or oversaturated. Pixels that are almost fully off can be lost in the noise, black level, or can be severely distorted.Especially for the second effect, it makes sense to combine the method with spatial filters such as the median filter to compensate for noise-related outliers.

[0048] In summary, the approach presented here improves the detection of spray arm blockages. Furthermore, proper circulation of any rinse water is enabled, ensuring that no rinse water remains in the dishes. This is achieved by detecting upwardly directed, basin-shaped objects. Furthermore, clean and dirty dishes can be identified using a camera image with suitable exposure control. Additionally, drop detection is optimized. This improves the significance and accuracy of a dryness estimation. Furthermore, the approach presented here increases the robustness of computer vision applications with the dishwashing camera by ensuring better color fidelity. This is particularly important when extraneous light penetrates the interior 105 when the door of the cleaning device 100 is opened.

[0049] Figure 2shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 1 , except that the illustration of the camera and the control device is omitted.

[0050] Figure 3 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 has, merely as an example, the two loading baskets 140, 150 and the one spray arm 175. The cutlery drawer, the lighting devices, the camera, and the control device are omitted.

[0051] In Figure 3 the third spray arm 175 has a first spray arm position.

[0052] Figure 4 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 3 , except that the third spray arm 175 has a second spray arm position.

[0053] Figure 5 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 3 and / or Figure 4 , except that the third spray arm 175 has a third spray arm position.

[0054] Figure 6 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 2 .

[0055] In Figure 6 a tray 600 partially protrudes from the loading basket 140.

[0056] Figure 7 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 7 , except that the third spray arm 175 is shown in an operative state. In the operative state, the third spray arm 175 is blocked by the tray 600.

[0057] Figure 8shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 6 and / or Figure 7 .

[0058] For example only, the first lighting device 115 is activated and illuminates the interior 105. Light rays 800 of the first lighting device 115 fall onto the washware in such a way that the tray 600 casts a shadow 805 onto the third spray arm 175, more precisely onto the head of the third spray arm 175.

[0059] Figure 9 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 8, except that the second lighting device 125 is activated merely by way of example. Light rays 900 from the second lighting device 125 fall onto the washware in such a way that the tray 600 casts a shadow 905 on the third spray arm 175. The shadow 905 is visible on the left arm of the third spray arm 175.

[0060] Figure 10 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 8 and / or Figure 9 , with the exception that the third lighting device 135 is activated only as an example.

[0061] Light rays 1000 of the third lighting device 135 fall onto the items to be washed in such a way that a shadow 1005 is created on the tray 600.

[0062] Figure 11shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 8 and / or Figure 9 and / or Figure 10 , with the exception that the fourth lighting device 145 is activated only as an example.

[0063] Light rays 1100 of the fourth lighting device 145 fall on the items to be washed in such a way that a shadow 1105 is created in the tray 600. The Figures 6 to 11 show a detection of the positioning of the blockage based on the shadow positions.

[0064] Figure 12shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from one of the figures described above. The third spray arm 175 is shown in an operational state. In the operational state, however, the third spray arm 175 is blocked by a pot 1200, which is arranged in the second loading basket 150 and protrudes from the second loading basket 150 in such a way that it blocks the third spray arm 175 and the fourth lighting device 145.

[0065] Figure 13shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from one of the figures described above. The third spray arm 175 is shown in an operational state, but is blocked by the tray 600. The tray 600 is arranged in the first loading basket 140 and partially protrudes from the first loading basket 140, thereby blocking the third spray arm 175 and the third lighting device 135. Figures 12 to 13 show that the lighting equipment is concealed by parts that are raised or lowered.

[0066] Figure 14 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from one of the figures described above.

[0067] A ladle 1400 protrudes at least partially from the first loading basket 140. More precisely, the handle of the ladle 1400 protrudes toward the second loading basket 150.

[0068] Figure 15 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 14 , except that the third spray arm 175 is shown in an operational state. In this state, however, the third spray arm 175 is blocked by the ladle 1400 or the handle of the ladle 1400.

[0069] Figure 16 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 14 , except that the first lighting device 115 is activated.

[0070] Light rays 800 of the first lighting device 115 fall onto the items to be washed in such a way that the ladle 1400 casts a shadow onto the third spray arm 175.

[0071] Figure 17 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 16 , except that the second lighting device 125 is activated.

[0072] Light rays 900 of the second lighting device 125 fall onto the items to be washed in such a way that the ladle 1400 casts a shadow onto the third spray arm 175.

[0073] Figure 18 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 16 and / or Figure 17 , except that the third lighting device 135 is activated.

[0074] Light rays 1000 of the third lighting device 135 fall on the items to be washed in such a way that a shadow is created on the ladle 1400.

[0075] Figure 19 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 16 and / or Figure 17 and / or Figure 18 , except that the fourth lighting device 145 is activated. Figures 14 to 19 show a characteristic shadow formation on down parts.

[0076] Light rays 1100 of the fourth lighting device 145 fall onto the items to be washed in such a way that a shadow is created in the ladle 1400.

[0077] Figure 20 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 13. The tray 600 partially protrudes from the first loading basket 140 and thereby blocks the third spray arm 175, wherein the third spray arm 175 is arranged directly below the first loading basket 140.

[0078] Figure 21 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 20 except that the ladle 1400 partially protrudes from the first loading basket 140, more precisely the handle of the ladle 1400, and thereby blocks the third spray arm 175, the first loading basket 140 being arranged directly above the spray arm 175.

[0079] Figure 22 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 20 and / or Figure 22except that a pan 2200 partially protrudes from the second loading basket 150 toward the third spray arm 175, more precisely the stem of the pan 2200, and thus blocks the third spray arm 175. The first loading basket 140, the spray arm 175, and the second loading basket 150 are arranged very close to one another. Figures 21 to 22 show different basket heights and position dependence, including different blocking situations.

[0080] Figure 23 shows an illustration of an embodiment of a cleaning device 100 that is similar to the cleaning device from one of the figures described above.

[0081] For example, a bowl 2300 having a concave bottom 2305 is arranged in the second loading basket 150. The bowl 2300 is arranged in the second loading basket 150 such that the concave bottom 2305 points toward the third spray arm 175. Figure 23shows the cleaning device 100 before a rinse cycle.

[0082] Figure 24 shows an illustration of an embodiment of a cleaning device 100 after a rinse cycle. The cleaning device 100 is similar to the cleaning device from Figure 23 except that rinse water 2400 has collected on the underside 2305 of the bowl 2300.

[0083] Figure 25 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 24 except that the third lighting device 135 is activated.

[0084] Light rays 1000 of the third lighting device 135 fall on the items to be washed in such a way that a shadow 2500 is created on the underside 2305 of the bowl 2300.

[0085] Figure 26 shows an illustration of an embodiment of a cleaning device 100. The cleaning device 100 is similar to the cleaning device from Figure 25 except that the fourth lighting device 145 is activated.

[0086] Light rays 1100 of the fourth lighting device 145 fall onto the items to be washed in such a way that a shadow 2500 is created on the underside 2305 of the bowl 2300.

[0087] Figure 27 shows a representation of an interior 105 of an embodiment of a cleaning device 100.

[0088] Arranged in the interior space 105, purely by way of example, is a cup 2700, which is illuminated by the second illumination device 125. Light rays from the second illumination device 125 strike the cup 2700 in such a way that the cup 2700 casts a shadow 2705.

[0089] Figure 28 shows a representation of an interior 105 of an embodiment of a cleaning device 100. The interior 105 is similar to the interior of Figure 27, except that the first illumination device 115 is activated. The cup 2700 is illuminated by the first illumination device 115 such that the cup 2700 casts a shadow 2705.

[0090] Figure 29 A flowchart of an embodiment of a method for detecting the interior of a cleaning device. The cleaning device corresponds to the cleaning device in one of the figures described above.

[0091] Block 2900 marks the desired start of a cleaning program. Subsequently, in block 2905, a check is made to determine whether an image of an empty interior of the cleaning appliance, i.e., an empty image, is present and / or whether the interior of the cleaning appliance is empty. If no empty image is present or the interior is empty, the program branches to block 2910. In block 2910, a basic lighting configuration is switched on. In block 2915, image data is recorded and stored in memory. In other words, a wash cabinet image is taken without wash grids, but with racks. Subsequently, the program branches to block 2920. In block 2920, the wash items are placed in the interior.

[0092] It then jumps back to block 2905 and from there to block 2925.

[0093] In block 2925, image data is captured by the camera and stored in memory.

[0094] The program then jumps to block 2930, where the lighting configuration is changed.

[0095] In block 2935, a check is made to determine whether all lighting configurations have been recorded. If not, the program returns to block 2925. This is repeated until all lighting configurations have been recorded.

[0096] In block 2940, the images are preprocessed. According to one embodiment, block 2930 jumps directly to block 2940. According to another embodiment, block 2920 jumps directly to block 2940.

[0097] In block 2945, the depth information is estimated. Then, the program jumps to block 2950, ​​where the depth information is provided to another process.

[0098] Block 2955 represents a start of the desired cleaning program or an end of the desired cleaning program.

[0099] Figure 30 shows a flowchart of an embodiment of a method 3000 for detecting an interior of a cleaning device.

[0100] Method 3000 includes a step 3005 of outputting a first illumination signal to an interface to the first illumination device in order to place the first illumination device into a first state. The first state represents, for example, a state in which the first illumination device is activated and illuminates the interior of the cleaning device.

[0101] In addition, the method 3000 comprises a step 3010 of reading in a first image signal via an interface to the camera, wherein the first image signal represents a first image captured by the camera while the first illumination device is in the first state.

[0102] The method 3000 further includes a step 3015 of outputting a second illumination signal to the interface to the first illumination device in order to place the first illumination device into a second state. The second state represents, for example, a state in which the first illumination device is deactivated or dimmed and does not illuminate or only partially illuminates the interior of the cleaning device.

[0103] Furthermore, the method 3000 comprises a step 3020 of reading in a second image signal via the interface to the camera, wherein the image signal represents a second image captured by the camera while the first illumination device is in the second state.

[0104] The method 3000 additionally comprises a step 3025 of determining a difference image signal using the first image signal and the second image signal, wherein the difference image signal represents depth information of the interior.

[0105] According to one embodiment, method 3000 includes a step 3030 of determining a parameter signal for controlling a program sequence of the cleaning device using the difference image signal. The determining step 3030 is performed after the determining step 3025.

[0106] In addition, the method 3000 includes, for example, a step 3035 of preprocessing the first image signal and the second image signal. Preprocessing step 3035 is performed before determining step 3025.

[0107] According to a further embodiment, method 3000 includes a step 3040 of capturing a reference image signal, wherein the reference image signal represents a reference image of an empty interior of the cleaning device captured by the camera. The capturing step 3040 is performed before the determining step 3025.

[0108] The method 3000 includes, for example, a step 3045 of outputting a third illumination signal to an interface to the second illumination device in order to place the second illumination device into the first state. In a reading step 3050, the third image signal is read in via the interface to the camera, wherein the third image signal represents a third image captured by the camera while the second illumination device is in the first state. In a determining step 3025, the difference image signal is determined using the first image signal, the second image signal, and the third image signal.

[0109] According to a further embodiment, in a step 3055 of outputting, a fourth illumination signal is output to an interface to the second illumination device in order to place the second illumination device into the second state. In a step 3060 of reading, a fourth image signal is read in via the interface to the camera, wherein the fourth image signal represents a fourth image captured by the camera while the second illumination device is in the second state. In step 3025 of determining, the difference image signal is determined using the first image signal, the second image signal, the third image signal, and the fourth image signal.

[0110] In other words, the method 3000 is designed to generate depth information.

[0111] In summary, the method 3000 consists of two steps: the generation of the image material and the estimation of depth information.

[0112] The image data is generated sequentially. One lighting configuration is switched on, an image is taken, and then the next lighting configuration is switched on, and so on. A lighting configuration can refer to an active lighting device. However, individual lamps can also be switched off, so that the switching sequence is perceptible to a person in front of the device with only minimal light fluctuations. Another image can be taken with all lighting units active. In the simplest form, only the right lamp is switched on once, only the left lamp is switched on once, and all lamps are switched on once, creating three images.

[0113] Depth information is estimated based on difference images. Different shadows are created in every lighting configuration. By comparing one image with the image with all lighting units switched on, shadows and halos can be identified. Particularly with structured objects, large differences are visible even on straight surfaces. To prevent this, a smoothing operator and further preprocessing, such as the 2D median filter, can be applied before the difference. The image data is then used to estimate depth information. An analytical method, such as classical image processing, can be used to approximate the shadow area and / or position, or a trained model, such as artificial intelligence, can be used to determine information on the shadow area, position, and / or orientation from the images.

[0114] Method 3000 can also be performed with dimmed lighting units. For example, each lamp can be dimmed by 50% to significantly change the lighting configuration.

[0115] Furthermore, an image can also be used with the light sources switched off in order to calculate the influence of ambient light that penetrates through a slightly open dishwasher door.

[0116] In order to achieve a better separation of shadows from the wash items and baskets / inner shelves, the process can be extended so that a reference image, i.e. an empty image, is taken when the object to be examined is not placed in the dishwasher interior. In this case, the lighting is switched on in the default setting, an image is taken and only then is the item added. This can be particularly advantageous when the items to be washed are geometrically complex. The taken reference image can then be used to calculate the difference images. In the dishwasher, the empty image is stored statically because it is not always clear when the interior is actually free of items to be washed. For this purpose, after the machine has been unloaded after a wash cycle, an image is taken which is verified to be empty by an external process, detected via Kl.

[0117] The entire 3000 method not only serves to statically determine depth information at a specific point in time, but can also be used dynamically, i.e., over time. For this purpose, the illumination and depth estimation process is repeated periodically, and the determined values ​​are stored as a time history in the memory of the process computer. The measurement points on the curves serve to improve the detection of rinse shadows or possible pools of water. If a dish is positioned upwards, it can sometimes be detected that it is gradually filling up, causing the standing water to change the shape of the contour. While this cannot be used to analyze the problem before the rinse process, it may provide indications that the cleaning performance of critical objects, such as kidney dishes in professional equipment, may have been insufficient.

[0118] Figure 31shows a block diagram of a control device 120 for detecting an interior of a cleaning device. The control device 120 is similar to or corresponds to the control device of Figure 1 .

[0119] The control device 120 has units 3105, 3115, 3145, 3155 for output, units 3110, 3120, 3150, 3160 for reading, a unit 3125 for determining and optionally a unit 3130 for determining, a unit 3135 for preprocessing and a unit 3140 for detecting.

[0120] The output unit 3105, 3115, 3145, 3155 are designed to output a lighting signal 3108, 3118, 3148, 3158 each.

[0121] The reading units 3110, 3120, 3150, 3160 are designed to read in one image signal each 3112, 3122, 3152, 3162.

[0122] The first image signal 3112 is read in while the first illumination signal 3108 is output. The second image signal 3122 is read in while the second illumination signal 3118 is output. The third image signal 3152 is read in while the third illumination signal 3148 is output. The fourth image signal 3162 is read in while the fourth illumination signal 3158 is output.

[0123] The determining unit 3125 is configured to determine a difference image signal 3128 using the first image signal 3112, the second image signal 3122 and optionally the third image signal 3152 and fourth image signal 3162.

[0124] The determining unit 3130 is designed to determine a parameter signal 3132 for controlling a program sequence of the cleaning device using the difference image signal 3128.

[0125] The preprocessing unit 3135 is configured to preprocess the image signals 3112, 3122, and optionally the image signals 3152, 3162.

[0126] The acquisition unit 3140 is configured to acquire a reference image signal 3142. The reference image signal 3142 represents a reference image of an empty interior of the cleaning device captured by the camera. The determination unit 3125 determines the difference image signal 3128 using the reference image signal 3142.

Claims

1. Method (3000) for detecting an interior (105) of a cleaning device (100), wherein the cleaning device (100) has a camera (110) for detecting the interior (105) of the cleaning device (100) and at least one first illumination device (115) and preferably a second illumination device (125) for illuminating the interior (105), wherein the cleaning device (100) has a control device (120), wherein the method (3000) comprises the following steps: outputting (3005) a first illumination signal (3108) to an interface to the first illumination device (115) in order to put the first illumination device (115) into a first state; Reading (3010) a first image signal (3112) via an interface to the camera (110), wherein the first image signal (3112) represents a first image captured by the camera (110) while the first illumination device (115) is in the first state;Outputting (3015) a second illumination signal (3118) to the interface to the first illumination device (115) in order to place the first illumination device (115) into a second state; reading in (3020) a second image signal (3122) via the interface to the camera (110), wherein the second image signal (3122) represents a second image recorded by the camera (110) while the first illumination device (115) is in the second state; and determining (3025) a difference image signal (3128) using the first image signal (3112) and the second image signal (3122), wherein the difference image signal (3128) represents depth information of the interior space (105).

2. Method (3000) according to claim 1, comprising a step (3030) of determining a parameter signal (3132) for controlling a program sequence of the cleaning device (100) using the difference image signal (3128).

3. Method (3000) according to one of the preceding claims, comprising a step (3035) of preprocessing the first image signal (3112) and the second image signal (3122), wherein the step (3035) of preprocessing is carried out before the step (3025) of determining.

4. Method (3000) according to one of the preceding claims, comprising a step of outputting (3045) a third illumination signal (3148) to an interface to the second illumination device (125) in order to put the second illumination device (125) into the first state, comprising a step (3050) of reading in a third image signal (3152) via the interface to the camera (110), wherein the third image signal (3152) represents a third image recorded by the camera (110) while the second illumination device (125) is in the first state, wherein in the step (3025) of determining, the difference image signal (3128) is determined using the first image signal (3112), the second image signal (3122) and the third image signal (3152).

5. The method (3000) according to any one of the preceding claims, comprising a step of outputting (3055) a fourth illumination signal (3158) to an interface to the second illumination device (125) in order to place the second illumination device (125) in the second state, comprising a step (3060) of reading in a fourth image signal (3162) via the interface to the camera (110), wherein the fourth image signal (3162) represents a fourth image recorded by the camera (110) while the second illumination device (125) is in the second state, wherein in the step (3025) of determining, the difference image signal (3128) is determined using the first image signal (3112), the second image signal (3122), the third image signal (3152) and the fourth image signal (3162).

6. The method (3000) according to any one of the preceding claims, wherein in the outputting steps (3005, 3045), the first state represents a state in which the first lighting device (115) and / or the second lighting device (125) is activated to illuminate the interior (105) of the cleaning device (100).

7. The method (3000) according to any one of the preceding claims, wherein in the steps (3015, 3055) of outputting, the second state represents a state in which the first lighting device (115) and / or the second lighting device (125) is deactivated or dimmed.

8. The method (3000) according to any one of the preceding claims, comprising a step (3040) of detecting a reference image signal (3142), wherein the reference image signal (3142) represents a reference image of an empty interior (105) of the cleaning device (100) recorded by the camera (110), wherein in the step (3025) of determining, the difference image signal (3128) is determined using the reference image signal (3142).

9. Control device (120) for detecting an interior space (105) of a cleaning device (100), wherein the control device (120) is designed to carry out and / or control the steps of the method (3000) according to one of the preceding claims in corresponding units.

10. Cleaning device (100) with a camera (110) for capturing an interior space (105) of the cleaning device (100) and with a first lighting device (115) and a second lighting device (125) for illuminating the interior space (105) of the cleaning device (100) and with a control device (120) according to claim 9.

11. Cleaning device (100) according to claim 10, comprising a door for closing the interior (105) of the cleaning device (100), wherein the camera (110) is arranged on the door.

Images