Dishwasher and method for detecting an object therein

Abstract

A dishwasher (105) comprises a dishwashing chamber (115); a dish-rack (120), which can be pulled out of the dishwashing chamber (115) and is intended for receiving an object (125); a camera (150) for capturing an image of the object (125); and a processing device (145). The processing device (145) is designed here to determine three-dimensional information concerning the object (125) with respect to a first and a second image of the camera (150); wherein the images are captured in different positions of the dish-rack (120) in relation to the camera (150).

Description

[0001] Dishwasher and method for detecting an object therein

[0002] The present invention relates to a dishwasher. In particular, the invention relates to the detection of information about an object placed in a dishwasher.

[0003] A dishwasher consists of a washing chamber and a dish basket. An object can be placed in the dish basket, which can then be cleaned in the washing chamber.

[0004] US 8 696 827 B2 proposes a dishwasher with a camera to determine the arrangement of objects in the dish basket and to coordinate a cleaning process of the objects accordingly.

[0005] DE 10 2020 211 543 A1 relates to a dishwasher with a horizontally extendable dish basket. A camera is configured to capture images of the dish basket in different horizontal positions. The images are stored in assigned memory locations and can be processed later.

[0006] US 2022 092 808 A1 proposes to optically scan an object in a dishwasher under different lighting conditions and to create a three-dimensional image of an object based on scanned images.

[0007] An object underlying the present invention is to provide an improved technique for providing information about an object placed in a dishwasher for cleaning. The invention achieves this object by means of the subject matter of the independent claims. Subclaims specify preferred embodiments.

[0008] According to a first aspect of the present invention, a dishwasher comprises a washing chamber; a dish basket that can be pulled out of the washing chamber for receiving an object; a camera for capturing an image of the object; and a processing device. The processing device is configured to determine three-dimensional information about the object with reference to a first and a second image from the camera; the images being captured in different positions of the dish basket relative to the camera.

[0009] It was recognized that a single camera is sufficient to provide images from different perspectives, from which the three-dimensional information can be determined. The camera is preferably mounted in a fixed position opposite the washing chamber, allowing images to be provided at different straight lines as the dish basket is pulled horizontally out of the washing chamber. A standard dishwasher can be equipped with a suitable camera at low cost.

[0010] The washing chamber may have a reflective inner surface. The camera may be configured to capture a reflection of the dish basket and / or the object on the surface. Preferably, the camera is arranged or aligned such that both a direct image and a reflection are visible in one image. The direct image and the reflection may have different content.

[0011] According to another aspect of the present invention, a system comprises a dishwasher as described herein and a mobile device. The dishwasher is configured to transmit certain three-dimensional information to the mobile device; and the mobile device is configured to provide an indication of the information.

[0012] The mobile device may, in particular, comprise a user's personal device, for example, a smartphone, a tablet computer, or a laptop computer. The mobile device may comprise a graphical output device on which a representation of a scanned object can be displayed. Optionally, further information may also be provided to a user, in particular an indication of a potentially more advantageous positioning of a scanned object.

[0013] According to yet another aspect of the present invention, a method for providing three-dimensional information about an object in a dish basket of a dishwasher with a camera comprises the steps of capturing a first image of the object while the dish basket is in a first position relative to the camera; capturing a second image of the object while the dish basket is in a second position relative to the camera; wherein the second position is different from the first position; determining the three-dimensional information with respect to the images; and providing the determined information.

[0014] The method can be carried out using a dishwasher or a system described herein. The processing device of the dishwasher can be configured to carry out the method partially or completely. For this purpose, the processing device can be implemented electronically and, for example, comprise a programmable microcomputer or microcontroller. The method can be in the form of a computer program product with program code means. The computer program product can be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the dishwasher or the system, and vice versa.

[0015] It is preferred that a plurality of images of the object are determined from different positions of the dish basket relative to the camera; and that the three-dimensional information is determined with respect to the plurality of images. The different positions of the dish basket relative to the washing chamber can be predetermined by a predetermined relative mobility of the dish basket. In one embodiment, the dish basket is arranged so as to be horizontally displaceable relative to the washing chamber by means of a drawer pull-out or a comparable construction. In a predetermined end position, the dish basket is received in the washing chamber so that it can be closed, usually by means of a flap that can be pivoted about a horizontal axis.

[0016] In a further embodiment, multiple cameras are provided, each configured to scan images of the object. The cameras are arranged offset from one another so that they provide different perspectives of the object. The images from two or more cameras, scanned in different horizontal positions of the dish basket, can contribute to creating three-dimensional information about a scanned object.

[0017] The three-dimensional information can be determined in the traditional way using triangulation. Based on the known positions of the object in the different images, corresponding optical features can be determined and placed in a three-dimensional context. In the case of only two images, this is also referred to as stereo triangulation.

[0018] Preferably, the three-dimensional information is determined using a neural radiation field (NeRF).

[0019] A neural radiation field (NeRF) is a deep learning-based method for reconstructing a three-dimensional (3D) representation of a scene from two-dimensional (2D) images. It is a graphical primitive that can be optimized from a set of 2D images to generate a 3D scene. The NeRF model can learn the geometry of the scene, the camera positions, and the reflectance properties of objects in a scene, allowing it to optionally render new views of the scene from new perspectives. Available software for implementing a NeRF includes ProteusNeRF, hypernerf, or nerfstudio.

[0020] To train the creation of 3D information about an object, reinforcement learning can be used on an artificial neural network (ANN). Training data preferably includes images of objects that are typically cleaned in a dishwasher, such as dishes, cutlery, glasses, pots, pans, or cooking utensils. Each object can be optically scanned from different perspectives to provide a set of images from which 3D information can be created. The perspectives of a set preferably lie on a straight line with respect to the object. An object can be scanned in different positions or from perspectives along different straight lines to generate training data.

[0021] The position of the dish basket can be determined, for example, using a dedicated sensor. The sensor can be configured, in particular, to determine the horizontal position of the movable dish basket relative to the washing chamber. However, such a sensor can be omitted by determining the position of the dish basket based on the scanned images.

[0022] In a preferred embodiment, a predetermined optical feature of the dish basket is detected; the position of the dish basket relative to the camera is determined with reference to the detected feature. A single image may be sufficient to determine the position. The dish basket is typically manufactured from a wire mesh and includes numerous edges or surfaces that can be used as optical features. For example, an optical feature may include an intersection of two wires, a corner, or a characteristic bend in a wire. The dish basket is permanently assigned to the dishwasher, so its optical properties are typically known and unchangeable. Optionally, an optical marker can also be attached or formed on the dish basket. For example, a predetermined reflective marking can be subsequently applied to the dish basket.Alternatively, the dish basket can be specifically shaped to form an optical marker that can be easily recognized in an image. In one embodiment, an optical marker on the dish basket can include a trademark of a dishwasher manufacturer. This can provide additional protection against the use of an unintended dish basket on the dishwasher.

[0023] It is further preferred that at least one of the images is captured while the dish basket is being pulled out of the washing chamber or pushed into the washing chamber. During normal operation of the dishwasher, such a process can occur frequently. In this case, a large number of images can be scanned, which can contribute to determining the three-dimensional information. After the object has been placed, the dish basket can be pulled out of the washing chamber and pushed into the washing chamber several times. Images of the object can be scanned each time, so that sufficient data can be available to determine the three-dimensional information. The method can determine the three-dimensional structure of the object from the relative movement of the object compared to the camera, which is why it can be referred to as a "structure from motion" method.

[0024] In a further embodiment, the object is classified. For the classification, a dedicated ANN or the same one as for creating the three-dimensional information can be used. Example classes for the object include cutlery, pots, glasses, plates, cups, pans, baking utensils, or cooking utensils. The classification can be assigned to the three-dimensional information. Optionally, the determination of the three-dimensional information can be further improved based on the determined object class. Optionally, an arrangement of the object in the washing chamber that is improved with regard to cleaning the object in the dishwasher can be determined. The arrangement can comprise a position or an attitude or orientation of the object. For example, it can be suggested that a bowl that has been placed in the dish basket with the opening facing upwards be turned over.Depending on information associated with a classified object, it can also be determined where in the washing chamber the object can best be treated. For example, an object containing plastic parts can be advantageously placed in a corner during a high-temperature washing program to avoid deformation of the plastic. Utensils made of wood, porcelain, glass, steel, or cast iron can each be placed in advantageous positions in the washing chamber. The dishwasher can comprise multiple dish baskets, and the arrangement can extend across multiple dish baskets. For example, it can be suggested that a pot located in an upper dish basket be more advantageously placed in the lower dish basket, where more intensive cleaning can take place.

[0025] Further preferably, three-dimensional information about multiple objects is determined, allowing an optimized arrangement of the objects in the washing chamber to be determined. The available space in the washing chamber can be utilized more effectively. Unfavorable mutual overlap of objects can be reduced or avoided. A cleaning result can be improved. A cleaning time can be reduced.

[0026] The invention will now be described in more detail with reference to the accompanying figures, in which:

[0027] Figure 1 shows a system with a dishwasher; and

[0028] Figure 2 illustrates a method for determining three-dimensional information of an object in a dishwasher.

[0029] Figure 1 shows a system 100 comprising a dishwasher 105 and a mobile device 110.

[0030] The dishwasher 105 comprises a washing chamber 115 in which at least one dish basket 120 is arranged. Purely by way of example, the illustrated dishwasher 105 has an upper, a middle, and a lower dish basket 120. The dish baskets 120 are each mounted horizontally displaceably on the dishwasher 105, so that they can be pulled out of or pushed into the washing chamber 115.

[0031] An object 125 to be cleaned can be placed in one of the dish baskets 120 while the basket is pulled out of the washing chamber 115. After all dish baskets 120 have been pushed into the washing chamber 115, the latter can be closed by means of a flap 130 that is folded upward about an axis arranged in the lower region of the dishwasher 105. After the washing chamber 115 is closed, water and a cleaning agent can be distributed in the washing chamber 115. The water can be directed from below through the dish baskets 120 to the objects 125 held therein by means of spray arms 135.

[0032] The dishwasher 105 preferably comprises a device 140 configured to determine and provide three-dimensional information about an object 125 accommodated in the dishwasher 105. For this purpose, the device 140 preferably comprises a processing device 145 and a camera 150. Further preferably, a communication device 155 is provided for communication with the mobile device 110. The communication device 155 is preferably wireless and can, for example, establish a Bluetooth (BT, BLE) or WLAN connection. If a mobile device 110 is not to be used, an output device configured to provide information to a human user of the system 100 or the dishwasher 105 can be provided instead of the communication device 155.

[0033] The device 140 is configured to scan images of an object 125 in a dish basket 120 from different perspectives and to determine three-dimensional information about the object 125 based on the images. For this purpose, a three-dimensional model of the object 125 can be created. Furthermore, the object 125 can be recognized or classified. Predetermined information can be associated with a recognized object 125 or a class into which the object 125 has been sorted. Such information can be associated with, for example, dimensions, a material, a preferred treatment, an incompatibility with certain cleaning agents, or other information that may be particularly relevant for a cleaning process in the dishwasher 105.

[0034] It is further proposed that, based on a detected object 125, its position and / or orientation in the washing chamber 115, it is determined whether a different position or orientation could possibly be more advantageous for a cleaning process. With regard to a plurality of objects 125, an arrangement of the objects 125 can be determined that can allow an optimized cleaning process for the objects 125. For example, using the mobile device 110, a user can be given instructions so that they can arrange the objects 125 more advantageously in the washing chamber 115 or the dish baskets 120 based on the determinations made. For an improved arrangement, it may be necessary under certain circumstances to remove an object 125 from the washing chamber 115.

[0035] Figure 2 shows a flowchart of an exemplary method 200 for determining three-dimensional information of an object 125 in a dishwasher 105. The method 200 can preferably be carried out using a system 100 or a dishwasher 105 according to Figure 1. In a step 205, a first image of an object 125 located in a dish basket 120 of the dishwasher 105 can be scanned using the camera 150. In a step 210, a first position of the dish basket 120 can be determined. To determine the position, an optically recognizable feature of the dish basket 120 can be determined in the first image. A position of the feature in the image can allow inferences to be drawn about the horizontal position of the dish basket 120 with respect to the washing chamber 115. The optical feature can, for example, comprise a dedicated reflective marking or a characteristic shape of the dish basket 120.A dish basket 120 can be designed as a basket, in particular as a wire basket. Wires or corresponding webs can form easily identifiable geometric features, either alone or in conjunction with other wires or webs, which can be determined in the first image.

[0036] In a step 215, a second image of the object 125 in the dish basket 120 can be scanned. In a step 220, a second position of the dish basket 120 can be determined. The procedure can be the same as in step 210 with regard to the first position.

[0037] In a step 225, it can be determined whether the difference between the determined positions is sufficiently large. If the positions do not differ sufficiently from each other, for example, if they do not differ at all, the second image can be discarded, and the method 200 can continue with step 215.

[0038] In steps 205 to 225, more than two images can be captured in different positions of the dish basket 120. During the repeated execution of steps 215 to 225, the dish basket 120 can be pulled out of the washing chamber 115 or pushed into it. Image capture can be completed, for example, if no further change in the specific position occurs for a predetermined period of time. Image capture can continue as long as the door 130 of the dishwasher 105 is open.

[0039] If a sufficient number of images have been scanned, a three-dimensional model of the object 125 can be determined in a step 230 based on the acquired images. The three-dimensional model can be generated as a numerical model or, for example, as a graphical representation. It is preferred that the model be determined using a neural radiation field. For this purpose, an ANN can be provided, which is trained to determine three-dimensional information of an object 125 based on two-dimensional images from different perspectives of the object 125.

[0040] In a step 235, the object 125 can be recognized or classified. A recognized object 125 or an associated class can be associated with information that can also be used to create the three-dimensional model.

[0041] In a step 240, an arrangement of the object 125 in the washing chamber 115 can be determined when the dish basket 120 is inserted into the washing chamber 115. In a step 245, an arrangement of multiple objects 125 in the washing chamber 115 can also be determined. In this case, objects 125 arranged in different dish baskets 120 can be taken into account. A relative arrangement of different objects 125 to one another can be determined or taken into account.

[0042] In a step 250, an optimized arrangement of one or more objects 125 can be determined. The arrangement can preferably be optimized with respect to a cleaning process. An object 125 can be arranged in an optimized manner such that it can be washed around by water in the washing chamber 115 in an assigned manner during a cleaning process. Mutual concealment of objects 125 can be avoided. Furthermore, an object 125 can be arranged such that liquid can advantageously drain from the object 125 at the end of the washing process. In a step 255, an indication can be determined as to which objects 125 need to be rearranged and in what manner in order to realize an improved arrangement of objects 125. The indication can be directed to a user of the dishwasher 105. The indication is preferably output to the user via a mobile device 110. In one embodiment, the indication can be graphic.For example, an animation may be provided of how an object 125 should be moved relative to a dish basket 120 to achieve the improved arrangement.

[0043] Reference symbol

[0044] 100 systems

[0045] 105 dishwashers

[0046] 110 Mobile device

[0047] 115 Rinse chamber

[0048] 120 dish basket

[0049] 125 objects

[0050] 130 flap

[0051] 135 spray arm

[0052] 140 device

[0053] 145 Processing facility

[0054] 150 Camera

[0055] 155 Communication device

[0056] 200 procedures

[0057] 205 scan first image

[0058] 210 Determine the first position of the dish basket

[0059] 215 scan second image

[0060] 220 Determine the second position of the dish basket

[0061] 225 Difference between positions sufficiently large?

[0062] 230 Determine 3D model of the object

[0063] 235 Classify object

[0064] 240 Determine the arrangement of the object in the closed dishwasher

[0065] 245 Determine the arrangement of multiple objects

[0066] 250 determine optimized arrangement

[0067] 255 Provide notice to user

Claims

PATENT CLAIMS 1 . Dishwasher (105), comprising: - a rinsing chamber (115); - a dish basket (120) which can be pulled out of the washing chamber (115) to hold an object (125); - a camera (150) for capturing an image of the object (125); and - a processing device (145) configured to determine three-dimensional information about the object (125) with respect to a first and a second image of the camera (150); - wherein the images are captured in different positions of the crockery basket (120) relative to the camera (150).

2. The dishwasher (105) according to claim 1, wherein the washing chamber (115) has a reflective inner surface; wherein the camera (150) is configured to detect a reflection of the washing basket and / or the object (125) on the surface.

3. A system (100) comprising a dishwasher (105) according to claim 1 or 2 and a mobile device (110); wherein the dishwasher (105) is configured to transmit certain three-dimensional information to the mobile device (110); and wherein the mobile device (110) is configured to provide an indication of the information.

4. A method (200) for providing three-dimensional information about an object (125) in a dish basket (120) of a dishwasher (105) using a camera (150); the method (200) comprising the following steps: - capturing (205) a first image of the object (125) while the dish basket (120) is in a first position relative to the camera (150); - capturing (215) a second image of the object (125) while the dish basket (120) is in a second position relative to the camera (150); - the second position being different from the first position; - determining (230) the three-dimensional information relating to the images; and - Providing (255) the specific information.

5. The method (200) of claim 4, wherein a plurality of images of the object (125) are determined (205-225) from different positions of the dish basket (120) relative to the camera (150); and wherein the three-dimensional information is determined with respect to the plurality of images.

6. The method (200) according to claim 4 or 5, wherein the three-dimensional information is determined (230) by means of a neural radiation field.

7. The method (200) according to any one of claims 4 to 6, wherein a predetermined optical feature of the dish basket (120) is detected; and wherein a position of the dish basket (120) relative to the camera (150) is determined (210, 220) with respect to the detected feature.

8. The method (200) according to any one of claims 4 to 7, wherein at least one of the images is captured while the dish basket (120) is being pulled out of the washing chamber (115) or pushed into the washing chamber (115).

9. The method (200) according to any one of claims 4 to 8, wherein the object (125) is classified (235).

10. The method (200) according to any one of claims 4 to 9, wherein an arrangement of the object (125) in the washing chamber (115) which is improved with respect to cleaning of the object (125) in the dishwasher (105) is determined (250).

11. The method (200) according to any one of the preceding claims, wherein three-dimensional information of a plurality of objects (125) is determined; wherein an optimized arrangement of the objects (125) in the rinsing chamber (115) is determined.

Images