Dishwasher and method for detecting an object therein

Abstract

A dishwasher (105) comprises a wash chamber (115); a dish rack (120) that can be pulled out from the wash chamber (115) for holding an object (125); a camera (150) for capturing an image of the object (125); and a processing unit (145). The processing unit (145) is configured to determine three-dimensional information about the object (125) with respect to a first and a second image from the camera (150), wherein the images are captured in different positions of the dish rack (120) relative to the camera (150).

Description

[0001] The present invention relates to a dishwasher. In particular, the invention relates to the detection of information about an object placed in a dishwasher.

[0002] A dishwasher consists of a washing chamber and a dish basket. An object can be placed in the dish basket, which can then be cleaned in the washing chamber.

[0003] US 8 696 827 B2 proposes a dishwasher with a camera to determine an arrangement of objects in the dish basket and to coordinate a cleaning process of the objects accordingly.

[0004] DE 10 2020 211 543 A1 relates to a dishwasher with a horizontally extendable dish basket. A camera is configured to capture images of the dish basket in different horizontal positions. The images are stored in assigned memory locations and can be processed later.

[0005] US 2022 092 808 A1 proposes to optically scan an object in a dishwasher under different lighting conditions and to create a three-dimensional image of an object based on scanned images.

[0006] An object underlying the present invention is to provide an improved technique for providing information about an object arranged for cleaning in a dishwasher. The invention solves this problem by means of the subject matter of the independent claims. Subclaims specify preferred embodiments.

[0007] According to a first aspect of the present invention, a dishwasher comprises a washing chamber; a dish basket that can be pulled out of the washing chamber for receiving an object; a camera for capturing an image of the object; and a processing device. The processing device is configured to determine three-dimensional information about the object with reference to a first and a second image from the camera; the images being captured in different positions of the dish basket relative to the camera.

[0008] It was recognized that a single camera is sufficient to provide images from different perspectives, from which the three-dimensional information can be determined. The camera is preferably mounted in a fixed position opposite the washing chamber, allowing images to be provided at different straight lines as the dish basket is pulled horizontally out of the washing chamber. A standard dishwasher can be equipped with a suitable camera at low cost.

[0009] The washing chamber may have a reflective inner surface. The camera may be configured to capture a reflection of the dish basket and / or the object on the surface. Preferably, the camera is arranged or aligned such that both a direct image and a reflection are visible in one image. The direct image and the reflection may have different content.

[0010] According to another aspect of the present invention, a system comprises a dishwasher as described herein and a mobile device. The dishwasher is configured to transmit certain three-dimensional information to the mobile device; and the mobile device is configured to provide an indication of the information.

[0011] The mobile device may, in particular, comprise a user's personal device, for example, a smartphone, a tablet computer, or a laptop computer. The mobile device may comprise a graphical output device on which a representation of a scanned object can be displayed. Optionally, further information may also be provided to a user, in particular an indication of a potentially more advantageous positioning of a scanned object.

[0012] According to yet another aspect of the present invention, a method for providing three-dimensional information about an object in a dish rack of a dishwasher with a camera comprises the steps of capturing a first image of the object while the dish rack is in a first position relative to the camera; capturing a second image of the object while the dish rack is in a second position relative to the camera; wherein the second position is different from the first position; determining the three-dimensional information with respect to the images; and providing the determined information.

[0013] The method can be carried out using a dishwasher or a system described herein. The processing device of the dishwasher can be configured to carry out the method partially or completely. For this purpose, the processing device can be implemented electronically and, for example, comprise a programmable microcomputer or microcontroller. The method can be in the form of a computer program product with program code means. The computer program product can be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the dishwasher or the system, and vice versa.

[0014] It is preferred that a plurality of images of the object are determined from different positions of the dish basket relative to the camera; and that the three-dimensional information is determined with respect to the plurality of images. The different positions of the dish basket relative to the washing chamber can be predetermined by a predetermined relative mobility of the dish basket. In one embodiment, the dish basket is arranged so as to be horizontally displaceable relative to the washing chamber by means of a drawer pull-out or a comparable construction. In a predetermined end position, the dish basket is received in the washing chamber so that it can be closed, usually by means of a flap that can be pivoted about a horizontal axis.

[0015] In a further embodiment, multiple cameras are provided, each configured to scan images of the object. The cameras are arranged offset from one another so that they provide different perspectives of the object. The images from two or more cameras, scanned in different horizontal positions of the dish basket, can contribute to creating three-dimensional information about a scanned object.

[0016] The three-dimensional information can be determined in the traditional way using triangulation. Based on the known positions of the object in the different images, corresponding optical features in the images can be determined and placed in a three-dimensional context. In the case of only two images, this is also referred to as stereo triangulation.

[0017] Preferably, the three-dimensional information is determined using a neural radiation field (NeRF).

[0018] A neural radiation field (NeRF) is a deep learning-based method for reconstructing a three-dimensional (3D) representation of a scene from two-dimensional (2D) images. It is a graphical primitive that can be optimized from a set of 2D images to generate a 3D scene. The NeRF model can learn the geometry of the scene, the camera positions, and the reflectance properties of objects in a scene, allowing it to optionally render new views of the scene from new perspectives. Available software for implementing a NeRF includes ProteusNeRF, hypernerf, or nerfstudio.

[0019] To train the creation of 3D information about an object, reinforcement learning can be used on an artificial neural network (ANN). Training data preferably includes images of objects that are typically cleaned in a dishwasher, such as dishes, cutlery, glasses, pots, pans, or cooking utensils. Each object can be optically scanned from different perspectives to provide a set of images from which 3D information can be created. The perspectives of a set preferably lie on a straight line with respect to the object. An object can be scanned in different positions or from perspectives along different straight lines to generate training data.

[0020] The position of the dish basket can be determined, for example, using a dedicated sensor. The sensor can be configured, in particular, to determine the horizontal position of the movable dish basket relative to the washing chamber. However, such a sensor can be omitted by determining the position of the dish basket based on the scanned images.

[0021] In a preferred embodiment, a predetermined optical feature of the dish basket is detected; the position of the dish basket relative to the camera is determined with reference to the detected feature. A single image may be sufficient to determine the position. The dish basket can usually be manufactured on the basis of a wire mesh and comprises numerous edges or surfaces that can be used as optical features. For example, an optical feature can include an intersection of two wires, a corner, or a characteristic bend in a wire. The dish basket is permanently assigned to the dishwasher, so that its optical properties are usually known and unchangeable. Optionally, an optical marker can also be attached or formed on the dish basket. For example, a predetermined reflective marking can be subsequently applied to the dish basket.Alternatively, the dish basket can be specifically shaped to form an optical marker that can be easily recognized in an image. In one embodiment, an optical marker on the dish basket can include a trademark of a dishwasher manufacturer. This can provide additional protection against the use of an unintended dish basket on the dishwasher.

[0022] It is further preferred that at least one of the images is captured while the dish basket is being pulled out of the washing chamber or pushed into the washing chamber. During normal operation of the dishwasher, such a process can occur frequently. In this case, a large number of images can be scanned, which can contribute to determining the three-dimensional information. After the object has been placed, the dish basket can be pulled out of the washing chamber and pushed into the washing chamber several times. Images of the object can be scanned each time, so that sufficient data can be available to determine the three-dimensional information. The method can determine the three-dimensional structure of the object from the relative movement of the object with respect to the camera, which is why it can be referred to as a "structure from motion" method.

[0023] In a further embodiment, the object is classified. A dedicated AINN or the same AINN as that used to create the three-dimensional information can be used for the classification. Example classes for the object include cutlery, pots, glasses, plates, cups, pans, baking utensils, or cooking utensils. The classification can be assigned to the three-dimensional information. Optionally, the determination of the three-dimensional information can be further improved based on the specific object class.

[0024] Optionally, an arrangement of the object in the washing chamber that is improved with regard to cleaning the object in the dishwasher can be determined. The arrangement can comprise a position or an attitude or orientation of the object. For example, it can be suggested that a bowl that has been placed in the dish basket with the opening facing upwards be turned over. Depending on information associated with a classified object, it can also be determined at which location in the washing chamber the object can best be treated. For example, an object that includes plastic parts can advantageously be placed in a corner during a high-temperature washing program to avoid deformation of the plastic. Utensils made of wood, porcelain, glass, steel or cast iron can each be placed in advantageous positions in the washing chamber.The dishwasher may include multiple dish racks, and the arrangement may extend across multiple dish racks. For example, it may be suggested that a pot located in an upper dish rack be more advantageously placed in the lower dish rack, where more intensive cleaning can take place.

[0025] Further preferably, three-dimensional information about multiple objects is determined, allowing an optimized arrangement of the objects in the rinsing chamber to be determined. The available space in the rinsing chamber can be utilized more effectively. Unfavorable mutual overlap of objects can be reduced or avoided. A cleaning result can be improved. A cleaning time can be reduced.

[0026] The invention will now be described in more detail with reference to the accompanying figures, in which: Fig. 1 a system with a dishwasher; and Fig. 2 discloses a method for determining three - dimensional information of an object in a dishwasher.

[0027] Fig. 1 shows a system 100, which includes a dishwasher 105 and a mobile device 110.

[0028] The dishwasher 105 includes a wash chamber 115, in which at least one dish basket 120 is arranged. By way of example only, the illustrated dishwasher 105 has an upper, a middle and a lower dish basket 120. The dish baskets 120 are each horizontally displaceably mounted on the dishwasher 105, so that they can be pulled out of or pushed into the wash chamber 115.

[0029] An object 125 to be cleaned can be placed into one of the dish baskets 120 while the basket is pulled out of the washing chamber 115. After all dish baskets 120 have been pushed into the washing chamber 115, the latter can be closed by means of a flap 130 that is folded upward about an axis arranged in the lower region of the dishwasher 105. After the washing chamber 115 is closed, water and a cleaning agent can be distributed in the washing chamber 115. The water can be directed from below through the dish baskets 120 to the objects 125 held therein by means of spray arms 135.

[0030] The dishwasher 105 preferably comprises a device 140 configured to determine and provide three-dimensional information about an object 125 accommodated in the dishwasher 105. For this purpose, the device 140 preferably comprises a processing device 145 and a camera 150. Further preferably, a communication device 155 is provided for communication with the mobile device 110. The communication device 155 is preferably wireless and can, for example, establish a Bluetooth (BT, BLE) or WLAN connection. If a mobile device 110 is not to be used, an output device configured to provide information to a human user of the system 100 or the dishwasher 105 can be provided instead of the communication device 155.

[0031] The device 140 is configured to scan images of an object 125 in a dish basket 120 from different perspectives and, based on the images, to determine three-dimensional information about the object 125. For this purpose, a three-dimensional model of the object 125 can be created. Furthermore, the object 125 can be recognized or classified. Predetermined information can be associated with a recognized object 125 or a class into which the object 125 has been sorted. Such information can, for example, include dimensions, a material, a preferred treatment, an incompatibility with certain cleaning agents, or other information that may be particularly relevant for a cleaning process in the dishwasher 105.

[0032] It is further proposed that, based on a detected object 125, its position and / or orientation in the washing chamber 115, it be determined whether another position or orientation could possibly be more advantageous for a cleaning process. With regard to a plurality of objects 125, an arrangement of the objects 125 can be determined that can allow an optimized cleaning process on the objects 125. For example, using the mobile device 110, a user can be given instructions so that they can arrange the objects 125 more advantageously in the washing chamber 115 or the dish baskets 120 based on the determinations made. For an improved arrangement, it may be necessary under certain circumstances to remove an object 125 from the washing chamber 115.

[0033] Fig. 2 shows a flowchart of an exemplary method 200 for determining three-dimensional information of an object 125 in a dishwasher 105. The method 200 can preferably be carried out by means of a system 100 or a dishwasher 105 according to Fig. 1 can be executed.

[0034] In a step 205, a first image of an object 125 located in a dish basket 120 of the dishwasher 105 can be scanned by means of the camera 150. In a step 210, a first position of the dish basket 120 can be determined. To determine the position, an optically recognizable feature of the dish basket 120 can be determined on the first image. A position of the feature on the image can allow inferences to be drawn about the horizontal position of the dish basket 120 with respect to the washing chamber 115. The optical feature can, for example, comprise a dedicated reflective marking or a characteristic shape of the dish basket 120. A dish basket 120 can be designed as a basket, in particular as a wire basket. Wires or corresponding webs can, alone or in conjunction with other wires or webs, form easily identifiable geometric features that can be determined on the first image.

[0035] In a step 215, a second image of the object 125 in the dish basket 120 can be scanned. In a step 220, a second position of the dish basket 120 can be determined. The procedure can be the same as in step 210 with respect to the first position.

[0036] In a step 225, it can be determined whether the difference between the determined positions is sufficiently large. If the positions do not differ sufficiently from each other, for example, if they do not differ at all, the second image can be discarded, and the method 200 can continue with step 215.

[0037] In steps 205 to 225, more than two images can be captured in different positions of the dish basket 120. During the repeated execution of steps 215 to 225, the dish basket 120 can be pulled out of or pushed into the washing chamber 115. Image capture can be completed, for example, if the specific position is not changed for a predetermined period of time. Image capture can continue as long as the door 130 of the dishwasher 105 is open.

[0038] If a sufficient number of images have been scanned, a three-dimensional model of the object 125 can be determined in a step 230 based on the acquired images. The three-dimensional model can be generated as a numerical model or, for example, as a graphical representation. It is preferred that the model be determined using a neural radiation field. For this purpose, an ANN can be provided, which is trained to determine three-dimensional information of an object 125 based on two-dimensional images from different perspectives of the object 125.

[0039] In a step 235, the object 125 can be recognized or classified. Information can be associated with a recognized object 125 or an associated class, which can also be used to create the three-dimensional model.

[0040] In step 240, an arrangement of the object 125 in the rinsing chamber 115 can be determined when the dish basket 120 is inserted into the rinsing chamber 115. In step 245, an arrangement of multiple objects 125 in the rinsing chamber 115 can also be determined. Objects 125 that are arranged in different dish baskets 120 can be taken into account. A relative arrangement of different objects 125 to each other can be determined or taken into account.

[0041] In a step 250, an optimized arrangement of one or more objects 125 can be determined. The arrangement can preferably be optimized with respect to a cleaning process. An object 125 can be arranged in such a way that it can be washed around by water in the rinsing chamber 115 in an assigned manner during a cleaning process. Mutual concealment of objects 125 can be avoided. Furthermore, an object 125 can be arranged such that liquid can advantageously drain from the object 125 at the end of the rinsing process.

[0042] In a step 255, an indication can be determined as to which objects 125 need to be rearranged and in what manner in order to achieve an improved arrangement of objects 125. The indication can be directed to a user of the dishwasher 105. Preferably, the indication is output to the user via a mobile device 110. In one embodiment, the indication can be graphical. For example, an animation can be provided showing how an object 125 should be moved relative to a dish rack 120 to achieve the improved arrangement. Reference symbol 100 systems 105 dishwashers 110 Mobile device 115 Rinse chamber 120 dish basket 125 objects 130 flap 135 spray arm 140 device 145 Processing facility 150 Camera 155 Communication device 200 procedures 205 scan first image 210 Determine the first position of the dish basket 215 scan second image 220 Determine the second position of the dish basket 225 Difference between positions sufficiently large? 230 Determine 3D model of the object 235 Classify object 240 Determine the arrangement of the object in the closed dishwasher 245 Determine the arrangement of multiple objects 250 determine optimized arrangement 255 Provide notice to user QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 8 696 827 B2

[0003] DE 10 2020 211 543 A1

[0004] US 2022 092 808 A1

[0005]

Claims

[1] Dishwasher (105), comprising: - a rinsing chamber (115); - a dish basket (120) which can be pulled out of the washing chamber (115) to hold an object (125); - a camera (150) for capturing an image of the object (125); and - a processing device (145) configured to determine three-dimensional information about the object (125) with respect to a first and a second image of the camera (150); - wherein the images are captured in different positions of the crockery basket (120) relative to the camera (150). [2] The dishwasher (105) of claim 1, wherein the washing chamber (115) has a reflective inner surface; wherein the camera (150) is configured to detect a reflection of the washing basket and / or the object (125) on the surface. [3] A system (100) comprising a dishwasher (105) according to claim 1 or 2 and a mobile device (110); wherein the dishwasher (105) is configured to transmit certain three-dimensional information to the mobile device (110); and wherein the mobile device (110) is configured to provide an indication of the information. [4] A method (200) for providing three-dimensional information about an object (125) in a dish basket (120) of a dishwasher (105) using a camera (150); the method (200) comprising the following steps: - capturing (205) a first image of the object (125) while the dish basket (120) is in a first position relative to the camera (150); - capturing (215) a second image of the object (125) while the dish basket (120) is in a second position relative to the camera (150); - the second position being different from the first position; - determining (230) the three-dimensional information relating to the images; and - Providing (255) the specific information. [5] The method (200) of claim 4, wherein a plurality of images of the object (125) are determined (205-225) from different positions of the dish basket (120) relative to the camera (150); and wherein the three-dimensional information is determined with respect to the plurality of images. [6] Method (200) according to one of claims 4 or 5, wherein the three-dimensional information is determined (230) by means of a neural radiation field. [7] Method (200) according to one of claims 4 to 6, wherein a predetermined optical feature of the dish basket (120) is detected; and wherein a position of the dish basket (120) relative to the camera (150) is determined (210, 220) with respect to the detected feature. [8] The method (200) of any one of claims 4 to 7, wherein at least one of the images is captured while the dish basket (120) is being pulled out of the washing chamber (115) or pushed into the washing chamber (115). [9] Method (200) according to one of claims 4 to 8, wherein the object (125) is classified (235). [10] Method (200) according to one of claims 4 to 9, wherein an arrangement of the object (125) in the washing chamber (115) which is improved with regard to cleaning of the object (125) in the dishwasher (105) is determined (250). [11] Method (200) according to one of the preceding claims, wherein three-dimensional information of a plurality of objects (125) is determined; wherein an optimized arrangement of the objects (125) in the rinsing chamber (115) is determined.

Images