Dishwasher with an image capture device and method for operating such a dishwasher
The method reduces computing power and storage needs by using reduced image resolution for object and contamination recognition in dishwashers, facilitating efficient contamination detection and optimized washing programs.
Patent Information
- Application Number
- DE102020112205
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-05-17
- Filing Date
- 2020-05-06
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2040-05-06
AI Technical Summary
Existing dishwashers with image capture devices require significant computing power and storage capacity for image evaluation, leading to increased costs or undesirable network data transmission.
A method involving image resolution reduction for object recognition, using smaller resolutions for initial detection and transferring high-resolution sections for detailed contamination identification, reducing computing power and memory requirements.
Enables efficient object and contamination recognition with lower hardware demands, allowing for optimized washing programs based on contamination detection.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The invention relates to a method for operating a dishwasher, in particular a household dishwasher, wherein the dishwasher serves to clean dishes in a wash cabinet and has an image capture device for capturing images of the wash cabinet. The invention further relates to such a dishwasher.
[0002] Dishwashers with image capture devices, for example a camera that provides images from the washing chamber, are known, for example, from the documents DE 100 48 081 A1 or EP 1 793 723 B1.
[0003] According to DE 100 48 081 A1, a camera image captured before or during a running cleaning program is evaluated to determine the degree of soiling of the washware. Depending on the degree of soiling, a computer-controlled wash program optimized for the soiling level can be set, or a set wash program can be optimized with regard to the soiling level. In addition to determining the soiling level, the load status can also be detected based on the captured images, which is taken into account when setting or optimizing the wash program. The images are evaluated by comparing them with stored images.
[0004] It is also known from EP 1 793 723 B1 to determine the load status of a dishwasher using an image capture device. The evaluation is based on a three-dimensional image created using an image capture system that takes light propagation times into account.
[0005] Both image analysis through comparison with stored images and the analysis of three-dimensional images require significant storage capacity and computing power for image analysis. To provide this computing power and / or storage capacity, the dishwasher can either be equipped with the appropriate computing power and / or storage capacity, or the analysis can be outsourced and performed by a remote analysis device accessible only via a network. The former increases the material requirements for the dishwasher, making it more expensive. In the latter case, data transmission over a network is required, which is undesirable either for security or privacy reasons, or due to the associated high data volume to be transmitted.
[0006] It is therefore an object of the present invention to provide a dishwasher of the type mentioned at the outset and a method for operating the same, in which an evaluation of recorded images from the washing chamber is possible with relatively low computing power and storage capacity.
[0007] This object is achieved by a method for operating a dishwasher and a dishwasher configured to carry out the method, having the features of the respective independent claim. Advantageous embodiments and further developments are the subject of the dependent claims.
[0008] A method according to the invention has the following steps: A first image of the wash cabinet is recorded at a first resolution. From the first image, a second image of the wash cabinet is generated at a second resolution that is lower than the first resolution. Then, at least one object in the second image and an image section surrounding the identified object are identified. Individual parts of the washware are preferably recognized as objects. Next, a high-resolution image section of the first image in the first resolution that corresponds to the image section is determined. In other words, the image section found and containing the at least one object is transferred back to the first high-resolution image in order to provide the image section in the high first resolution. Soiling on the recognized object is then identified in this high-resolution image section.
[0009] Due to the reduced resolution compared to the original image and thus smaller number of pixels, object detection can be performed with less computing power and less memory requirements. The reduction in the level of detail due to the reduced resolution does not in any way hinder the detection of washware objects that are large compared to the resolution. By transferring the image section containing the washware object to the original image, the high resolution is again available for the subsequent identification of the soiling, which is advantageous for detecting smaller soiling. Since object detection in this case is limited to small image sections, this is also possible with less computing power and less memory requirements.Image sections that are assigned to several different found washware objects can be processed successively, so that even if a plurality of found washware objects does not result in increased demands on computing power or storage capacity.
[0010] In an advantageous embodiment of the method, the first image is captured at the native resolution of the image capture device. The second resolution is preferably a factor of between approximately 3 and 10 smaller than the first resolution. The required storage space scales linearly with the resolution, and the required computing power (depending on the object recognition method) scales at least linearly with the resolution. The selected reduction in resolution significantly reduces the demands on the hardware used in an evaluation device, so that even smaller sequence controllers or processors of the same, which are used in dishwashers, can perform the image evaluation.
[0011] In a further advantageous embodiment of the method, conventional image recognition methods, e.g., edge detection, are used for object recognition and / or for identifying contamination. Alternatively or additionally, a neural network, preferably a self-learning neural network, can be used for object recognition and / or for identifying contamination.
[0012] In further advantageous embodiments of the method, the number of pixels of images to be evaluated can be further reduced by ignoring edge regions of the first image and / or the image section.
[0013] Based on the detected objects, the load status of the dishwasher can be advantageously determined. Preferably, the identified soils are also classified based on their size and / or appearance. Furthermore, the distribution of the identified soils can be determined.
[0014] Advantageously, a dishwasher washing program can then be selected or optimized based on the detected objects and / or the identified soiling.
[0015] A dishwasher according to the invention is designed to carry out such a method. The advantages mentioned in connection with the method result.
[0016] The invention is explained in more detail below using an exemplary embodiment and the accompanying figures. The figures show: Fig. 1 a schematic sectional view of a dishwasher; and Fig. 2 a representation of a washing chamber of a dishwasher with indicated image sections during the implementation of a method according to the invention.
[0017] Fig. 1 shows a schematic view of a dishwasher in a vertical section.
[0018] The dishwasher, which can be a household appliance as well as a machine for professional use, has a body 1 with a door 2, which surrounds a washing chamber 3. Within the washing chamber 3, there are usually several, here for example two, wash baskets 4 arranged one above the other. In addition, a Fig. 1 unspecified cutlery carrier is provided. Various items of washware 10 are arranged in the washware baskets 4, for example. In alternative embodiments of the dishwasher, a cutlery basket may be provided in one of the washware baskets 4 instead of or in addition to the cutlery carrier.
[0019] Each of the dishware baskets 4 is assigned a spray arm 5, which rotates during the dishwashing operation and sprays dishwashing liquid onto the dishware 10. In a lower area (base area) of the dishwasher, the dishwashing liquid collects in a dishwashing sump, also called a dishwashing liquor pot, where it can be heated and pumped to the spray arms 5. The dishwashing sump and components for circulating the dishwashing liquid are shown in the schematic drawing of the Fig. 1 not reproduced.
[0020] A control device 6 for controlling the dishwasher is located in the door 2 of the dishwasher. The control device 6 is designed as a sequence controller based on a microcontroller or microprocessor. A dosing unit 7 is also located in the door 2, with which the required cleaning agents can be automatically dosed into the wash cabinet 3 during a wash cycle.
[0021] In the illustrated embodiment, a camera 8 is integrated into the dosing unit 7 as an image capture device, which is directed into the wash cabinet 3 in order to capture an image of the wash cabinet 3. Furthermore, a light source 9 is arranged in the wash cabinet 3 and can be switched on as needed when an image of the wash cabinet 3 is captured by the camera 8. Alternatively or additionally, a light source can also be arranged in the area of the camera 8 itself, which light source is, for example, designed in a ring shape around a lens of the camera 8. The camera 8 is a two-dimensional recording camera. Alternatively, a three-dimensional recording camera can also be used, for example a TOF (time of flight) camera, which takes light travel times into account. A stereoscopically recording arrangement of two cameras is also considered an image capture device within the scope of the application.
[0022] In the exemplary embodiment shown, there is only one camera 8, which essentially records the dishes 10 sorted in the lower dishware basket 4. There may be a further camera, which is arranged, for example, above the illustrated camera 8 and is directed at the dishes 10 sorted in the upper dishware basket 4. In the exemplary embodiment shown, the recording direction (optical axis) of the camera 8 is horizontal. Other orientations of the camera 8 are conceivable. For a camera assigned to the upper dishware basket 4, it may be useful, for example, to align the camera so that it looks diagonally from below onto the upper dishware basket 4, since cups or vessels with the dirty surface facing downwards are often sorted into the upper dishware basket 4.
[0023] In Fig. 2 is a view into the washing chamber 3 of a dishwasher, for example the dishwasher of the Fig. 1, as it results from the viewing direction of the camera 8. A first image 20 captured by the camera 8, also called overall image 20, is represented by a corresponding frame in the Fig. 2. In order to detect the degree of soiling of the washware 10, the recorded image is subjected to various image processing steps in succession, which are explained in more detail below.
[0024] The overall image 20 is an image captured with the native camera resolution of camera 8 and has a number of pixels (picture elements) in the horizontal and vertical directions corresponding to the resolution of camera 8. The number of picture elements can be, for example, 1280x960 pixels.
[0025] Methods and algorithms for detecting contamination are generally known. These can be based on classical computational techniques such as edge detection and the recognition of shapes in the areas defined by the edges. Object detection using a preferably self-learning neural network is also conceivable.
[0026] According to the application, however, the methods for contamination detection are not performed in the entire area of the overall image 20. Instead, the area in which contamination detection is performed is reduced in size, possibly in several steps, and processed at different resolutions during the process.
[0027] In a first step, those areas are cut out of the captured overall image 20 in which no evaluable image of the wash ware 10 can be found. These are the areas that can be predetermined based on the geometry of the wash chamber 3 and the arrangement of the camera 8, in which the wash ware 10 either cannot be located or which cannot be meaningfully evaluated. In the example of the Fig. 2, sections of the overall image 20 in which the upper dish basket 4 and its associated spray arm 5 are positioned are ignored. Right and left edge areas located outside the lower dish basket 4 are also ignored. Thus, an image section 21 is extracted from the overall image 20 for further processing, which already has a significantly smaller area and thus pixel count, for example, only approximately 70% of the original area of the overall image 20.
[0028] In a second step, an image section with reduced resolution is created for image section 21. This image section represents a copy of image section 21 in terms of image content, but with a smaller number of pixels. For example, the resolution is reduced in both spatial directions by a factor of approximately 2 to 3, i.e., by a factor of 4 to 9 in terms of area.
[0029] Using this reduced-resolution image section, an initial object recognition process is then performed to identify objects representing individual washware items 10. Due to the reduced resolution of the image section used for this purpose compared to image section 21, this object recognition process can be performed with less computing power and less memory. The reduction in the level of detail due to the reduced resolution does not in any way hinder the recognition of washware objects that are large compared to the resolution.
[0030] As a result of the object detection, several washware objects are usually found, around which a rectangular frame is placed as an additional image section 22. In an optional further step, a defined edge area can be removed from this additional frame 22, the width of which can be fixed depending on the detected washware 10.
[0031] The area thus identified for a piece of washware 10 is then transferred to the higher-resolution original image 20 or section 21, so that for each detected washware object, a high-resolution image section 23 reproducing the washware 10 is available. For ease of differentiation, this image section is also referred to below as the object section 23. A second object recognition method is applied to each of these object sections 23 to identify soiling 11 on the respective washware 10. Object recognition with the aid of a preferably self-learning neural network is particularly advantageous in this step. The soiling 11 can be identified and classified based on its appearance (size, shape, color, etc.) and / or other image properties.
[0032] In a further optional step, contrast normalization can be carried out within the high-resolution object section 23, whereupon the extent to which the soiling is distributed on the washware is estimated by means of edge detection and / or histogram formation.
[0033] During operation of the dishwasher, information obtained from image analysis can be used to optimize the washing process. For example, before starting a washing program, an initial image of the wash cabinet 3 can be taken and evaluated using the method described above. The collected information, in particular the classification of soiling, the size of the soiled regions, and the distribution of soiling within the region, can then be used to suggest a washing program to the dishwasher user. This can be shown directly on a display or other indicator of the control device 6 or by means of a mobile device that communicates with the dishwasher via a data connection.When suggesting a washing program, the user's preferences can be taken into account, for example whether he or she would prefer to use a quick washing program or an ecologically sensible washing program.
[0034] After the start of the washing process, the circulation pump, which circulates the wash water, can be stopped at regular intervals to take another image of wash cabinet 3 and evaluate it again according to the previously described procedures. A comparison of the evaluation results allows for a determination of cleaning progress. If necessary, it may be possible to adaptively lengthen or shorten cleaning phases depending on the cleaning progress.
[0035] In addition to the image analysis, other recorded parameters can be taken into account, for example the turbidity level of the rinsing solution, which can be detected using a separate optical turbidity sensor. Reference symbol 1 corpus 2 doors 3 Washing-up area 4 Dishwasher basket 5 spray arm 6 Control device 7 Dosing unit 8 Camera 9 Light source 10 items to be washed 11 Pollution 20 first picture (overall picture) 21, 22 Image detail 23 high-resolution image section (object section)
Claims
[1] Method for operating a dishwasher for cleaning items (10) to be washed in a washing chamber (3), wherein the dishwasher has an image capture device for taking images of the washing chamber (3), comprising the following steps: - taking a first image (20) of the washing chamber (3) in a first resolution; - generating a second image of the washing chamber (3) from the first image in a second resolution which is smaller than the first resolution; - identifying at least one object in the second image and determining an image section (22) surrounding the identified object; - determining a high-resolution image section (23) of the first image corresponding to the image section (22) in the first resolution; and - Identifying contamination (11) on the identified object in the high-resolution image section (23). [2] Method according to claim 1, wherein the first image (20) is captured in the native resolution of the image capture device. [3] A method according to claim 1 or 2, wherein the second resolution is smaller than the first resolution by a factor of between about 3 and 10. [4] Method according to one of claims 1 to 3, in which parts of the items to be washed (10) are recognized as objects. [5] Method according to one of claims 1 to 4, in which classical image recognition methods, e.g. edge detection, are used for object recognition and / or for identifying the contamination (11). [6] Method according to one of claims 1 to 5, in which a neural network is used for object recognition and / or for identifying the contamination (11). [7] Method according to one of claims 6, in which a self-learning neural network is used for object recognition and / or for identifying the contamination (11). [8] Method according to one of claims 1 to 7, wherein edge regions of the first image (20) are disregarded when generating the second image. [9] Method according to one of claims 1 to 8, in which edge regions of the image section (22) are not taken into account when determining the high-resolution image section (23). [10] Method according to one of claims 1 to 9, in which a loading state of the dishwasher is determined on the basis of the detected objects. [11] Method according to one of claims 1 to 10, wherein the identified contaminants (11) are classified according to their size and / or their appearance. [12] Method according to claim 11, wherein a distribution of the identified contaminants (11) is determined. [13] Method according to one of claims 1 to 12, in which a rinsing program is selected or adapted based on the detected objects and / or the identified soilings (11). [14] Dishwasher for cleaning items (10) to be washed in a washing chamber (3), comprising an image capture device for taking images of the washing chamber (3), characterized by that the dishwasher is designed to carry out a method according to one of the preceding claims.
Citation Information
Patent Citations
Determining loading and degree of soiling of dishwashing machine contents, employs one or more image recognition systems
DE10048081A1
Method for detecting the load of items to be washed, and dishwasher machine
EP1793723B1
