A method of detecting and classifying dirt type and an autonomous cleaning robot implementing this method

Abstract

The invention relates to a method for detecting and classifying dirt, implemented through a computer using a unmanned autonomic cleaning robotic unit comparing set of optical sensors, mentioned method comprising step of processing data from RGB camera recording signals in the visible spectrum and data from a depth camera recording RGB signals with depth information providing RGB-D image The invention also relates to an autonomous cleaning robot containing an RGB-D camera and a processor configured to perform the steps of the method.

Description

[0001] A method of detecting and classifying dirt type and an autonomous cleaning robot implementing this method

[0002] Technical field

[0003] The present invention relates to the field of autonomous cleaning systems. More specifically, the present invention relates to a cleaning robot having a control system enabling automatic detection and classification of the type of soiling or dirt.

[0004] State of the art

[0005] As technology advances, smart home robots such as cleaning robots are becoming more and more popular. They make housework easier and allow you to significantly save time spent on cleaning. There are many types of autonomous cleaning robots on the market that perform specific tasks, during which the robot moves across the floor and at the same time absorbs dirt from the surface. Cleaning robots can function, for example, as a vacuum cleaner or a mop.

[0006] In the case of industrial and home cleaning robots, it is important to achieve the highest possible cleaning efficiency so that the consumption of resources such as water and chemicals used for cleaning is as low as possible. Moreover, cleaning robots often encounter various types of obstacles while working, for example thresholds, doors or furniture elements, therefore they should be equipped with a system that allows for effective avoidance of obstacles, as well as effective detection and classification of the type of dirt.

[0007] For example, EP3900604A1 discloses a cleaning robot including a lifting mechanism for lifting a cleaning module of the robot from a first position relative to the work surface to a second position. Thanks to this, the cleaning robot works better when the floor level changes or it encounters an obstacle - changing the height of the cleaning element prevents secondary contamination of the surface, for example when the robot moves from the floor level to the carpet, and also allows it to bypass small obstacles without changing direction driving.

[0008] Another patent document EP4072390A1 discloses a method for assessing the cleanliness of a robot cleaning head assembly, i.e. the level of wear or the level of contamination accumulation, allowing the remaining useful life of the cleaning head to be estimated based on the collected data.

[0009] Document US11287828B2, in turn, discloses a method for detecting obstacles for various types of moving robots, allowing for planning a safe and fast movement path. The robot is equipped with an RGB-D camera and a distance sensor electrically connected to the processor. The method basically includes: obtaining one or more images using a camera and recognizing an obstacle in them according to set parameters, wherein the set parameters of the obstacle include the size and hardness of the obstacle; obtaining information about the position of an obstacle via the distance sensor; linking the features of the obstacle image with information about its location; calculating the obstacle's movement state based on information about the obstacle's position at different times; estimating information about the position of an obstacle in the robot's detection blind zone based on the movement state; and planning the robot's path based on information about the position of the obstacle in the detection blind zone and in the robot's detection zone.

[0010] However, known autonomous floor cleaning systems have limited capabilities - they are unable to effectively clean various types of contaminants due to imperfect soil detection and classification systems, and they require manual adjustments to complete tasks. For example, even if the user manually adjusts the cleaning head to the optimal height (or the height is fixed at a given height as in document EP3900604A1), it may not be optimal for a given type of surface and given contaminants.

[0011] Conventional autonomous cleaning robots may also have problems cleaning dried and difficult-to- remove stains that require multiple passes of the robot over the surface because they are unable to check whether the stain has already been removed or whether another pass is required.

[0012] Another disadvantage of known cleaning robots is navigation based on the environmental map loaded in the robot's system, and the change of driving direction when encountering an obstacle that was not there before is based on reflection from a given obstacle or detecting it using sensors (e.g. . IR sensors) - this makes it difficult for the cleaning robot to move in a changing environment and may result in missing some areas of the surface during cleaning.

[0013] Another very important disadvantage of the devices known from the state of the art is the often ineffective detection and incorrect classification of the type of dirt. This is especially important in the case of transparent objects - especially when it is necessary to distinguish glass fragments from water. While water is a mess for the cleaning robot to clean up, larger glass shards are a dangerous obstacle because they can damage the cleaning head or even the entire cleaning robot. Therefore, it is important that the cleaning robot is equipped with a system that allows it to effectively distinguish spilled liquids (constituting dirt in the case of cleaning robots and an obstacle in the case of vacuuming robots) from smaller ones (constituting dirt in the case of vacuuming robots and an obstacle in the case of washing robots) and larger ones, (constituting a threat and an obstacle) glass fragments, so that it is possible to avoid these fragments and, as a result, reduce the risk to the robot that could result from contact with such fragments.

[0014] In addition, known control systems and algorithms used in cleaning robots are complicated and use high computing power, which translates into a high demand for electricity for cleaning robots. SUMMARY OF THE INVENTION

[0015] The invention relates to a method for detecting and classifying dirt, implemented through a computer using a unmanned autonomic cleaning robotic unit comparing set of optical sensors, mentioned method comprising step of processing data from RGB camera recording signals in the visible spectrum and data from a depth camera recording RGB signals with depth information providing RGB-D image , comprising the following steps:

[0016] A. inputting initial parameters regarding expected forms, including the shape of obstacles, surfaces, including the shape of walls and floors, and objects, including doors, windows, and stairs, expected dirt height;

[0017] B. obtaining cleaning robotic unit surrounding image in RGB color and an 3D RGB plus depth image using the RGB-D camera;

[0018] C. transforming the obtained depth image into a point cloud;

[0019] D. merging RGB color image and the RGB plus depth image;

[0020] E. identifying the shape of the floor and its position and level in accordance to the robotic unit and creating a binary mask containing pixels corresponding to the floor area;

[0021] F. Filtering out all points form the point cloud that are higher then the dirt height above the floor level.

[0022] G. identifying, possible floor optic anomalies using RGB color image;

[0023] H. identifying, possible floor height anomalies using point cloud data;

[0024] I. corelating anomalies from step G. and H.;

[0025] J. analyzing the identified floor area using a machine learning model, wherein the model is trained on a dataset comprising RGB-D images divided into subsets of images belonging to each of the types: clean floor, floor with a dirty area, floor covered with water, floor covered with shards of broken glass;

[0026] K. classifying the dirt type into having a heigh or flat;

[0027] M. Informing a human personnel of a dirt hazardous to the unmanned autonomic cleaning robotic unit with its location and estimated type of the dirt.

[0028] Preferably after step K. there is additional step of classification into easy to remove or dangerous to the cleaning robotic unit especially broken glass or paint spill. Preferably that the RGB image has a resolution of 4096x3072 pixels.

[0029] Preferably the depth image has a resolution of 512x512 pixels.

[0030] Preferably it utilizes the EfficientNet neural network architecture.

[0031] The invention also relates to an autonomous cleaning robot containing an RGB-D camera and a processor configured to perform the steps of the method according to any of claims 1 to 6.

[0032] This invention, thanks to the detection of dirt or liquids on the floor using a camera, allows you to adjust the resources used in each region of the floor and achieve the required cleaning results more effectively and in a shorter time. Additionally, the ability to detect glass shards allows you to avoid them and, as a result, reduce the risk to the robot that could result from contact with such shards, and information about the location of detected objects can be used by other robot systems responsible for planning its movement. Object detection is carried out in a sufficiently short time, which allows the acquired information to be used on an ongoing basis during cleaning. Thanks to the above features, the robot can make faster decisions and is less susceptible to failures. In addition, deep integration of Al algorithms with the dirt and obstacle recognition system allows for significant savings in computing power and significantly reduces the robot's demand for electricity.

[0033] Description of figures

[0034] The invention will now be described in more detail in a preferred embodiment with reference to the attached drawing, in which:

[0035] Fig. 1 shows a labeled photo from the created dataset with a marked area of the floor and areas of the floor with pieces of broken glass;

[0036] Fig. 2 shows a depth image obtained from a Kinect camera;

[0037] Fig. 3 shows the point cloud obtained by transforming the depth image;

[0038] Fig. 4 shows a mask defining the floor area;

[0039] Fig. 5 shows the input image of the neural network model;

[0040] Fig. 6 shows the output images of the neural network model, expressing the model's confidence for each pixel that it belongs to the class: clean, dirty, water-covered, or glass-covered floor.

[0041] Embodiment

[0042] The invention generally discloses a cleaning robot equipped with a control system that detects and classifies the type of dirt or type of floor within the robot's field of view using depth cameras (RGB-D) and artificial intelligence algorithms. The solution utilizes the correlation between color images and depth data, and includes deep integration of artificial intelligence algorithms in the dirt and obstacle recognition system, which pose a threat to the robot's operation, such as glass shards or water. This enables the robot to make quicker decisions and reduces the risk of failures. The deep integration of Al algorithms also allows for the omission of some control systems or algorithms used in robots known from the state of the art, conserving computational power and significantly reducing the robot's demand for electrical energy.

[0043] An autonomous cleaning robot equipped with the system according to the invention can be configured, for example, as an autonomous mop, vacuum cleaner, or another type of robot, such as one that combines two or more cleaning functions (e.g., vacuuming, sweeping, wiping, washing).

[0044] The developed invention is based on solutions known from the scientific literature, e.g.: R. Bormann, J. Fischer, G. Arbeiter, F. Weisshardt, and A. Verl, "A visual dirt detection system for mobile service robots"; A. Grunauer, G. Halmetschlager-Funek, J. Prankl, M. Vincze, "The power of gmms: Unsupervised dirt spot detection for industrial floor cleaning robots"; Ramalingam, V. Prabakaran, M. Ilyas, R. E. Mohan, and M. Arunmozhi, "Vision-based dirt detection and adaptive tiling scheme for selective area coverage"; Pathmakumar, M. Kalimuthu, M. R. Elara, and B. Ramalingam, "An autonomous robot-aided auditing scheme for floor cleaning." Various artificial intelligence algorithms that can be applied in the object of the invention are also known in the state of the art. This is a broad area of technology generally included in the knowledge possessed by experts in the field of artificial intelligence algorithms.

[0045] As mentioned, the system is based on dirt detection on the floor using depth cameras (RGB-D). This system, as described in the aforementioned literature, operates through image analysis from the RGB- D camera in two main steps:

[0046] 1. Finding the area representing the floor within the entire image. This step is accomplished in various ways, including the use of semantic segmentation of the RGB image or by finding the floor plane in the depth image.

[0047] 2. Analyzing the found floor area in the image to detect dirt. In this step, techniques such as image filtering and analysis using neural networks are applied.

[0048] In step 2, dirt can be detected in various ways, and the results of the analysis include, for example, binary masks indicating dirt positions or areas of the image containing the most dirt. The analysis is based on data processing and machine learning solutions.

[0049] Example Implementation

[0050] The architecture and operation of the system will be presented below as an example implementation. The system uses data from a depth camera (RGB-D camera). In this example, data from the Microsoft Azure Kinect camera were used. The system utilizes data obtained directly from the camera in the form of two images: a color image (RGB at a resolution of 4096x3072 pixels) and a depth image (D at a resolution of 512x512 pixels).

[0051] The images are processed using external libraries:

[0052] • Microsoft Azure SDK - development tools provided by Microsoft for processing images from the camera;

[0053] • PCL (Point Cloud Library) - a library for processing point clouds, used to analyze the processed depth image in order to detect the floor area in the entire image;

[0054] • OpenCV (Open Source Computer Vision Library) - a library of functions used for image processing, designed for working with 2D images;

[0055] • Tensorflow - a library used for implementing machine learning algorithms. The system was implemented in C++ and Python. ROS (Robot Operating System, a suite of software libraries and tools that help in building applications for robots) is also utilized to enable communication with other software systems of the robot.

[0056] Data processing requires the use of a computing unit on which the solution is implemented (this can be a unit performing parallel computations, such as a GPU). In the example implementation, an Nvidia AGX Xavier unit was used.

[0057] Preparing a functioning system required a preparatory step involving the creation of a dataset and the implementation of a system that performs analysis in two main steps. The dataset created consists of RGB-D images taken with the Microsoft Azure Kinect camera. The images depict various types of floors. The images are divided into subsets. The subsets contain specific types of images:

[0058] • clean floor,

[0059] • floor with a dirty area,

[0060] • floor covered with water, liquid spills

[0061] • floor covered with shards of broken glass.

[0062] All images in the dataset were manually annotated by marking the floor area, dirt, water, and broken glass. An annotated image from the created dataset, featuring the marked floor area and areas of the floor with shards of broken glass, is presented in grayscale in Fig. 1 (in practice, marked areas are preferably indicated in color, e.g., yellow for clean floor areas and red for areas with shards of broken glass).

[0063] Using the dataset, a neural network model was trained. The EfficientNet neural network architecture was used. It is possible to apply a different architecture, such as AlexNet, but the EfficientNet architecture is preferred due to its efficiency.

[0064] The implemented system performs the analysis of the input image in two steps:

[0065] 1. Identifying the floor area in the entire image. In this step, the depth image obtained from the Microsoft Azure Kinect camera is transformed into a point cloud. An example depth image captured by the Kinect camera is shown in Fig. 2. The point cloud obtained by transforming the depth image is presented in Fig. 3 - the floor is visible in the point cloud as a horizontal plane. In the point cloud, based on the known position of the camera mounted on the robot, the largest plane closest to the expected floor plane is found. Points belonging to the found plane are considered to correspond to the floor. In the color image associated with the depth image, pixels corresponding to the found floor points in the depth image are marked as pixels corresponding to the floor. The resulting binary mask, which marks the pixels corresponding to the floor region, is then subjected to morphological operations such as dilation and erosion to obtain a more coherent mask defining the floor area, which is the result of the analysis in step 1. The determined floor area resulting from the analysis in step 1 is shown in grayscale in Fig. 4 (where the determined floor area is indicated as a lighter spot).

[0066] 2. The identified floor area is analyzed using the trained machine learning model. The model takes a color image (RGB) as input, in which pixels not belonging to the identified floor area have been masked (based on annotations in the dataset). The input image for the neural network is presented in Fig. 5.

[0067] The model takes as input a color image (RGB) in which the pixels not belonging to the identified floor area have been masked (based on annotations in the dataset). The input image for the neural network is presented in Fig. 5. The model's output consists of up to four images (more may be included, e.g., 6, 8, depending on the proposed complexity of the model and other types of dirt that the model is intended to reveal), representing the model's confidence (i.e., the probability of correctly identifying the type of data, including the type of dirt or the type of object) for each pixel that it belongs to a given class among: clean floor, dirty floor, floor covered with water, floor covered with shards of glass. The images (graphical representations / projections of data) that constitute the neural network model's output are presented in Fig. 6.

[0068] More specifically, in Fig. 6, Fig. 6a refers to the confidence of the model that the pixels in this image belong to the class of clean floor. Thus, in Fig. 6a, the pixels with the brightest shade express 100% confidence that a given pixel belongs to the clean floor class, while the pixels with the darkest shade (within the area of the clean floor) express 0% confidence that a given pixel belongs to the clean floor class. The pixels with transitional shades express a degree of confidence between 0% and 100%. In Fig. 6b, the pixels with a lighter shade represent a greater degree of confidence from the model that a given pixel belongs to the dirty floor class, while the darker shaded pixels convey a lower degree of confidence that a given pixel belongs to the dirty floor class. In Fig. 6c, the pixels with a lighter shade indicate a greater degree of confidence from the model that a given pixel belongs to the class of floor covered with water, while the darker shaded pixels indicate a lower degree of confidence that a given pixel belongs to the class of floor covered with water. In Fig. 6d, the pixels with a lighter shade indicate a greater degree of confidence from the model that a given pixel belongs to the class of floor covered with shards of glass, while the darker shaded pixels signal a lower degree of confidence that a given pixel belongs to the class of floor covered with shards of glass. Water, oil, paint spills can be detected corelating tis height (mostly flat) and it level of light reflection. Different liquids have different reflections which is measured by the RGB camera. Similar solution lays with the powders, powder is safe to be cleaned until is has low particles but particles have heigh. Correlation of light reflection and height give a certain dirt type. Lage, reflective particles if translucent - glass, if not other particles type but still dangerous for the cleaning unit. Similar with the liquids reflective spill without height ( less that fraction of a cm) probably water or similar if no color, if less reflective an have some color possibly paint dangerous for some cleaning units.

Claims

Patent Claims1. A method for detecting and classifying dirt, implemented through a computer using a unmanned autonomic cleaning robotic unit comparing set of optical sensors, mentioned method comprising step of processing data from RGB camera recording signals in the visible spectrum and data from a depth camera recording RGB signals with depth information providing RGB-D image , comprising the following steps:A. inputting initial parameters regarding expected forms, including the shape of obstacles, surfaces, including the shape of walls and floors, and objects, including doors, windows, and stairs, expected dirt height;B. obtaining cleaning robotic unit surrounding image in RGB color and an 3D RGB plus depth image using the RGB-D camera;C. transforming the obtained depth image into a point cloud;D. merging RGB color image and the RGB plus depth image;E. identifying the shape of the floor and its position and level in accordance to the robotic unit and creating a binary mask containing pixels corresponding to the floor area;F. Filtering out all points from the point cloud that are higher then the dirt height above the floor level.G. identifying, possible floor optic anomalies using RGB color image;H. identifying, possible floor height anomalies using point cloud data;I. corelating anomalies from step G. and H.;J. analyzing the identified floor area using a machine learning model, wherein the model is trained on a dataset comprising RGB-D images divided into subsets of images belonging to each of the types: clean floor, floor with a dirty area, floor covered with water, floor covered with shards of broken glass;K. classifying the dirt type into having a heigh or flat, reflective or non-reflective, translucent of having a color;L. identifying a hazardous type of a dirt and avoiding cleaning this dirt and adding its location to the navigation warnings;M. Informing a human personnel of a dirt hazardous to the unmanned autonomic cleaning robotic unit with its location and estimated type of the dirt.

2. The method according to claim 1, characterized in that after step K. there is additional step of classification into easy to remove or dangerous to the cleaning robotic unit especially broken glass or paint spill.

3. The method according to claim 3, characterized in that the RGB image has a resolution of 4096x3072 pixels.

4. The method according to claim 3 or 4, characterized in that the depth image has a resolution of 512x512 pixels.

5. The method according to any of the preceding claims, characterized in that it utilizes the EfficientNet neural network architecture.

6. An autonomous cleaning robot containing an RGB-D camera and a processor configured to perform the steps of the method according to any of claims 1 to 6.

Images