Floor surface stain identification method, apparatus, and autonomous cleaning device

Abstract

This application discloses a method, apparatus, and autonomous cleaning device for identifying floor surface stains, the method comprising the steps of: acquiring an RGB image of a field of view; acquiring a depth image of the field of view; acquiring a common viewing area based on the RGB image and the depth image; and acquiring the spatial coordinates of a stained area based on the common viewing area. The scheme of this application can accurately detect the type and location of stains.

Description

Related Application

[0001] This application claims the priority of Chinese Patent Application No. 202311084777.6 filed on August 25, 2023, and the entire content of said Chinese patent application is incorporated herein by reference. Technical Field

[0002] The present application relates to the technical field of autonomous cleaning devices, and specifically relates to an obstacle avoidance method, an apparatus for an autonomous cleaning device, and an autonomous cleaning device. Background Art

[0003] Autonomous cleaning devices frequently encounter scenarios where the floor is contaminated during operation, and it is necessary to accurately detect the type and position of dirt on the floor to perform avoidance or cleaning. Generally, RGB image detection is the mainstream method for floor dirt detection. There are a wide variety of floor dirts, and currently most commercially available floor-sweeping autonomous cleaning devices only perform rough image clustering segmentation on the dirt in the image during cleaning, cannot determine the type of dirt, and cannot perform effective cleaning. Summary of the Invention

[0004] In view of the above situation, embodiments of the present application provide a floor dirt identification method, an apparatus, and an autonomous cleaning device that can accurately detect the type and position of dirt.

[0005] In a first aspect, embodiments of the present application provide a floor dirt identification method, the method comprising: acquiring an RGB image of a field of view, acquiring a depth image of the field of view, acquiring a common image visible area based on the RGB image and the depth image, acquiring spatial coordinates of a dirt area based on the common image visible area.

[0006] In a second aspect, embodiments of the present application provide a floor dirt identification apparatus, the apparatus comprising: A first identification module used to acquire a single field of view RGB image, A second identification module used solely for acquiring depth images of the aforementioned field of view, A pixel alignment module used to acquire a common viewing area of ​​an image based on RGB images and depth images, It includes a coordinate calculation module used to obtain the spatial coordinates of the soiled area based on the common viewing area of ​​the image.

[0007] In a third aspect, embodiments of the present application further provide an autonomous cleaning device comprising an autonomous cleaning device body and a floor surface dirt identification device that controls the autonomous cleaning device body, wherein the floor surface dirt identification device employs the floor surface dirt identification method provided in the first aspect.

[0008] The proposed solution enables an autonomous cleaning device to accurately detect the type and location of dirt. This is achieved through a floor surface dirt detection method based on RGB camera images, green eye auxiliary light, structural optical ranging (or RGBD, binocular, TOF), and spectral auxiliary identification. This method allows for accurate detection of dirt type and location through methods such as image feature enhancement by green eye, dirt identification model detection, multi-sensor fusion ranging, and spectral image identification. After the autonomous cleaning device identifies the type of dirt in a dirty area, it controls cleaning equipment capable of processing that type of dirt to perform the corresponding processing, thereby achieving accurate matching. [Brief explanation of the drawing]

[0009] The accompanying drawings described herein provide a further understanding of this application and constitute part of this application. Exemplary embodiments and descriptions thereof are used for interpreting this application and do not constitute an unreasonable limitation of this application. In the accompanying drawings, [Figure 1] This is a schematic flowchart illustrating a floor surface stain identification method according to the first embodiment of this application. [Figure 2] This is a schematic diagram showing a floor surface stain identification device according to a second embodiment of this application. [Modes for carrying out the invention]

[0010] To further clarify the purpose, technical solution, and merits of this application, the technical solution of this application will be clearly and completely described below with reference to specific embodiments of this application and corresponding accompanying drawings. Clearly, the embodiments described are only a selection of embodiments of this application, not all embodiments. Any other embodiments that can be obtained by a person skilled in the art without creative work based on the embodiments of this application are all included within the scope of protection of this application.

[0011] The technical solutions provided by each embodiment of this application will be described in detail below with reference to the attached drawings.

[0012] Autonomous cleaning devices, also known as automatic vacuum cleaners, autonomous cleaning devices, smart vacuum cleaners, autonomous cleaning device dust collectors, and mobile autonomous cleaning devices, are a type of smart home appliance that utilizes artificial intelligence to automatically complete floor cleaning tasks within a room. They generally employ a brush cleaning and vacuuming method, first sucking debris from the floor into their own dust collection box to complete the floor cleaning function. Generally, cleaning devices that automatically complete cleaning, dusting, and floor wiping tasks are also classified as autonomous cleaning devices.

[0013] Due to the wide variety of floor stains, conventional methods generally only roughly cluster the stains in the image, failing to determine the type of stain and thus making effective cleaning impossible. Simultaneously, the operating environment of autonomous cleaning devices is complex, with lighting changes significantly impacting RGB images, affecting stain detection effectiveness and image depth calculation.

[0014] In light of these issues, this application provides a floor surface dirt detection method based on RGB camera images + green eye auxiliary light + structural optical ranging (or RGBD, binocular, TOF) + spectral auxiliary identification. Through methods such as enhancement of image features by green eye, detection by dirt identification model, ranging by multi-sensor fusion, and spectral image identification, the type and location of dirt can be accurately detected. After the autonomous cleaning device identifies the type of dirt in the dirty area, it controls cleaning equipment capable of processing that type of dirt to perform the corresponding processing, achieving accurate matching.

[0015] The concept of this application is as follows: First, an RGB image is input to a target detection model, and the pixel positions of dirt are marked on the RGB image. Next, in the spatial identification section, 1) a depth camera generates a depth image, each image containing depth information, which is converted into 3D spatial coordinates. 2) Structural optical ranging (a line laser obstacle avoidance method currently used in equipment) is performed, the structural light generates a grayscale image through a grayscale camera, the pixel positions of the structural light are found within the grayscale image, and then the 3D coordinates of the pixel positions corresponding to the structural light are calculated based on the internal and external parameters of the structural light and the grayscale camera. Finally, the RGB image, depth image, and structural light grayscale diagram are superimposed on a common field of view based on internal and external parameters, so that the RGB identification result (a box for one target detection) can acquire depth information, further identifying the 3D coordinates of the dirt. After acquiring these 3D coordinates, if the z-axis of the 3D coordinates is 0, it is considered the floor surface, satisfying the condition; if the z-axis is not 0, it is considered a misidentification and discarded.

[0016] This application detects soiled areas using RGB image recognition (monocular or binocular) and accurately identifies the type and contour of the soiling through target detection or semantic segmentation.

[0017] This application further uses depth measurement to accurately determine distance information to an autonomous cleaning device for a soiled area, and specifically includes: 1) Use depth information from a depth camera alone (binocular stereoscopic camera or ToF), 2) Ranging is performed on the target area by using structured light alone, 3) The depth information of the depth camera and the structured light ranging are fused.

[0018] The present application uses a green eye / infrared supplementary lamp to enhance the texture of a stained area, improve the feature identification degree of floor stains, and also acts as auxiliary light in dark environments.

[0019] By adopting the solution of the present application and using spectral image stain identification, not only the stain identification result and position are more accurate, but also the dry-wet state and turbidity of stains can be identified, which provides information for the cleaning strategy of an autonomous cleaning device.

[0020] Currently, there are three main types of mainstream depth cameras: structured light, time-of-flight, and binocular stereo.

[0021] The basic principle of a structured light depth camera is as follows: the main hardware includes a projector and a camera. The projector actively emits IR infrared light invisible to human eyes onto the surface of a target (which is called active measurement), then one or more cameras photograph the target to collect structured light images, transmit the data to a computing unit, and calculate and obtain position and depth information according to the triangulation principle, so as to realize 3D reconstruction. As the name suggests, structured light is structured light, and there are various projection pattern methods, such as the phase-shift method of sine wave fringes, binary-coded gray code, and phase-shift method + gray code.

[0022] The basic principle of a time-of-flight (TOF) depth camera is as follows: TOF is the time-of-flight method. It irradiates a target with continuous light pulses, after the light is reflected by the target, a sensor receives the reflected light, records the flight time, and calculates the distance to the target. TOF depth cameras are generally divided into two types: pulse modulation and continuous wave modulation. Since they actively emit pulsed light, they belong to the active measurement mode. Unlike binocular cameras and structured light, which require algorithm processing to output three-dimensional data, TOF can directly output the three-dimensional data of a target.

[0023] Basic principle of binocular stereo depth camera: image information is acquired by two left and right cameras, and parallax is calculated. Since a binocular camera does not actively emit light sources to the outside, it is called a passive depth camera.

[0024] Optionally, the present application can identify floor stains through the solution of an RGB camera, a depth camera (RGBD or binocular stereo camera or TOF), and a green eye / infrared supplementary lamp.

[0025] Optionally, the present application can identify floor stains through the solution of an RGB camera, an infrared camera plus structured light, and a green eye / infrared supplementary lamp.

[0026] Optionally, the present application can identify floor stains through the solution of an RGB camera, a depth camera (RGBD or binocular stereo camera or TOF), an infrared camera plus structured light, and a green eye / infrared supplementary lamp.

[0027] Optionally, the present application can identify floor stains through the solution of an RGB camera, a spectral camera, a depth camera (RGBD or binocular stereo camera or TOF), an infrared camera plus structured light, and a green eye / infrared supplementary lamp.

[0028] An RGB-D depth camera (also called a 3D camera, where D stands for Depth, representing depth information) can acquire distance information from an object to the camera. In addition to the X and Y coordinates on a 2D plane, it can calculate the three-dimensional coordinates of each point, thereby enabling inference for applications of depth cameras, such as three-dimensional reconstruction, target positioning, and identification.

[0029] The following embodiments are described in detail based on the above optional solutions.

[0030] FIG. 1 is a schematic flowchart showing the floor stain identification method according to the first embodiment of the present application. As can be seen from FIG. 1, the present application includes at least steps S110 to S140.

[0031] Step S110: Acquire an RGB image of one field of view.

[0032] The step of acquiring an RGB image of one field of view further includes acquiring a dirty area based on the RGB image.

[0033] Based on an RGB image collected by an RGB camera, stain features in the RGB image are obtained, and these stain features include stained areas and stain types. An RGB image is obtained from an RGB camera, preprocessing is performed on the RGB image such as distortion correction, cropping, and aspect ratio adjustment, and the image is input into a trained stain identification model to perform tasks such as image target detection and semantic segmentation to obtain stain types and the coordinate values ​​of stained areas in the image coordinate system (pixel positions of stained areas). In possible embodiments of this application, the step of obtaining stain features in an RGB image based on an RGB image collected by an RGB camera further includes preprocessing the RGB image and obtaining a preprocessed image, the preprocessing method including, but not limited to, image distortion correction, cropping, and aspect ratio adjustment. A trained stain identification model is invoked, and the processed image is input into the stain identification model to obtain the pixel positions of the stains and stain types. Based on the pixel positions (x, y) of the stains, stained areas are further obtained.

[0034] Step S120: A depth image of the field of view is acquired.

[0035] In this embodiment, the depth image of the field of view is acquired by a depth camera or a structural optical infrared camera, or by a combination of both.

[0036] The acquisition of the depth image of the aforementioned field of view by a depth camera includes the following steps: Based on the aforementioned depth image, a depth image including only the floor surface is acquired. Specifically, this includes: Based on the internal and external parameters of the depth camera and the pixel depth values ​​of the depth image, a floor image of the depth image is acquired. The above step, "Obtaining a floor image from the depth image," further includes removing obstacle images from the floor image and obtaining the floor image.

[0037] In this embodiment, the depth image of the field of view is acquired by a structural optical infrared camera, which includes the following steps: A structural light grayscale image collected by a structural light infrared camera is acquired, the structural light grayscale image is converted into a depth image, and a depth image including only the floor surface is acquired based on the distance measurement information of the structural light grayscale image.

[0038] In this embodiment, a depth image of the field of view is acquired by combining the two, and further, a depth image including only the floor surface acquired based on the depth image is acquired. The specific steps are as follows: A depth image of the field of view collected by the depth camera is acquired, and a first floor image of the depth image is acquired based on the internal and external parameters of the depth camera and the pixel depth values ​​of the depth image. A structural light grayscale image is acquired using a structural light infrared camera, the structural light grayscale image is converted into a depth image, and a second floor image of the depth image converted from the structural light grayscale image is acquired based on the distance measurement information of the structural light grayscale image. The first floor image and the second floor image are superimposed, the difference in depth values ​​of corresponding pixel points is calculated, and if the difference in depth values ​​exceeds a threshold, the image depth of the pixel point is discarded. If the difference in depth values ​​is within the threshold range, the average pixel depth of the pixel point is obtained, and a depth image containing only the accurate floor surface is obtained.

[0039] Step S130: Obtain the common viewing area of ​​the image based on the RGB image and depth image.

[0040] The step of obtaining a common viewing area for the image based on the aforementioned RGB image and depth image is further: To acquire the dirty areas based on the aforementioned RGB image, The process includes pixel alignment of a depth image containing only the soiled area and the floor surface, and obtaining a common viewing area for the image, wherein all pixel points of the soiled area in the common viewing area correspond one-to-one with all pixel points of the soiled area in the depth image.

[0041] In possible embodiments of this application, a depth image is obtained that includes only the floor surface in the depth image, based on depth images collected by a depth camera, and the depth camera may be a binocular stereoscopic camera or a time-of-flight (TOF) depth camera.

[0042] The depth image is a depth image collected by a depth camera, which provides the depth of all locations within the camera's visible range, and based on internal and external parameters, it finds and marks all floor surface portions that the depth camera can see to obtain an image containing only the floor surface.

[0043] Step S140: The spatial coordinates of the stained area are obtained based on the common viewing area of ​​the image.

[0044] The step of obtaining the spatial coordinates of the stained area based on the aforementioned common viewing area of ​​the image further includes: The spatial coordinates of the stained area are obtained based on the pixel depth value and pixel position of the pixel points in the stained area of ​​the aforementioned common viewing area of ​​the image.

[0045] In possible embodiments of this application, all pixel points in the common viewing area of ​​the image, obtained by aligning the pixel positions of the dirt area acquired from the depth image (which includes only the floor surface) and the RGB image, correspond to all pixel points in the depth image. The alignment operation between the RGB image and the depth image is specifically achieved through the following steps: obtain the intrinsic parameter matrix of the RGB camera, the intrinsic parameter matrix of the depth camera, and the external parameter rotation matrix and offset matrix of the RGB camera and the depth camera; scan the coordinates of each pixel point in the depth image; and calculate the coordinates of the pixel points in the RGB image corresponding to the pixel points in the depth image using a formula (see prior art).

[0046] The step of obtaining the spatial coordinates of dirt based on the pixel depth value and pixel position of the pixel points in the dirt area of ​​the aforementioned common viewing area of ​​the image allows for the calculation of the distance from each pixel point to the camera, i.e., the pixel depth value, from the image values ​​of the depth image. Depending on the camera parameters, each depth value can correspond to one corresponding depth distance. After aligning the pixel positions of the depth image and the RGB image, all pixel points in the RGB image correspond to all pixel points in the depth image. Therefore, based on the image position and the depth information corresponding to the pixel point, the corresponding 3D coordinates in the world coordinate system can be obtained for each image in the image.

[0047] Convert the spatial coordinates of the soiled area in the image coordinate system to the spatial coordinates of the soiled area in the autonomous cleaning device coordinate system.

[0048] Based on the real-time orientation of the autonomous cleaning device, the spatial coordinates of the soiled area under the autonomous cleaning device's coordinate system are converted to the spatial coordinates of the soiled area under the world coordinate system.

[0049] When acquiring RGB images, illuminating the surface and edges of the dirt with a green eye / infrared supplement lamp enhances the texture information of the dirt area, improving the RGB image features, while also enhancing the spectral features of the dirt area. In dark conditions, it acts as a supplementary light, allowing the equipment to see the contours of obstacles and improving spectral discrimination.

[0050] In the first embodiment of this application, floor stains can be identified by an RGB camera, a depth camera (RGBD or binocular stereoscopic camera or TOF), and a green eye / infrared light supplementing lamp. In the first embodiment of this application, floor stains can be identified by an RGB camera, a structural light infrared camera, and a green eye / infrared light supplementing lamp. In the first embodiment of this application, floor stains can be identified by an RGB camera, a depth camera (RGBD or binocular stereoscopic camera or TOF), an infrared camera + structural light, and a green eye / infrared light supplementing lamp.

[0051] The floor stain identification method according to the second embodiment of this application includes at least the following steps: Acquire an RGB image of one field of view.

[0052] The step of acquiring an RGB image of one field of view further includes acquiring RGB stain regions based on the RGB image.

[0053] In this embodiment, dirt features in an RGB image are obtained based on an RGB image collected by an RGB camera, and these dirt features include the type of dirt and the area of ​​the dirt. The RGB camera may be a monocular or binocular camera.

[0054] In possible embodiments of this application, the steps of acquiring an RGB image of a single field of view and acquiring a stained area based on the RGB image are further: The process includes preprocessing the RGB image and obtaining a preprocessed image, and the preprocessing method includes, but is not limited to, image distortion correction, cropping, and aspect ratio adjustment.

[0055] A trained stain identification model is invoked, and the processed image is input to the stain identification model to obtain stain features. These stain features specifically include the type of stain and the stained area. The stained area is obtained by the pixel position of the stain, and the pixel position of the stain is the coordinate value of the stain in the image coordinate system.

[0056] Furthermore, the process includes inputting the processed image into the stain identification model and executing a target detection algorithm, a semantic segmentation algorithm, or an image segmentation algorithm to obtain the stain features, where the stain features are the type of stain and the coordinates of the stain in the image coordinate system. For example, if the parameter value calculated by the stain identification model is 0.88, it is determined to be a coffee stain; if the parameter value is 0.86, it is determined to be a water stain; and if the parameter value is 0.7, it is determined to be a yogurt stain.

[0057] A spectral image of the aforementioned field of view is acquired, The step of "acquiring a spectral image of the field of view" further includes acquiring spectral contamination regions based on the spectral image.

[0058] In this embodiment, stain characteristics of a spectral image can be obtained based on a spectral image collected by a spectral camera, and these stain characteristics include the type of stain and the pixel position of the stain.

[0059] In possible embodiments of this application, the steps of acquiring a spectral image of the field of view and acquiring a contaminated area based on the spectral image are further: The spectral cluster is obtained by clustering the spectra of each pixel point in the spectral image. The process includes calling a spectral identification model and inputting each spectral cluster into the spectral identification model to obtain stain characteristics, wherein the stain characteristics specifically include stain regions and stain types, and the stain regions specifically represent the coordinate values ​​of the stain in the image coordinate system. Exemplarily, in the case of cola stains on a patterned tile floor, the stain morphology is determined based on the spectral intensity at different pixel positions.

[0060] The common area between the contaminated region obtained based on the RGB image and the contaminated region obtained based on the spectral image is obtained, and this common area is the selected contaminated region.

[0061] The RGB identification result is modified using the spectral identification result, specifically including: obtaining RGB stain regions based on the RGB image; obtaining a spectral image of the field of view; obtaining spectral stain regions based on the spectral image; obtaining the common portion of the RGB stain regions obtained based on the RGB image and the spectral stain regions obtained based on the spectral image, and the common portion is a stain region.

[0062] Spectral image-based stain identification not only provides more accurate stain identification results and location, but also allows for the identification of the stain's wet / dry state and turbidity, providing valuable information for the cleaning strategy of autonomous cleaning devices.

[0063] The stained areas acquired based on the spectral image and the stained areas acquired based on the RGB image are aligned, a common viewing area RGB image (common portion) is acquired, and using the RGB image as a reference, the common viewing area of ​​the spectral image and RGB image is aligned using the internal and external parameters of the RGB camera and spectral camera, and the RGB identification result is corrected using the spectral identification result. The correction includes acquiring the common portion from the identification area and secondary verification of the identification category label. The RGB identification result becomes more accurate, the final result is obtained, and the result is converted to the RGB image coordinate system.

[0064] In possible embodiments of this application, the step of "obtaining the intersection of the stained area obtained based on the RGB image and the stained area obtained based on the spectral image" is further: The common visibility region is obtained from the stained areas acquired based on the spectral discrimination model and the stained areas acquired based on the stain identification model. The common visibility region is the intersection of the stained areas acquired by the spectral discrimination model and the stained areas acquired by the stain identification model. The identification type label model is called, and the agreement between the stain type obtained based on the spectral identification model and the stain type obtained based on the stain identification model is determined. If the types of stains on both items are the same, select that type of stain; otherwise, do not select that type of stain.

[0065] In this application, the steps of "acquiring an RGB image of one field of view" and "acquiring a spectral image of the field of view" are not limited in order. The steps from "acquiring an RGB image of one field of view" to "acquiring the common portion of the dirty area acquired based on the RGB image and the dirty area acquired based on the spectral image" constitute the image identification part. The step of "acquiring an RGB image of one field of view" is the process of acquiring the RGB identification result. The step of "acquiring a spectral image of the field of view" is the process of acquiring the spectral identification result. The step of "acquiring the common portion of the dirty area acquired based on the RGB image and the dirty area acquired based on the spectral image" is the fusion process of RGB and spectrum. The spectral identification result is used to modify the RGB identification result to obtain the final result. The final result is converted to the RGB image coordinate system. The fusion of RGB and spectrum is performed under the image coordinate system, and after identification in cooperation with the spatial identification module, it is converted to the autonomous cleaning device coordinate system.

[0066] A depth image of the aforementioned field of view is acquired.

[0067] The step of acquiring a depth image of the field of view further includes acquiring a depth image including only the floor surface based on the depth image. In this embodiment, the depth image of the field of view is acquired by fusing a depth camera and a structural optical infrared camera, thereby acquiring a depth image including only the floor surface based on the depth image, and the specific steps are as follows: A depth image of the field of view collected by the depth camera is acquired, and a first floor image of the depth image is acquired based on the internal and external parameters of the depth camera and the pixel depth values ​​of the depth image. A structural light grayscale image collected by a structural light infrared camera is acquired, the structural light grayscale image is converted into a depth image, and a second floor image of the depth image converted from the structural light grayscale image is acquired based on the distance measurement information of the structural light grayscale image.

[0068] The mounting positions of the structural light and grayscale camera are fixed, and the emission angle of the light emitted by the structural light is also fixed. There is a one-to-one correspondence between the position where the structural light hits an obstacle (3D coordinates under the camera coordinate system) and the image on the structural light grayscale image. In other words, 3D coordinates are obtained, and these 3D coordinates are the distance measurement information. For example, if the z-axis of the 3D coordinates is 0 (or near 0 in actual use), it is determined to be the floor, and if it is greater than 0, it is determined to be an obstacle.

[0069] The step of converting the structured light grayscale image to a depth image further includes turning on the structured light, acquiring a grayscale image with structured light information from an infrared camera, calculating distance information from the autonomous cleaning device to the floor or obstacle using the infrared camera and structural light calibration parameters, and marking the depth information in the grayscale image.

[0070] The first floor image and the second floor image are superimposed, the difference in depth values ​​of corresponding pixel points is calculated, and if the difference in depth values ​​exceeds a threshold, the image depth of the pixel point is discarded. If the difference in depth values ​​is within the threshold range, the average pixel depth of the pixel point is obtained, and a depth image containing only the accurate floor surface is obtained.

[0071] The depth image provided by the depth camera provides the depth of all locations within the camera's visible range, and based on internal and external parameters, it is possible to find and mark all parts of the floor that the camera can see and obtain an image that includes only the floor.

[0072] The depth camera in this application is selected from one of the following: RGBD, a binocular stereoscopic camera, or a Time-of-Flight (TOF).

[0073] In another possible embodiment, the depth image of the field of view is acquired solely by a depth camera, and the step of acquiring a depth image including only the floor based on the depth image is further: A depth image of the field of view collected by the depth camera is acquired, and a floor image of the depth image is obtained based on the internal and external parameters of the depth camera and the pixel depth values ​​of the depth image.

[0074] The depth camera can employ either a dual-lens stereoscopic camera or a time-of-flight (TOF) depth camera.

[0075] The process involves acquiring a grayscale image collected by a structural light infrared camera, converting it into a depth image, and specifically including: turning on structural light, acquiring a grayscale image with structural light information from the infrared camera, calculating distance information from the autonomous cleaning device to the floor surface or obstacle using calibration parameters of the infrared camera and structural light, marking it in the grayscale image, and marking the depth information. The grayscale image is converted into a depth image (only the portion corresponding to the structural light has depth), and then fused with an RGB image to provide depth information.

[0076] When adopting a fusion scheme of a depth camera and a structural optical infrared camera, based on the above steps, a grayscale image collected by the structural optical infrared camera is acquired, converted into a depth image, the two are pixel-aligned, the difference in depth values ​​of corresponding pixel points is found, a relatively accurate depth image is obtained, and when adopting a scheme of a depth camera and a structural optical infrared camera, secondary verification is performed to further improve the accuracy of the depth image of the field of view. Specifically, the first floor image and the second floor image are superimposed, the difference in depth values ​​of corresponding pixel points is calculated, if the difference in depth values ​​exceeds a threshold, the image depth of that pixel point is discarded, and if the difference in depth values ​​is within the threshold range, the average pixel depth of the pixel point is obtained, and a more accurate depth image containing only the floor surface is obtained.

[0077] The common viewing area of ​​an image is obtained based on the RGB image and depth image.

[0078] The step of obtaining a common viewing area for images based on an RGB image and a depth image further involves pixel alignment of the depth image, which includes only the floor surface, and the dirt area obtained from the RGB image, obtaining a common viewing area for images, and ensuring that all pixel points of the dirt area in the common viewing area for images correspond one-to-one with all pixel points of the dirt area in the depth image.

[0079] Pixel alignment is a process that uses an RGB camera as a reference and utilizes internal and external camera parameters to transform depth images and structural light grayscale images into the RGB camera coordinate system, thereby obtaining a common viewing area for the image.

[0080] A depth camera generates depth images, each image containing depth information, and each image in the depth image is converted into a 3D spatial coordinate system.

[0081] The spatial coordinates of the stained area are obtained based on the pixel depth value and pixel position of the pixel points in the stained area within the common viewing area of ​​the image.

[0082] Convert the spatial coordinates of the soiled area under the image coordinate system to the spatial coordinates of the soiled area under the autonomous cleaning device coordinate system.

[0083] For dirt on the floor surface, floor position information is assigned to the dirt and converted to the autonomous cleaning device coordinate system. Here, floor position information can be obtained from the 3D coordinates of each pixel point on the floor surface under the camera coordinate system.

[0084] Based on the real-time orientation of the autonomous cleaning device, the spatial coordinates of the soiled area under the autonomous cleaning device's coordinate system are converted to the spatial coordinates of the soiled area under the world coordinate system.

[0085] The system acquires the real-time orientation of the autonomous cleaning device, converts the coordinates of the soiled area in the autonomous cleaning device's coordinate system to coordinates in the world coordinate system, and marks it on the cleaning map.

[0086] When acquiring RGB images, illuminating the stained surface and edges with a green eye / infrared supplement lamp enhances the texture information of the stained area, improving the RGB image features, while also enhancing the spectral features of the stained area. In dark conditions, it acts as auxiliary light, allowing the equipment to see the contours of obstacles and improving spectral identification. The second embodiment of this application can identify floor stains using an RGB camera, spectral camera, depth camera (RGBD or binocular stereoscopic camera or TOF), structural light + infrared camera, and green eye / infrared supplement lamp.

[0087] Figure 2 is a schematic diagram showing a floor surface stain identification device according to a third embodiment of this application, and the floor surface stain identification device is applied to an autonomous cleaning device, and as can be seen from Figure 2, the floor surface stain identification device 300 is A first identification module 310 used to acquire an RGB image of one field of view, A second identification module 320 used to acquire a depth image of the field of view, A pixel alignment module 330 used to acquire the common viewing area of ​​RGB images and depth images, The system includes a coordinate calculation module 340 used to obtain the spatial coordinates of the soiled area based on the common viewing area of ​​the image.

[0088] In possible embodiments, the pixel alignment module 330 further acquires a dirty area based on the RGB image, acquires a depth image including only the floor surface based on the depth image, performs pixel alignment on the dirty area and the depth image including only the floor surface, acquires the common viewing area of ​​the image, and establishes a one-to-one correspondence between all pixel points of the dirty area in the common viewing area and all pixel points of the dirty area in the depth image.

[0089] In possible embodiments, the coordinate calculation module 340 further obtains the spatial coordinates of the stained area based on the pixel depth value and pixel position of the pixel points of the stained area in the common viewing area of ​​the image.

[0090] In a possible embodiment, the first identification module 310 further preprocesses the RGB image, obtains a preprocessed image, invokes a trained stain identification model, and inputs the preprocessed image into the stain identification model to obtain the stained areas.

[0091] In possible embodiments, the pixel alignment module 330 further acquires a spectral image of the field of view, acquires a dirty area based on the spectral image, acquires the intersection of the dirty area acquired based on the RGB image and the dirty area acquired based on the spectral image, and the intersection is the selected dirty area.

[0092] In possible embodiments, the step of calling the trained stain identification model and acquiring the stained area further includes: The preprocessed image is input to the stain identification model, and a target detection algorithm, a semantic segmentation algorithm, or an image splitting algorithm is executed to obtain the stain type and the stained area, where the stained area is the coordinate of the stain in the image coordinate system.

[0093] In possible embodiments, the second identification module 320 further clusters the spectrum of each pixel point in the spectral image to obtain spectral clusters, A spectral identification model is invoked, and each spectral cluster is input to the spectral identification model to obtain stain characteristics, which specifically include stain regions and stain types.

[0094] In possible embodiments, the second identification module 320 further calls an identification type label model, determines the consistency between the stain type obtained based on the spectral identification model and the stain type obtained based on the stain identification model, selects the stain type if the two stain types are the same, and does not select the stain type if the two stain types are not the same.

[0095] In possible embodiments, the coordinate calculation module 340 is used to convert the spatial coordinates of the soiled area under the image coordinate system to the spatial coordinates of the soiled area under the autonomous cleaning device coordinate system, and to convert the spatial coordinates of the soiled area under the autonomous cleaning device coordinate system to the spatial coordinates of the soiled area under the world coordinate system, based on the real-time orientation of the autonomous cleaning device.

[0096] Furthermore, the above-described floor stain identification device can implement each of the floor stain identification methods of the first embodiment described above, and the same technical features as those of the floor stain identification method will not be repeated here.

[0097] This application further provides a fourth embodiment of an autonomous cleaning device, the autonomous cleaning device comprising an autonomous cleaning device body and a controller for controlling the autonomous cleaning device body, the controller employing the floor surface dirt identification method of the first embodiment described above.

[0098] The above-described apparatus corresponds to the method embodiment and has the same technical effects as the method embodiment; a detailed explanation should be found in the method embodiment. The apparatus embodiment is derived from the method embodiment, and a detailed explanation should be found in the section of the method embodiment, which will not be repeated here. Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of the embodiments, and that the modules or flows in the accompanying drawings are not necessarily essential for carrying out this disclosure.

[0099] Those skilled in the art will understand that the modules in the apparatus of the embodiment may be distributed throughout the apparatus of the embodiment as described in the embodiment, and may be modified accordingly and placed in one or more different apparatuses. The modules of the above embodiment may be integrated into a single module, or further divided into multiple submodules.

[0100] Finally, the above embodiments are used solely to illustrate the technical solutions of the present disclosure and are not intended to limit them. While the present disclosure has been described in detail with reference to the above embodiments, it should be understood that those skilled in the art can still modify the technical solutions described in the above embodiments or make equivalent substitutions to some of their technical features, and that such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. The steps include acquiring RGB images of the field of view, The steps include acquiring a depth image of the aforementioned field of view, The steps include acquiring a common viewing area for the image based on the RGB image and the depth image, A method for identifying floor surface stains, comprising the step of obtaining the spatial coordinates of a stained area based on the aforementioned common viewing area of ​​the image.

2. The floor surface stain identification method according to claim 1, wherein the step of acquiring an RGB image of the field of view further includes acquiring a stained area based on the RGB image.

3. The step of acquiring the RGB image of the aforementioned field of view further includes: Based on the aforementioned RGB image, the RGB contamination region is acquired, The spectral image of the field of view is acquired, and the spectral contamination region is acquired based on the spectral image, A method for identifying floor surface stains according to claim 1, comprising obtaining a common portion between an RGB stain region obtained based on the RGB image and a spectral stain region obtained based on the spectral image, wherein the common portion is a stain region.

4. The step of acquiring a depth image of the aforementioned field of view further includes: A method for identifying floor surface stains according to claim 2 or 3, comprising obtaining a depth image including only the floor surface based on the aforementioned depth image.

5. The step of obtaining the spatial coordinates of the soiled area based on the aforementioned common viewing area of ​​the image further includes: A method for identifying floor surface stains according to claim 4, comprising: pixel alignment of a depth image including only the stained area and the floor surface; obtaining the common viewing area of ​​the image; and ensuring that all pixel points of the stained area in the common viewing area of ​​the image correspond one-to-one with all pixel points of the stained area in the depth image.

6. The step of obtaining the spatial coordinates of the soiled area based on the aforementioned common viewing area of ​​the image further includes: The floor surface stain identification method according to claim 5, further comprising obtaining the spatial coordinates of a stained area based on the pixel depth value and pixel position of the pixel points of the stained area in the common viewing area of ​​the image.

7. The step of acquiring the contaminated area based on the RGB image further includes: The RGB image is preprocessed to obtain a preprocessed image, A method for identifying floor stains according to claim 2, comprising calling a trained stain identification model and inputting the preprocessed image into the stain identification model to obtain the stained area.

8. The step of calling the trained stain identification model to obtain the stained area further includes, The floor surface stain identification method according to claim 7, comprising inputting the preprocessed image into the stain identification model, executing a target detection algorithm, a semantic segmentation algorithm, or an image splitting algorithm to obtain the type of stain and the stained area, wherein the stained area is the coordinate of the stain in the image coordinate system.

9. The steps of acquiring a spectral image of the field of view and acquiring a contaminated area based on the spectral image further include: The spectral cluster is obtained by clustering the spectra of each pixel point in the spectral image, A method for identifying floor surface stains according to claim 3, comprising calling a spectral identification model, inputting each spectral cluster into the spectral identification model to obtain stain characteristics, wherein the stain characteristics include a stained area and a stain type.

10. The steps include calling the identification type label model and determining the consistency between the stain type obtained based on the spectral identification model and the stain type obtained based on the stain identification model, A method for identifying floor stains according to claim 9, further comprising the step of selecting the stain type if the stain types of both are the same, and not selecting the stain type if the stain types of both are not the same.

11. The floor surface stain identification method according to any one of claims 1 to 10, wherein the depth image of the field of view is acquired by a depth camera, a structural optical infrared camera, or a combination of both.

12. The floor surface stain identification method according to claim 11, wherein the depth camera is selected from one of RGBD, a binocular stereoscopic camera, or TOF.

13. The floor surface stain identification method according to claim 11, wherein the step of acquiring the depth image of the field of view by a combination of a structural optical infrared camera and a depth camera further includes acquiring a depth image including only the floor surface based on the depth image.

14. The step of obtaining a depth image including only the floor surface based on the aforementioned depth image is further: The depth image of the field of view collected by the depth camera is acquired, and a first floor surface image is acquired based on the depth image. The process involves acquiring a structural light grayscale image collected by a structural light infrared camera, and then acquiring a second floor surface image based on the structural light grayscale image. A method for identifying floor surface stains according to claim 13, comprising superimposing the first floor surface image and the second floor surface image to obtain a depth image including only the floor surface.

15. The floor surface stain identification method according to claim 14, wherein the step of acquiring a first floor surface image based on the depth image further comprises the step of acquiring a first floor surface image of the depth image based on internal and external parameters of the depth camera and the pixel depth values ​​of the depth image.

16. The floor surface stain identification method according to claim 14, wherein the step of acquiring a second floor surface image based on the aforementioned structural light grayscale image further includes converting the structural light grayscale image into a depth image, and acquiring a second floor surface image of the depth image converted from the structural light grayscale image based on the distance measurement information of the structural light grayscale image.

17. The step of superimposing the first floor image and the second floor image to obtain a depth image including only the floor surface is further: A method for identifying floor surface stains according to claim 14, comprising: calculating the difference in depth values ​​of corresponding pixel points; discarding the image depth of the pixel point if the difference in depth values ​​exceeds a threshold; obtaining the average pixel depth of the pixel point if the difference in depth values ​​is within the threshold range; and further obtaining a depth image that includes only the floor surface.

18. The floor stain identification method according to claim 11, further comprising the step of irradiating the stained surface and edges with a green eye / infrared light supplement lamp when acquiring an RGB image or spectral image.

19. A step of converting the spatial coordinates of the soiled area in the image coordinate system to the spatial coordinates of the soiled area in the autonomous cleaning device coordinate system, A method for identifying floor surface stains according to claim 18, further comprising the step of converting the spatial coordinates of a stained area in the coordinate system of an autonomous cleaning device to the spatial coordinates of a stained area in a world coordinate system, based on the real-time orientation of the autonomous cleaning device.

20. A first identification module used to acquire RGB images of the field of view, A second identification module used to acquire a depth image of the field of view, A pixel alignment module used to acquire a common viewing area of ​​an image based on RGB images and depth images, A floor surface stain identification device comprising a coordinate calculation module used to obtain the spatial coordinates of a stained area based on a common viewing area of ​​an image.

21. An autonomous cleaning device comprising an autonomous cleaning device body and a controller for controlling the autonomous cleaning device body, wherein the controller employs the floor surface dirt identification method described in any one of claims 1 to 19.

Images