Floor stain recognition method and apparatus, and automatic cleaning device

Abstract

A floor stain recognition method and apparatus, and an automatic cleaning device. The method comprises: obtaining an RGB image of a field of view; obtaining a depth image of the field of view; and on the basis of the RGB image and the depth image, obtaining a common area of view of said images; and, on the basis of the common area of view of said images, obtaining spatial coordinates of the stain area. The method allows for accurate detection of stain type and position.

Description

Ground stain identification method, device and self-cleaning equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311084777.6 filed on August 25, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the technical field of self-cleaning equipment, and in particular to an obstacle avoidance method and device for self-cleaning equipment and self-cleaning equipment. Background Art

[0004] Self-cleaning devices often encounter dirt on the floor during operation, requiring accurate detection of the type and location of dirt to facilitate avoidance or cleaning. RGB image detection is typically used for floor dirt detection. Due to the wide variety of floor stains, most current self-cleaning devices on the market can only perform image clustering and segmentation, failing to determine the type of stain and preventing effective cleaning.

[0005] Summary of the Invention

[0006] In response to the above situation, the embodiments of the present application propose a floor stain recognition method, device and self-cleaning equipment, which can accurately detect the type and location of the stain.

[0007] In a first aspect, an embodiment of the present application provides a method for identifying floor stains, the method comprising:

[0008] Get an RGB image of a field of view;

[0009] Acquire a depth image of the field of view;

[0010] Obtaining the image common viewing area based on the RGB image and the depth image;

[0011] The spatial coordinates of the stain area are obtained based on the image common view area.

[0012] In a second aspect, an embodiment of the present application further provides a floor stain identification device, the device comprising:

[0013] A first recognition module is used to obtain an RGB image of a field of view;

[0014] a second recognition module, configured to obtain a depth image of the field of view;

[0015] Image alignment module, which obtains the image common viewing area based on RGB image and depth image;

[0016] The coordinate calculation module obtains the spatial coordinates of the stain area based on the image common viewing area.

[0017] In a third aspect, an embodiment of the present application further provides a self-cleaning device, comprising a self-cleaning device body and a floor stain recognition device for controlling the self-cleaning device body, wherein the floor stain recognition device adopts the floor stain recognition method provided in the first aspect above.

[0018] The solution of the present application enables the self-cleaning equipment to accurately detect the type and location of dirt. The ground dirt detection method based on RGB camera images, green eye fill light, structured light ranging (or RGBD, binocular, TOF), and spectral assisted recognition can accurately detect the type and location of dirt through green eye enhanced image features, stain recognition model detection, multi-sensor fusion ranging, spectral image recognition and other methods. When the self-cleaning equipment identifies the type of stain in the stain area, it controls the cleaning machine that can handle the stain type to perform corresponding processing to achieve precise matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] FIG1 is a schematic flow chart of a method for identifying floor stains according to a first embodiment of the present application.

[0021] FIG2 shows a schematic structural diagram of a floor stain identification device according to a second embodiment of the present application. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0024] Self-cleaning devices, also known as automatic sweepers, self-cleaning devices, smart vacuums, self-cleaning vacuum cleaners, and mobile self-cleaning devices, are smart home appliances that utilize artificial intelligence to automatically clean floors in a room. They typically use a brushing and vacuuming method to collect debris into their own waste collection bins, completing the floor cleaning process. Generally speaking, self-cleaning devices that perform sweeping, vacuuming, and mopping are also categorized as self-cleaning devices.

[0025] There are many types of floor stains. Traditional methods can only roughly cluster and segment stains in images, but cannot determine the type of stain, making effective cleaning impossible. Furthermore, self-cleaning equipment operates in a complex environment, especially where lighting changes significantly impact RGB images, thus affecting stain detection and image depth calculation.

[0026] Based on this, this application proposes a ground dirt detection method based on RGB camera image + green eye fill light + structured light ranging (or RGBD, binocular, TOF) + spectral assisted recognition. Through green eye enhanced image features, stain recognition model detection, multi-sensor fusion ranging, spectral image recognition and other methods, the type and location of dirt can be accurately detected. When the self-cleaning equipment identifies the stain type in the stain area, it controls the cleaning machine that can handle the stain type to perform corresponding processing to achieve precise matching.

[0027] The concept of this application is as follows: First, the RGB image passes through the target detection model, and the pixel position of the stain can be marked on the RGB image. Second, the spatial recognition part, 1) the depth camera will generate a depth image, and each pixel is the depth information, which is converted into three-dimensional spatial coordinates. 2) Structured light ranging (which is the line laser obstacle avoidance solution currently used on machines) uses a grayscale camera to generate a grayscale image. The pixel position of the structured light is found in the grayscale image, and then the three-dimensional coordinates of the pixel position corresponding to the structured light can be calculated based on the internal and external parameters of the structured light and the grayscale camera. Third, the RGB image is overlapped with the depth map and the structured light grayscale map based on the internal and external parameters. In this way, the RGB recognition result (a target detection box) can obtain the depth information, and further the three-dimensional coordinates of the identified stain. After obtaining the three-dimensional coordinates, if the z-axis of the three-dimensional coordinate is 0, it is considered to be the ground, which meets the requirements. If the z-axis is not 0, it means it is a misidentification and is filtered out.

[0028] This application uses RGB image recognition to detect stain areas (can be monocular or binocular), and uses target detection or semantic segmentation to accurately identify the stain type and outline.

[0029] This application also uses depth ranging to accurately determine the distance information of the stain area relative to the self-cleaning device, including:

[0030] 1) Using the depth information of the depth camera alone (binocular stereo camera or ToF);

[0031] 2) Use structured light alone to measure the distance to the target area;

[0032] 3) Fusion of depth information from the depth camera and structured light ranging.

[0033] This application also uses green eye / infrared fill light to enhance the texture of the stained area and improve the recognition of dirty features on the ground; in dim environments, it can also serve as a fill light.

[0034] The solution of this application uses spectral images for stain recognition, which not only makes the stain recognition results and location more accurate, but also can further identify the dry / wet state and turbidity of the stain, providing information for the cleaning strategy of the self-cleaning equipment.

[0035] Currently, the mainstream depth cameras include structured light, time-of-flight, and binocular stereo.

[0036] The basic principle of a structured light depth camera: The main hardware consists of a projector and a camera. The projector actively emits (hence the term "active measurement") infrared light, invisible to the naked eye, onto the surface of the object being measured. One or more cameras then capture the structured light image of the object being measured, and the data is sent to a computing unit. Position and depth information are calculated using triangulation principles, enabling 3D reconstruction. As the name suggests, structured light is structured, and there are various ways to project patterns, such as phase-shifting sinusoidal stripes, binary-coded Gray code, and phase-shifting combined with Gray code.

[0037] Time-of-Flight (TOF) depth camera, basic principle: TOF is the time-of-flight method, which is to emit continuous light pulses to the target. After being reflected by the target, the sensor receives the light back, records the flight time, and calculates the distance to the target. It is generally divided into pulse modulation and continuous wave modulation. Because it actively emits pulsed light, it is also an active measurement method. Unlike binocular cameras and structured light, which require algorithm processing to output three-dimensional data, TOF can directly output three-dimensional data of the measured object.

[0038] Binocular stereo depth camera, basic principle: obtain image information through the left and right cameras and calculate parallax. The binocular camera does not actively emit light to the outside, so it is called a passive depth camera.

[0039] Optionally, the present application can identify ground stains through RGB cameras, depth cameras (RGBD or binocular stereo vision cameras or TOF), and green eye / infrared fill lights.

[0040] Optionally, this application can identify ground stains through RGB cameras, infrared cameras + structured light, and green eye / infrared fill light solutions.

[0041] Optionally, the present application can identify ground stains through RGB cameras, depth cameras (RGBD or binocular stereo vision cameras or TOF), infrared cameras + structured light, and green eye / infrared fill lights.

[0042] Optionally, the present application can identify ground stains through RGB cameras, spectral cameras, depth cameras (RGBD or binocular stereo vision cameras or TOF), infrared cameras + structured light, and green eye / infrared fill lights.

[0043] An RGB-D depth camera (also known as a 3D camera, where D stands for depth) can obtain the distance between an object and the camera. Combined with the X and Y coordinates of the 2D plane, it can calculate the three-dimensional coordinates of each point. This allows us to infer the applications of depth cameras, such as 3D reconstruction, target positioning, and recognition.

[0044] The following embodiments are described in detail based on the above optional solutions.

[0045] [Corrected 13.12.2024 according to Rule 91] Figure 1 shows a flow chart of the ground stain identification method according to the first embodiment of the present application. As can be seen from Figure 1, the present application includes at least steps S110 to S140.

[0046] Step S110: Acquire an RGB image of a field of view.

[0047] The acquiring of an RGB image of a field of view further includes acquiring a stain area based on the RGB image.

[0048] [Corrected 13.12.2024 according to Rule 91] Based on the RGB image captured by the RGB camera, the stain features in the RGB image are obtained, and the stain features include the stain area and the stain type. The RGB image is obtained from the RGB camera, and the RGB image is preprocessed by dedistortion, cropping, and proportional adjustment, and input into the trained stain recognition model. By performing tasks such as image target detection and semantic segmentation, the stain type and the coordinate value of the stain area in the pixel coordinate system (the pixel position of the stain area) are obtained. In a possible embodiment of the present application, based on the RGB image captured by the RGB camera, the stain features in the RGB image are obtained, further comprising: preprocessing the RGB image to obtain a preprocessed image, and the preprocessing method includes but is not limited to dedistortion, cropping, and proportional adjustment of the image. The trained stain recognition model is called, and the processed image is input into the stain recognition model to obtain each pixel position of the stain and the stain type. Based on each pixel position (x, y) of the stain, the stain area is obtained.

[0049] Step S120: Acquire a depth image of the field of view.

[0050] In an embodiment, the depth image of the field of view is obtained by a depth camera or a structured light infrared camera or a fusion of the two.

[0051] The depth image of the field of view is obtained by a depth camera, which includes the following steps: obtaining a depth image containing only the ground based on the depth image. Specifically including:

[0052] [Corrected 13.12.2024 according to Rule 91] Based on the intrinsic and extrinsic parameters of the depth camera and the pixel depth values ​​of the depth image, a ground image of the depth image is obtained;

[0053] [Corrected 13.12.2024 according to Rule 91] The above step of “obtaining a ground image of the depth image” further includes: deleting obstacle pixels of the ground image to obtain the ground image.

[0054] In an embodiment, the depth image of the field of view is obtained by a structured light infrared camera, which includes the following steps:

[0055] [Corrected 13.12.2024 according to Rule 91] Obtain a structured light grayscale image captured by a structured light infrared camera, convert the structured light grayscale image into a depth image, and obtain a depth image containing only the ground based on ranging information of the structured light grayscale image.

[0056] In the embodiment, the depth image of the field of view is obtained by fusing the two, thereby obtaining a depth image containing only the ground based on the depth image. The specific steps are as follows:

[0057] [Corrected 13.12.2024 according to Rule 91] Obtaining a depth image of the field of view captured by a depth camera, and obtaining a first ground image of the depth image based on intrinsic and extrinsic parameters of the depth camera and pixel depth values ​​of the depth image;

[0058] [Corrected 13.12.2024 according to Rule 91] obtaining a structured light grayscale image captured by a structured light infrared camera, converting the structured light grayscale image into a depth image, and obtaining a second ground image of the depth image converted from the structured light grayscale image based on ranging information of the structured light grayscale image;

[0059] [Corrected 13.12.2024 according to Rule 91] Overlap the first ground image with the second ground image, calculate the depth value difference of the corresponding pixels, and if the depth difference exceeds a threshold, discard the pixel depth of the pixel; if the depth difference is within the threshold, obtain the average pixel depth of the pixel, and then obtain an accurate depth image containing only the ground.

[0060] Step S130: Acquire an image common viewing area based on the RGB image and the depth image.

[0061] The acquiring of the image common viewing area based on the RGB image and the depth image further includes:

[0062] Acquire a stain area based on the RGB image;

[0063] The stained area is aligned with the depth image containing only the ground to obtain the image common view area, where all pixel points of the stained area in the image common view area correspond one-to-one to all pixel points of the stained area in the depth image.

[0064] In a possible embodiment of the present application, a depth image containing only the ground in the depth image is obtained based on a depth image captured by a depth camera. The depth camera may be a binocular stereo vision camera or a time-of-flight (TOF) depth camera.

[0065] The depth image is a depth image captured by a depth camera. The depth image provides the depth of all positions within the camera's visible range. According to the internal and external parameters, all ground parts that the depth camera can see are found and marked, that is, an image containing only the ground is obtained.

[0066] Step S140 : acquiring the spatial coordinates of the stain area based on the image common viewing area.

[0067] The acquiring of the spatial coordinates of the stain area based on the image common viewing area further comprises:

[0068] The spatial coordinates of the stain area are obtained based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

[0069] In a possible embodiment of the present application, after aligning the depth image containing only the ground with the stain area obtained by the RGB image, all pixels in the common view area of ​​the image obtained correspond to all pixels in the depth image. The specific steps for implementing the RGB image and depth image alignment operation are as follows: obtain the intrinsic parameter matrix of the RGB camera, the intrinsic parameter matrix of the depth camera, and the extrinsic parameter rotation matrix and offset matrix of the RGB camera and the depth camera; traverse the coordinates of each pixel point in the depth image, and calculate the coordinates of the pixel point in the depth image corresponding to the pixel point in the RGB image according to the formula (refer to the existing technology).

[0070] The method of obtaining the spatial coordinates of the stain based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area also includes: the pixel values ​​of the depth image can calculate the distance from each pixel point to the camera as the pixel depth value. Depending on the camera parameters, each depth value can correspond to a corresponding depth distance. After the depth image and the RGB image are aligned, all the pixels in the RGB image correspond to all the pixels in the depth image, so that the three-dimensional coordinates corresponding to each pixel in the image in the world coordinate system can be obtained based on the position of the pixel and the depth information corresponding to the pixel point.

[0071] [Corrected 13.12.2024 according to Rule 91] The spatial coordinates of the stain area in the pixel coordinate system are converted into the spatial coordinates of the stain area in the self-cleaning device coordinate system.

[0072] [Corrected 13.12.2024 according to Rule 91] Based on the real-time posture of the self-cleaning device, the spatial coordinates of the stain area under the self-cleaning device coordinates are converted to the spatial coordinates of the stain area under the world coordinate system.

[0073] When acquiring RGB images, the green eye / infrared fill light is directed onto the surface and edges of the stain. This not only enhances the texture information of the stain area and improves the RGB image features, but also enhances the spectral characteristics of the stain area, acting as fill light in dim conditions, allowing the machine to clearly see the outline of the obstacle and improving the spectral recognition effect.

[0074] In the first embodiment of the present application, the stains on the ground can be identified by using an RGB camera, a depth camera (RGBD or a binocular stereo vision camera or TOF), and a green eye / infrared fill light solution. In the first embodiment of the present application, the stains on the ground can be identified by using an RGB camera, a structured light infrared camera, and a green eye / infrared fill light solution. In the first embodiment of the present application, the stains on the ground can be identified by using an RGB camera, a depth camera (RGBD or a binocular stereo vision camera or TOF), an infrared camera + structured light, and a green eye / infrared fill light solution.

[0075] [Corrected 13.12.2024 according to Rule 91] The floor stain identification method according to the second embodiment of the present application includes at least the following steps:

[0076] [Corrected 13.12.2024 according to Rule 91] Acquire an RGB image of a field of view.

[0077] The acquiring of an RGB image of a field of view further includes acquiring an RGB stain area based on the RGB image.

[0078] In an embodiment, stain features in an RGB image are obtained based on an RGB image captured by an RGB camera. The stain features include a stain type and a stain area. The RGB camera may be a monocular or binocular camera.

[0079] In a possible embodiment of the present application, obtaining an RGB image of a field of view, and obtaining a stain area based on the RGB image further includes:

[0080] [Corrected 13.12.2024 according to Rule 91] Preprocessing the RGB image to obtain a preprocessed image, wherein the preprocessing method includes but is not limited to dedistorting, cropping, and scaling the image.

[0081] [Corrected 13.12.2024 according to Rule 91] Call the trained stain recognition model, input the processed image into the stain recognition model to obtain stain features, where the stain features specifically include the stain type and the stain area. The stain area can be obtained through the pixel position of the stain, where the pixel position of the stain is the coordinate value of the stain in the pixel coordinate system.

[0082] [Corrected 13.12.2024 in accordance with Rule 91] Further comprising inputting the processed image into the stain recognition model and executing an object detection algorithm, a semantic segmentation algorithm, or an image segmentation algorithm to obtain stain features, wherein the stain features are the stain type and the coordinates of the stain in a pixel coordinate system. For example, if the parameter value calculated by the stain recognition model is 0.88, the stain is determined to be a coffee stain; if the parameter value is 0.86, the stain is determined to be a water stain; and if the parameter value is 0.7, the stain is determined to be a yogurt stain.

[0083] [Corrected 13.12.2024 in accordance with Rule 91] obtaining a spectral image of the field of view;

[0084] [Corrected 13.12.2024 according to Rule 91] The step of “obtaining a spectral image of the field of view” further comprises: obtaining a spectral stain area based on the spectral image.

[0085] In an embodiment, based on the spectral image captured by the spectral camera, the stain features of the spectral image may be obtained, where the stain features include the stain type and the pixel position of the stain.

[0086] In a possible embodiment of the present application, obtaining a spectral image of the field of view, and obtaining a stain area based on the spectral image further includes:

[0087] [Corrected 13.12.2024 according to Rule 91] clustering the spectrum of each pixel in the spectral image to obtain a spectral cluster;

[0088] [Corrected 13 / 12 / 2024 in accordance with Rule 91] The spectral recognition model is invoked and each spectral cluster is input into the spectral recognition model to obtain stain features. The stain features specifically include the stain area and stain type. The stain area can specifically be the coordinates of the stain in the pixel coordinate system. For example, in the case of cola on a patterned tile floor, the stain shape is determined based on the spectral intensity at different pixel locations.

[0089] [Corrected 13.12.2024 according to Rule 91] Obtain the intersection of the stain area obtained based on the RGB image and the stain area obtained based on the spectral image, and the intersection is the selected stain area.

[0090] The RGB recognition result is corrected using the spectral recognition result, specifically including: obtaining an RGB stain area based on the RGB image; obtaining a spectral image of the field of view, and obtaining a spectral stain area based on the spectral image; obtaining an intersection of the RGB stain area obtained based on the RGB image and the spectral stain area obtained based on the spectral image, wherein the intersection is the stain area.

[0091] Spectral image stain recognition not only makes the stain identification results and location more accurate, but can also further identify the dry / wet state and turbidity level of the stain, providing information for the cleaning strategy of the self-cleaning equipment.

[0092] The stain area obtained based on the spectral image is aligned with the stain area obtained based on the RGB image to obtain the RGB image of the common view area (intersection). Using the RGB image as a reference, the spectral image and the RGB image common view area are aligned using the internal and external parameters of the RGB camera and the spectral camera. The RGB recognition result is corrected using the spectral recognition result. This correction includes: obtaining the intersection of the recognition areas and rechecking the recognition category labels. This makes the RGB recognition result more accurate, and the final result is obtained and converted to the RGB image coordinate system.

[0093] [Corrected 13.12.2024 according to Rule 91] In a possible embodiment of the present application, the step of “obtaining the intersection of the stain area obtained based on the RGB image and the stain area obtained based on the spectral image” further includes:

[0094] [Corrected 13.12.2024 according to Rule 91] obtaining a common view area of ​​the stain area obtained based on the spectral recognition model and the stain area obtained based on the stain recognition model, wherein the common view area is the intersection of the stain area obtained by the spectral recognition model and the stain area obtained by the stain recognition model;

[0095] [Corrected 13.12.2024 according to Rule 91] Call the recognition category label model to determine the consistency between the stain type obtained based on the spectral recognition model and the stain type obtained based on the stain recognition model;

[0096] [Corrected 13.12.2024 according to Rule 91] If the two stain types are the same, then that stain type is selected; if the two stain types are different, then that stain type is not selected.

[0097] [Corrected on 13.12.2024 according to Rule 91] It should be noted that the steps of "obtaining an RGB image of a field of view" and "obtaining a spectral image of the field of view" in this application do not limit the order of precedence. The steps of "obtaining an RGB image of a field of view" to "obtaining the intersection of a stain area obtained based on the RGB image and a stain area obtained based on the spectral image" are the image recognition part, the step of "obtaining an RGB image of a field of view" is the process of obtaining RGB recognition results, the step of "obtaining a spectral image of the field of view" is the process of obtaining spectral recognition results, and the step of "obtaining the intersection of a stain area obtained based on the RGB image and a stain area obtained based on the spectral image" is the RGB and spectral fusion process. The spectral recognition result is used to correct the RGB recognition result to obtain the final result, and the final result is converted to the RGB image coordinate system. The RGB and spectral fusion is in the pixel coordinate system, and is subsequently recognized with the spatial recognition module before being transferred to the self-cleaning device coordinate system.

[0098] [Corrected 13.12.2024 according to Rule 91] Acquire a depth image of the field of view.

[0099] Acquiring the depth image of the field of view further includes: acquiring a depth image containing only the ground based on the depth image. In an embodiment, the depth image of the field of view is acquired by fusing a depth camera and a structured light infrared camera, thereby obtaining a depth image containing only the ground based on the depth image. The specific steps are as follows:

[0100] [Corrected 13.12.2024 according to Rule 91] Obtaining a depth image of the field of view captured by a depth camera, and obtaining a first ground image of the depth image based on intrinsic and extrinsic parameters of the depth camera and pixel depth values ​​of the depth image;

[0101] [Corrected 13.12.2024 according to Rule 91] Obtain a structured light grayscale image captured by a structured light infrared camera, convert the structured light grayscale image into a depth image, and obtain a second ground image of the depth image converted from the structured light grayscale image based on ranging information of the structured light grayscale image.

[0102] It needs to be explained that the installation position of the structured light and grayscale cameras is fixed, and the emission angle of the light emitted by the structured light is also fixed. There is a one-to-one correspondence between the position where the structured light hits the obstacle (the three-dimensional coordinate in the camera coordinate system) and the pixels on the structured light grayscale image, that is, the three-dimensional coordinates are obtained, and the three-dimensional coordinates are the ranging information. For example, if the z-axis of the three-dimensional coordinate is 0 (actually used near 0), it is the ground, and if it is greater than 0, it is an obstacle.

[0103] The converting of the structured light grayscale image into a depth image further includes: turning on the structured light, obtaining a grayscale image with structured light information from an infrared camera, using the calibration parameters of the infrared camera and the structured light to calculate the distance information of the self-cleaning device from the ground or obstacles, and marking the depth information in the grayscale image.

[0104] [Corrected 13.12.2024 according to Rule 91] Overlap the first ground image with the second ground image, calculate the depth value difference of the corresponding pixels, and if the depth difference exceeds a threshold, discard the pixel depth of the pixel; if the depth difference is within the threshold, obtain the average pixel depth of the pixel, and then obtain an accurate depth image containing only the ground.

[0105] The depth image provided by the depth camera provides the depth of all locations within the camera's visual range. Based on internal and external parameters, all ground parts that the camera can see are found and marked, that is, an image containing only the ground is obtained.

[0106] The depth camera of this application is selected from any one of RGBD, binocular stereo vision camera, and TOF.

[0107] In another possible implementation, the depth image of the field of view is acquired only by a depth camera, and acquiring a depth image containing only the ground based on the depth image further includes:

[0108] [Corrected 13.12.2024 according to Rule 91] Obtain a depth image of the field of view captured using a depth camera, and obtain a ground image of the depth image based on internal and external parameters of the depth camera and pixel depth values ​​of the depth image.

[0109] The depth camera may be a binocular stereo vision camera or a time-of-flight (TOF) depth camera.

[0110] The grayscale image captured by the structured light infrared camera is converted into a depth image. This involves turning on structured light, obtaining a grayscale image with structured light information from the infrared camera, using the calibration parameters of the infrared camera and structured light to calculate the distance between the self-cleaning device and the ground or obstacles, and marking them in the grayscale image with depth information. The grayscale image is then converted into a depth map (of course, only the areas corresponding to the structured light have depth), which is then fused with the RGB image to provide depth information.

[0111] [Corrected on 13.12.2024 according to Rule 91] When adopting the fusion solution of depth camera and structured light infrared camera, on the basis of the above steps, the grayscale image captured by the structured light infrared camera is obtained and converted into a depth image. It is also necessary to align the two images, find the depth difference of corresponding pixels, and obtain a more accurate depth image. If a secondary verification is performed when the depth camera and structured light infrared camera solution is adopted, the accuracy of the depth image of the field of view is further improved. Specifically, the first ground image is overlapped with the second ground image, and the depth value difference of the corresponding pixel points is calculated. If the depth difference exceeds the threshold, the pixel depth of the pixel point is discarded. If the depth difference is within the threshold range, the average pixel depth of the pixel point is obtained, and then an accurate depth image containing only the ground is obtained.

[0112] [Corrected 13.12.2024 according to Rule 91] Obtain the image common view area based on the RGB image and depth image.

[0113] Obtaining the image common view area based on the RGB image and the depth image further includes: aligning the depth image containing only the ground with the stain area obtained by the RGB image to obtain the image common view area, wherein all pixel points of the stain area in the image common view area correspond one-to-one to all pixel points of the stain area in the depth image.

[0114] The so-called image alignment is based on the RGB camera and uses the camera's internal and external parameters to convert the depth image and structured light grayscale image into the RGB camera coordinate system to obtain the image common viewing area.

[0115] The depth camera generates a depth image, each pixel of which is depth information, and converts each pixel of the depth image into a three-dimensional space coordinate.

[0116] [Corrected 13.12.2024 according to Rule 91] The spatial coordinates of the stained area are obtained based on the pixel depth values ​​and pixel positions of the pixel points in the stained area in the common viewing area of ​​the image.

[0117] [Corrected 13.12.2024 according to Rule 91] The spatial coordinates of the stain area in the pixel coordinate system are converted into the spatial coordinates of the stain area in the self-cleaning device coordinate system.

[0118] For stains on the ground, the ground position information is assigned to the stain and converted to the self-cleaning device coordinate system. The ground position information can be obtained by the three-dimensional coordinates of each pixel on the ground in the camera coordinate system.

[0119] [Corrected 13.12.2024 according to Rule 91] Based on the real-time posture of the self-cleaning device, the spatial coordinates of the stain area under the self-cleaning device coordinates are converted to the spatial coordinates of the stain area under the world coordinate system.

[0120] Obtain the real-time pose of the self-cleaning device, convert the coordinates of the stained area in the self-cleaning device coordinate system to the coordinates in the world coordinate system, and mark it on the cleaning map.

[0121] When acquiring an RGB image, a green-eye / infrared fill light is applied to the surface and edges of the stain. This not only enhances the texture information of the stain area and improves the RGB image features, but also enhances the spectral characteristics of the stain area, acting as fill light in dim conditions, allowing the machine to clearly see the outline of obstacles and improving spectral recognition. The second embodiment of this application can identify ground stains using an RGB camera, a spectral camera, a depth camera (RGBD or binocular stereo vision camera or TOF), a structured light + infrared camera, and a green-eye / infrared fill light solution.

[0122] [Corrected 13.12.2024 according to Rule 91] FIG2 shows a schematic structural diagram of a floor stain identification device according to a third embodiment of the present application, which is applied to a self-cleaning device. As can be seen from FIG2 , the floor stain identification device 300 includes:

[0123] A first recognition module 310 is used to obtain an RGB image of a field of view;

[0124] A second recognition module 320 is configured to obtain a depth image of the field of view;

[0125] The image alignment module 330 is used to obtain the image common viewing area of ​​the RGB image and the depth image.

[0126] The coordinate calculation module 340 obtains the spatial coordinates of the stain area based on the image common viewing area.

[0127] In one possible embodiment, the image alignment module 330 further includes obtaining a stained area based on the RGB image; obtaining a depth image containing only the ground based on the depth image; and aligning the stained area with the depth image containing only the ground to obtain the image common view area, wherein all pixel points of the stained area in the image common view area correspond one-to-one to all pixel points of the stained area in the depth image.

[0128] In a possible implementation, the coordinate calculation module 340 further includes obtaining the spatial coordinates of the stain area based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

[0129] In a possible implementation, the first recognition module 310 further includes: preprocessing the RGB image to obtain a preprocessed image; calling a trained stain recognition model, and inputting the preprocessed image into the stain recognition model to obtain the stain area.

[0130] In one possible embodiment, the image alignment module 330 is further used to obtain a spectral image of the field of view, and obtain a stain area based on the spectral image; obtain an intersection of the stain area obtained based on the RGB image and the stain area obtained based on the spectral image, where the intersection is the selected stain area.

[0131] In a possible implementation, calling the trained stain recognition model to obtain the stain area further includes:

[0132] The preprocessed image is input into the stain recognition model, and the stain type and the stain area are obtained by executing a target detection algorithm, a semantic segmentation algorithm or an image segmentation algorithm. The stain area is the coordinates of the stain in a pixel coordinate system.

[0133] In a possible implementation, the second recognition module 320 is further configured to cluster the spectrum of each pixel in the spectral image to obtain a spectrum cluster;

[0134] The spectrum recognition model is called, and each spectrum cluster is input into the spectrum recognition model to obtain stain features, where the stain features specifically include a stain area and a stain type.

[0135] In one possible embodiment, the second recognition module 320 is further used to call the recognition category label model to perform consistency judgment on the stain type obtained based on the spectral recognition model and the stain type obtained based on the stain recognition model; if the two stain types are the same, then the stain type is selected; if the two stain types are different, then the stain type is not selected.

[0136] In one possible embodiment, the coordinate calculation module 340 is used to convert the spatial coordinates of the stain area in the pixel coordinate system into the spatial coordinates of the stain area in the self-cleaning device coordinates; based on the real-time posture of the self-cleaning device, the spatial coordinates of the stain area in the self-cleaning device coordinates are converted into the spatial coordinates of the stain area in the world coordinate system.

[0137] It should be noted that the above-mentioned floor stain recognition device can implement the floor stain recognition method in the above-mentioned first embodiment one by one, and the technical features that are the same as the floor stain recognition method are not repeated here.

[0138] The present application also discloses a self-cleaning device according to a fourth embodiment, which includes a self-cleaning device body and a controller for controlling the self-cleaning device body. The controller adopts the floor stain recognition method according to the first embodiment.

[0139] The above-mentioned device and method embodiments correspond to each other and have the same technical effects as the method embodiments. For detailed descriptions, please refer to the method embodiments. The device embodiments are obtained based on the method embodiments. For detailed descriptions, please refer to the method embodiments section and will not be repeated here. It should be understood by those skilled in the art that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present disclosure.

[0140] Those skilled in the art will appreciate that the modules in the devices of the embodiments may be distributed in the devices of the embodiments as described in the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module or further divided into multiple submodules.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for identifying ground stains, comprising: Get an RGB image of a field of view; Acquire a depth image of the field of view; Acquire an image common viewing area based on the RGB image and the depth image; The spatial coordinates of the stain area are acquired based on the image common viewing area.

2. The method for identifying floor stains according to claim 1, wherein: The acquiring of an RGB image of a field of view further includes: acquiring a stain area based on the RGB image.

3. The floor stain identification method according to claim 1, wherein: The acquiring of an RGB image of a field of view further comprises: An RGB stain area obtained based on the RGB image; Acquire a spectral image of the field of view, and acquire a spectral stain region based on the spectral image; An intersection of an RGB stain region obtained based on the RGB image and a spectral stain region obtained based on the spectral image is obtained, and the intersection is the stain region.

4. The method for identifying floor stains according to claim 2 or 3, wherein: The acquiring of the depth image of the field of view further comprises: A depth image containing only the ground is acquired based on the depth image.

5. The method for identifying floor stains according to claim 4, wherein: The step of obtaining the spatial coordinates of the stain area based on the image common viewing area further comprises: The stained area is aligned with the depth image containing only the ground to obtain the image common view area, in which all pixel points of the stained area in the image common view area correspond one-to-one to all pixel points of the stained area in the depth image.

6. The method for identifying floor stains according to claim 5, wherein: The step of obtaining the spatial coordinates of the stain area based on the image common viewing area further comprises: The spatial coordinates of the stain area are obtained based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

7. The method for identifying floor stains according to claim 2, wherein: The obtaining of the stain area based on the RGB image further comprises: Preprocessing the RGB image to obtain a preprocessed image; The trained stain recognition model is called, and the preprocessed image is input into the stain recognition model to obtain the stain area.

8. The method for identifying floor stains according to claim 7, wherein: The calling of the trained stain recognition model to obtain the stain area further comprises: The preprocessed image is input into the stain recognition model, and the stain type and the stain area are obtained by executing a target detection algorithm, a semantic segmentation algorithm or an image segmentation algorithm, where the stain area is the coordinates of the stain in a pixel coordinate system.

9. The method for identifying floor stains according to claim 3, wherein: The acquiring of the spectral image of the field of view and acquiring the stain area based on the spectral image further comprises: clustering the spectrum of each pixel point in the spectral image to obtain a spectral cluster; The spectrum recognition model is called, and each spectrum cluster is input into the spectrum recognition model to obtain stain features, wherein the stain features specifically include a stain area and a stain type.

10. The floor stain identification method according to claim 9, further comprising: Calling the recognition category label model to perform consistency judgment between the stain type obtained based on the spectrum recognition model and the stain type obtained based on the stain recognition model; If the two stain types are the same, the stain type is selected; if the two stain types are different, the stain type is not selected.

11. The method for identifying floor stains according to any one of claims 1 to 10, wherein: The depth image of the field of view is acquired by a depth camera or a structured light infrared camera or a fusion of the two.

12. The floor stain identification method according to claim 11, wherein: The depth camera is selected from any one of RGBD, binocular stereo vision camera, and TOF.

13. The method for identifying floor stains according to claim 11, wherein: The depth image of the field of view is obtained by fusing the structured light infrared camera with the depth camera, and further includes: obtaining a depth image containing only the ground based on the depth image.

14. The floor stain identification method according to claim 13, wherein: The step of acquiring a depth image containing only the ground based on the depth image further comprises: Acquire a depth image of the field of view captured by a depth camera, and acquire a first ground image based on the depth image; Acquire a structured light grayscale image captured by a structured light infrared camera, and acquire a second ground image based on the structured light grayscale image; The first ground image is overlapped with the second ground image to obtain the depth image containing only the ground.

15. The method for identifying floor stains according to claim 14, wherein: The acquiring of a first ground image based on the depth image further includes: acquiring a first ground image of the depth image based on internal and external parameters of the depth camera and pixel depth values ​​of the depth image.

16. The method for identifying floor stains according to claim 14, wherein The acquiring of a second ground image based on the structured light grayscale image further comprises: The structured light grayscale image is converted into a depth image, and based on the ranging information of the structured light grayscale image, a second ground image of the depth image converted from the structured light grayscale image is obtained.

17. The method for identifying floor stains according to claim 14, wherein: The step of overlapping the first ground image with the second ground image to obtain the depth image containing only the ground further comprises: The depth value difference of the corresponding pixel points is calculated. If the depth difference exceeds the threshold, the pixel depth of the pixel point is discarded. If the depth difference is within the threshold range, the average pixel depth of the pixel point is obtained, and then the depth image containing only the ground is obtained.

18. The floor stain identification method according to claim 11, further comprising: When acquiring RGB images or spectral images, shine the green eye / infrared fill light onto the surface and edges of the stain.

19. The floor stain identification method according to claim 18, wherein: Convert the spatial coordinates of the stain area in the pixel coordinate system to the spatial coordinates of the stain area in the self-cleaning device coordinate system; Based on the real-time position and posture of the self-cleaning device, the spatial coordinates of the stain area under the self-cleaning device coordinates are converted to the spatial coordinates of the stain area under the world coordinate system.

20. A floor stain identification device, comprising: A first recognition module is used to obtain an RGB image of a field of view; A second recognition module, used to obtain a depth image of the field of view; Image alignment module, which obtains the image common viewing area based on RGB image and depth image; The coordinate calculation module obtains the spatial coordinates of the stain area based on the image common viewing area.

21. A self-cleaning device, comprising a self-cleaning device body and a controller for controlling the self-cleaning device body, wherein the controller adopts the floor stain recognition method according to any one of claims 1 to 19.

Images