Ground stain identification method and device and self-cleaning equipment

Abstract

The invention discloses a ground stain identification method and device and self-cleaning equipment. The method comprises the steps that an RGB image of a view field is acquired; acquiring a depth image of the visual field; and obtaining an image common-view area based on the RGB image and the depth image, and obtaining space coordinates of the stain area based on the image common-view area. According to the scheme, the type and the position of the stain can be accurately detected.

Description

Technical Field

[0001] The present application relates to the technical field of self-cleaning equipment, and in particular to an obstacle avoidance method and device for self-cleaning equipment, and self-cleaning equipment. Background Art

[0002] Self-cleaning equipment often encounters scenes of dirty floors during operation, and needs to accurately detect the type and location of dirt on the floor in order to avoid or clean it. Dirt detection on the floor is generally based on RGB image detection. There are many types of stains on the floor. Most of the self-cleaning equipment on the market can only roughly perform image clustering and segmentation on the stains in the image when cleaning, and cannot determine the type of stains, and cannot perform effective cleaning. Summary of the invention

[0003] In response to the above situation, the embodiments of the present application propose a floor stain recognition method, device and self-cleaning equipment, which can accurately detect the type and location of the stain.

[0004] In a first aspect, an embodiment of the present application provides a method for identifying floor stains, the method comprising:

[0005] Get an RGB image of a field of view;

[0006] Acquire a depth image of the field of view;

[0007] Obtain the image common viewing area based on the RGB image and the depth image;

[0008] The spatial coordinates of the stain area are obtained based on the image common view area.

[0009] In a second aspect, an embodiment of the present application further provides a floor stain identification device, the device comprising:

[0010] A first recognition module is used to obtain an RGB image of a field of view;

[0011] A second recognition module, used to obtain a depth image of the field of view;

[0012] Image alignment module, which obtains the image common viewing area based on RGB image and depth image;

[0013] The coordinate calculation module obtains the spatial coordinates of the stain area based on the image common viewing area.

[0014] In a third aspect, an embodiment of the present application further provides a self-cleaning device, comprising a self-cleaning device body and a floor stain recognition device for controlling the self-cleaning device body, wherein the floor stain recognition device adopts the floor stain recognition method provided in the first aspect above.

[0015] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0016] The solution of the present application can enable the self-cleaning device to accurately detect the type and location of dirt. The ground dirt detection method based on RGB camera images, green eye fill light, structured light ranging (or RGBD, binocular, TOF), and spectral assisted recognition can accurately detect the type and location of dirt through green eye enhanced image features, stain recognition model detection, multi-sensor fusion ranging, spectral image recognition and other methods. When the self-cleaning device identifies the stain type of the stain area, it controls the cleaning machine that can handle the stain type to perform corresponding processing to achieve precise matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A schematic flow chart of a method for identifying floor stains according to a first embodiment of the present application is shown.

[0019] Figure 2 A schematic structural diagram of a floor stain identification device according to a second embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0022] Self-cleaning equipment, also known as automatic sweepers, self-cleaning equipment, smart vacuums, self-cleaning equipment vacuum cleaners, mobile self-cleaning equipment, etc., is a type of smart home appliance that can automatically complete floor cleaning in the room with a certain degree of artificial intelligence. Generally, it uses brushing and vacuuming to absorb the debris on the ground into its own garbage collection box, thereby completing the function of floor cleaning. Generally speaking, self-cleaning equipment that completes sweeping, vacuuming, and mopping is also uniformly classified as self-cleaning equipment.

[0023] There are many types of stains on the ground. Traditional methods can generally only perform image clustering and segmentation on the stains in the image, but cannot determine the type of stains and cannot perform effective cleaning. At the same time, the operating environment of self-cleaning equipment is complex, especially the change of lighting has a great impact on RGB images, thus affecting the dirt detection effect and image depth calculation.

[0024] Based on this, the present application proposes a ground dirt detection method based on RGB camera image + green eye fill light + structured light ranging (or RGBD, binocular, TOF) + spectral assisted recognition. Through green eye enhanced image features, stain recognition model detection, multi-sensor fusion ranging, spectral image recognition and other methods, the type and location of dirt can be accurately detected. When the self-cleaning equipment identifies the stain type in the stain area, it controls the cleaning machine that can handle the stain type to perform corresponding processing to achieve precise matching.

[0025] The concept of this application is as follows: In the first step, the RGB image passes through the target detection model, and the pixel position of the stain can be marked on the RGB image. In the second step, the spatial recognition part, 1) the depth camera will generate a depth image, and each pixel is the depth information, which is converted into three-dimensional spatial coordinates. 2) Structured light ranging (that is, the line laser obstacle avoidance solution used on the machine now), the structured light generates a grayscale image through the grayscale camera, finds the pixel position of the structured light in the grayscale image, and then according to the internal and external parameters of the structured light and the grayscale camera, the three-dimensional coordinates of the pixel position corresponding to the structured light can be solved. In the third step, the RGB image is overlapped with the depth map and the structured light grayscale map according to the internal and external parameters, so that the RGB recognition result (a target detection box) can obtain the depth information, and further the three-dimensional coordinates of the identified stains. After obtaining the three-dimensional coordinates, if the z-axis of the three-dimensional coordinate is 0, it is considered to be the ground, which meets the requirements. If the z-axis is not 0, it means that it is misidentified and filtered out.

[0026] This application uses RGB image recognition to detect stain areas (can be monocular or binocular), and uses target detection or semantic segmentation to accurately identify the stain type and outline.

[0027] This application also uses depth ranging to accurately determine the distance information of the stain area relative to the self-cleaning device, including:

[0028] 1) Use the depth information of the depth camera alone (stereo camera or ToF)

[0029] 2) Use structured light alone to measure the distance to the target area

[0030] 3) Fusion of depth information from depth camera and structured light ranging

[0031] This application can also enhance the texture of the stained area and improve the recognition of dirty features on the ground through green eye / infrared fill light; in dim environments, it can also serve as a fill light.

[0032] The solution of the present application uses spectral images for stain recognition, which not only makes the stain recognition results and positions more accurate, but can also further identify the dry / wet state and turbidity level of the stains, providing information for the cleaning strategy of the self-cleaning equipment.

[0033] Currently, the mainstream depth cameras include structured light, time-of-flight, and binocular stereo.

[0034] The basic principle of structured light depth camera: the main hardware includes projector and camera. The projector actively emits (hence the name active measurement) IR infrared light invisible to the naked eye to the surface of the object to be measured, and then one or more cameras are used to capture the structured light image of the object to be measured, and the data is sent to the computing unit, and the position and depth information are calculated through the triangulation principle to achieve 3D reconstruction. As the name suggests, structured light is to structure light, and there are many ways to project patterns, such as the phase shift method of sinusoidal stripes, binary coded Gray code, phase shift method + Gray code, etc.

[0035] Time-of-Flight (TOF) depth camera, basic principle: TOF is the time-of-flight method, which is to emit continuous light pulses to the target, which are reflected by the target, and the sensor receives the returned light, records the flight time, and calculates the distance to the target. It is generally divided into pulse modulation and continuous wave modulation. Because it actively emits pulsed light, it is also an active measurement method. Unlike binocular cameras and structured light, which need to be processed by algorithms to output three-dimensional data, TOF can directly output three-dimensional data of the object being measured.

[0036] Binocular stereo depth camera, basic principle: obtain image information through the left and right cameras and calculate parallax. The binocular camera does not actively emit light to the outside, so it is called a passive depth camera.

[0037] Optionally, the present application can identify ground stains through an RGB camera, a depth camera (RGBD or a binocular stereo vision camera or TOF), and a green eye / infrared fill light.

[0038] Optionally, the present application can identify ground stains through RGB camera, infrared camera + structured light, green eye / infrared fill light solutions.

[0039] Optionally, the present application can identify ground stains through RGB camera, depth camera (RGBD or binocular stereo vision camera or TOF), infrared camera + structured light, green eye / infrared fill light solution.

[0040] Optionally, the present application can identify ground stains through RGB camera, spectral camera, depth camera (RGBD or binocular stereo vision camera or TOF), infrared camera + structured light, green eye / infrared fill light.

[0041] RGB-D depth cameras (also known as 3D cameras, where D stands for Depth) can obtain the distance information from the object to the camera, and add the X and Y coordinates of the 2D plane to calculate the three-dimensional coordinates of each point. From this, we can infer the applications of depth cameras, such as three-dimensional reconstruction, target positioning, and recognition.

[0042] The following embodiments can be described in detail based on the above optional solutions.

[0043] Figure 1 A schematic diagram showing a flow chart of a method for identifying floor stains according to a first embodiment of the present application is shown. Figure 1 It can be seen that the present application at least includes steps S110 to S160:

[0044] Step S110: Acquire an RGB image of a field of view.

[0045] The acquiring of an RGB image of a field of view further includes: acquiring a stain area based on the RGB image.

[0046] Based on the RGB image captured by the RGB camera, the stain features in the RGB image are obtained, and the stain features include the stain area and the stain type. The RGB image is obtained from the RGB camera, and the RGB image is preprocessed by dedistortion, cropping, and proportional adjustment, and input into the trained stain recognition model. By performing tasks such as image target detection and semantic segmentation, the stain type and the coordinate value of the stain area in the pixel coordinate system (the pixel position of the stain area) are obtained. In a possible embodiment of the present application, based on the RGB image captured by the RGB camera, the stain features in the RGB image are obtained, further comprising: step S111, preprocessing the RGB image to obtain a preprocessed image, and the preprocessing method includes but is not limited to dedistortion, cropping, and proportional adjustment of the image. Step S112, calling the trained stain recognition model, inputting the processed image into the stain recognition model to obtain each pixel position of the stain and the stain type. Based on each pixel position (x, y) of the stain, the stain area is then obtained.

[0047] Step S120: Acquire a depth image of the field of view.

[0048] In this embodiment, the depth image of the field of view is obtained by a depth camera or a structured light infrared camera or a fusion of the two.

[0049] The depth image of the field of view is obtained by a depth camera, which includes the following steps: based on the depth image, a depth image containing only the ground is obtained. Specifically, it includes:

[0050] Step S121a, acquiring a ground image of the depth image based on the internal and external parameters of the depth camera and the pixel depth value of the depth image;

[0051] Step S121a further includes S122: deleting obstacle pixels of the ground image to obtain the ground image.

[0052] In this embodiment, the depth image of the field of view is obtained by a structured light infrared camera, which includes the following steps:

[0053] Step S121b, obtaining a structured light grayscale image captured by a structured light infrared camera, converting the structured light grayscale image into a depth image, and obtaining a depth image containing only the ground based on the ranging information of the structured light grayscale image.

[0054] In this embodiment, the depth image of the field of view is obtained by fusing the two, thereby obtaining a depth image containing only the ground based on the depth image. The specific steps are as follows:

[0055] Step S121c, obtaining a depth image of the field of view captured by a depth camera, and obtaining a first ground image of the depth image based on internal and external parameters of the depth camera and pixel depth values ​​of the depth image;

[0056] Step S122c, acquiring a structured light grayscale image acquired by a structured light infrared camera, converting the structured light grayscale image into a depth image, and acquiring a second ground image of the depth image converted from the structured light grayscale image based on ranging information of the structured light grayscale image;

[0057] Step S123c, overlap the first ground image with the second ground image, calculate the depth value difference of the corresponding pixel points, if the depth difference exceeds the threshold, discard the pixel depth of the pixel point, the depth difference is within the threshold range, obtain the average pixel depth of the pixel point, and then obtain an accurate depth image containing only the ground.

[0058] Step S130: acquiring an image common viewing area based on the RGB image and the depth image.

[0059] The method of acquiring the image common viewing area based on the RGB image and the depth image further includes:

[0060] Acquire a stain area based on the RGB image;

[0061] The stained area is aligned with the depth image containing only the ground to obtain the image common view area, in which all pixel points of the stained area in the image common view area correspond one-to-one to all pixel points of the stained area in the depth image.

[0062] In a possible embodiment of the present application, a depth image containing only the ground in the depth image is obtained based on a depth image captured by a depth camera. The depth camera may be a binocular stereo vision camera or a time-of-flight (TOF) depth camera.

[0063] The depth image is a depth image captured by a depth camera, which provides the depth of all positions within the visible range of the camera. All ground parts that the depth camera can see are found and marked according to internal and external parameters, that is, an image containing only the ground is obtained.

[0064] Step S140: acquiring the spatial coordinates of the stain area based on the image common viewing area.

[0065] The step of obtaining the spatial coordinates of the stain area based on the image common viewing area further comprises:

[0066] The spatial coordinates of the stain area are obtained based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

[0067] In a possible embodiment of the present application, after aligning the depth image containing only the ground with the stain area obtained by the RGB image, all the pixels in the common view area of ​​the image obtained correspond to all the pixels in the depth image. The specific steps for implementing the alignment operation of the RGB image and the depth image are as follows: obtain the intrinsic parameter matrix of the RGB camera, the intrinsic parameter matrix of the depth camera, and the extrinsic rotation matrix and offset matrix of the RGB camera and the depth camera; traverse the coordinates of each pixel point in the depth image, and calculate the coordinates of the pixel point in the depth image corresponding to the pixel point in the RGB image according to the formula (refer to the prior art).

[0068] The method of obtaining the spatial coordinates of the stain based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area also includes: the pixel value of the depth image can calculate the distance from each pixel to the camera as the pixel depth value. Depending on the camera parameters, each depth value can correspond to a corresponding depth distance. After the depth image and the RGB image are aligned, all the pixels in the RGB image correspond to all the pixels in the depth image, so that the three-dimensional coordinates corresponding to each pixel in the image in the world coordinate system can be obtained based on the position of the pixel and the depth information corresponding to the pixel.

[0069] Step S150, converting the spatial coordinates of the stain area in the pixel coordinate system into the spatial coordinates of the stain area in the self-cleaning device coordinate system.

[0070] Step S160: based on the real-time position and posture of the self-cleaning device, convert the spatial coordinates of the stain area under the self-cleaning device coordinates into the spatial coordinates of the stain area under the world coordinate system.

[0071] When acquiring RGB images, the green eye / infrared fill light is directed to the surface and edge of the stain. On the one hand, it enhances the texture information of the stain area and improves the RGB image features. On the other hand, it enhances the spectral features of the stain area and acts as a fill light in dim conditions, allowing the machine to see the outline of the obstacle clearly and improving the spectral recognition effect.

[0072] In the first embodiment of the present application, the stains on the ground can be identified by using an RGB camera, a depth camera (RGBD or a binocular stereo camera or TOF), and a green eye / infrared fill light. In the first embodiment of the present application, the stains on the ground can be identified by using an RGB camera, a structured light infrared camera, and a green eye / infrared fill light. In the first embodiment of the present application, the stains on the ground can be identified by using an RGB camera, a depth camera (RGBD or a binocular stereo camera or TOF), an infrared camera + structured light, and a green eye / infrared fill light.

[0073] Figure 2 A schematic diagram showing a flow chart of a method for identifying floor stains according to a second embodiment of the present application is shown. Figure 2 It can be seen that the present application at least includes steps S210 to S260:

[0074] Step S210: Acquire an RGB image of a field of view.

[0075] The acquiring of an RGB image of a field of view further includes: acquiring an RGB stain area based on the RGB image.

[0076] In this embodiment, stain features in the RGB image are obtained based on the RGB image captured by the RGB camera. The stain features include the stain type and the stain area. The RGB camera may be a monocular or binocular camera.

[0077] In a possible embodiment of the present application, obtaining an RGB image of a field of view, and obtaining a stain area based on the RGB image further includes:

[0078] Step S211, preprocessing the RGB image to obtain a preprocessed image, wherein the preprocessing method includes but is not limited to dedistortion, cropping, and proportional adjustment of the image.

[0079] Step S212, calling the trained stain recognition model, inputting the processed image into the stain recognition model to obtain stain features, the stain features specifically include the stain type and the stain area, the stain area can be obtained through the pixel position of the stain, and the pixel position of the stain is the coordinate value of the stain in the pixel coordinate system.

[0080] Step S212 further includes inputting the processed image into the stain recognition model, and obtaining the stain feature by executing the target detection algorithm, the semantic segmentation algorithm or the image segmentation algorithm, wherein the stain feature is the stain type and the coordinates of the stain in the pixel coordinate system. For example, if the parameter value calculated by the stain recognition model is 0.88, it is judged as a coffee stain, if the parameter value is 0.86, it is judged as a water stain, and if the parameter value is 0.7, it is judged as a yogurt stain.

[0081] Step S220: acquiring a spectral image of the field of view;

[0082] Step S220 also includes: acquiring a spectral stain region based on the spectral image.

[0083] In this embodiment, based on the spectral image collected by the spectral camera, the stain features of the spectral image can be obtained, and the stain features include the stain type and the pixel position of the stain.

[0084] In a possible embodiment of the present application, obtaining a spectral image of the field of view, and obtaining a stain area based on the spectral image further includes:

[0085] Step S221, clustering the spectrum of each pixel in the spectral image to obtain a spectrum cluster;

[0086] Step S222, calling the spectrum recognition model, inputting each spectrum cluster into the spectrum recognition model to obtain the stain feature, the stain feature specifically includes the stain area and the stain type, the stain area can specifically be the coordinate value of the stain in the pixel coordinate system. For example, in the example of cola on the patterned tile floor, the stain shape is judged based on the spectral intensity of different pixel positions.

[0087] Step S230: obtaining an intersection of the stain region obtained based on the RGB image and the stain region obtained based on the spectral image, wherein the intersection is the selected stain region.

[0088] The RGB recognition result is corrected using the spectral recognition result, specifically including: obtaining an RGB stain area based on the RGB image; obtaining a spectral image of the field of view, and obtaining a spectral stain area based on the spectral image; obtaining an intersection of an RGB stain area obtained based on the RGB image and a spectral stain area obtained based on the spectral image, wherein the intersection is the stain area.

[0089] Spectral image stain recognition not only makes the stain recognition results and location more accurate, but can also further identify the dry / wet state and turbidity level of the stain, providing information for the cleaning strategy of the self-cleaning equipment.

[0090] Align the stain area obtained based on the spectrum image with the stain area obtained based on the RGB image, obtain the RGB image (intersection) of the common view area, use the RGB image as the reference, use the internal and external parameters of the RGB camera and the spectrum camera to align the common view area of ​​the spectrum image with the RGB image, and use the spectrum recognition result to correct the RGB recognition result, including: taking the intersection of the recognition area, and rechecking the recognition category label. Make the RGB recognition result more accurate, get the final result, and convert the result to the RGB image coordinate system.

[0091] In a possible embodiment of the present application, step S230 further includes:

[0092] Step S231 , obtaining a common view area of ​​the stain area obtained based on the spectrum recognition model and the stain area obtained based on the stain recognition model, wherein the common view area is the intersection of the stain area obtained based on the spectrum recognition model and the stain area obtained based on the stain recognition model.

[0093] Step S232, calling the recognition category label model to perform consistency judgment between the stain type obtained based on the spectrum recognition model and the stain type obtained based on the stain recognition model;

[0094] Step S233: if the two stain types are the same, then the stain type is selected; if the two stain types are different, then the stain type is not selected.

[0095] It should be noted that steps S210 and S220 in the present application do not limit the order of sequence. The above steps S210 to S230 are the image recognition part, step S210 is the process of obtaining the RGB recognition result, step S220 is the process of obtaining the spectral recognition result, and step S230 is the RGB and spectrum fusion process. The spectral recognition result is used to correct the RGB recognition result to obtain the final result, and the final result is converted to the RGB image coordinate system. The RGB and spectrum fusion is in the pixel coordinate system, and then it is recognized with the spatial recognition module before it is transferred to the self-cleaning equipment coordinate system.

[0096] Step S240: acquiring a depth image of the field of view.

[0097] The obtaining of the depth image of the field of view further includes: obtaining a depth image containing only the ground based on the depth image. In this embodiment, the depth image of the field of view is obtained by fusing a depth camera and a structured light infrared camera, thereby obtaining a depth image containing only the ground based on the depth image. The specific steps are as follows:

[0098] Step S241a, obtaining a depth image of the field of view captured by a depth camera, and obtaining a first ground image of the depth image based on internal and external parameters of the depth camera and pixel depth values ​​of the depth image;

[0099] Step S242, obtaining a structured light grayscale image captured by a structured light infrared camera, converting the structured light grayscale image into a depth image, and obtaining a second ground image of the depth image converted from the structured light grayscale image based on the ranging information of the structured light grayscale image.

[0100] It needs to be explained that the installation position of the structured light and grayscale cameras is fixed, the emission angle of the light emitted by the structured light is also fixed, and there is a one-to-one correspondence between the position where the structured light hits the obstacle (the three-dimensional coordinates in the camera coordinate system) and the pixels on the structured light grayscale image, that is, the three-dimensional coordinates are obtained, and the three-dimensional coordinates are the ranging information. For example, if the z-axis of the three-dimensional coordinate is 0 (actually used near 0), it is the ground, and if it is greater than 0, it is an obstacle.

[0101] The converting of the structured light grayscale image into a depth image further includes: turning on the structured light, obtaining a grayscale image with structured light information from an infrared camera, using calibration parameters of the infrared camera and the structured light to calculate the distance information of the self-cleaning device from the ground or obstacles, and marking the depth information in the grayscale image.

[0102] Step S243, overlap the first ground image with the second ground image, calculate the depth value difference of the corresponding pixel points, if the depth difference exceeds the threshold, discard the pixel depth of the pixel point, the depth difference is within the threshold range, obtain the average pixel depth of the pixel point, and then obtain an accurate depth image containing only the ground.

[0103] The depth image provided by the depth camera provides the depth of all positions within the camera's visible range. Based on the internal and external parameters, all the ground parts that the camera can see are found and marked, that is, an image containing only the ground is obtained.

[0104] The depth camera of the present application is selected from any one of RGBD, binocular stereo vision camera, and TOF.

[0105] In another possible implementation, the depth image of the field of view is acquired only by a depth camera, and acquiring a depth image containing only the ground based on the depth image further includes:

[0106] Step S241b: acquiring a depth image of the field of view acquired by using a depth camera, and acquiring a ground image of the depth image based on internal and external parameters of the depth camera and pixel depth values ​​of the depth image.

[0107] The depth camera may be a binocular stereo vision camera or a time-of-flight (TOF) depth camera.

[0108] The grayscale image captured by the structured light infrared camera is obtained and converted into a depth image; specifically, the process includes: turning on the structured light, obtaining a grayscale image with structured light information from the infrared camera, using the calibration parameters of the infrared camera and structured light to calculate the distance between the self-cleaning device and the ground or obstacles, and marking the depth information in the grayscale image. The grayscale image is converted into a depth image (of course, only the part corresponding to the structured light has depth), which is then fused with the RGB image to provide depth information.

[0109] When the depth camera and structured light infrared camera fusion solution is adopted, on the basis of step S241b, the grayscale image captured by the structured light infrared camera is obtained and converted into a depth image. It is also necessary to align the two images and find the depth difference of the corresponding pixels to obtain a more accurate depth image. If a secondary verification is performed when the depth camera and structured light infrared camera solution is adopted, the accuracy of the depth image of the field of view is further improved. Specifically, in step S243, the first ground image is overlapped with the second ground image, and the depth value difference of the corresponding pixel points is calculated. If the depth difference exceeds the threshold, the pixel depth of the pixel point is discarded. If the depth difference is within the threshold range, the pixel depth average value of the pixel point is obtained, and then an accurate depth image containing only the ground is obtained.

[0110] Step S250: acquiring an image common viewing area based on the RGB image and the depth image.

[0111] Acquiring the image common view area based on the RGB image and the depth image further includes: aligning the depth image containing only the ground with the stain area acquired by the RGB image to obtain the image common view area, wherein all pixel points of the stain area in the image common view area correspond one-to-one to all pixel points of the stain area in the depth image.

[0112] The so-called image alignment takes the RGB camera as the reference and uses the internal and external parameters of the camera to convert the depth image and structured light grayscale image into the RGB camera coordinate system to obtain the image common viewing area.

[0113] The depth camera generates a depth image, each pixel of which is depth information, and each pixel of the depth image is converted into a three-dimensional space coordinate.

[0114] Step S260 , obtaining the spatial coordinates of the stain area based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

[0115] Step S270, converting the spatial coordinates of the stain area in the pixel coordinate system into the spatial coordinates of the stain area in the self-cleaning device coordinate system.

[0116] For the stains on the ground, the ground position information is assigned to the stains and converted to the self-cleaning device coordinate system. The ground position information can be obtained through the three-dimensional coordinates of each pixel point on the ground in the camera coordinate system.

[0117] Step S280: based on the real-time position and posture of the self-cleaning device, convert the spatial coordinates of the stain area under the self-cleaning device coordinates into the spatial coordinates of the stain area under the world coordinate system.

[0118] The real-time position and posture of the self-cleaning device are obtained, the coordinates of the stained area in the self-cleaning device coordinate system are converted to the coordinates in the world coordinate system, and marked on the cleaning map.

[0119] When acquiring the RGB image, the green eye / infrared fill light is directed to the surface and edge of the stain. On the one hand, the texture information of the stain area is enhanced, and the RGB image features are improved. On the other hand, the spectral features of the stain area are enhanced, which plays a fill light role in dim conditions, allowing the machine to see the outline of obstacles and improve the spectral recognition effect. The second embodiment of the present application can identify ground stains through the solution of RGB camera, spectral camera, depth camera (RGBD or binocular stereo vision camera or TOF), structured light + infrared camera, and green eye / infrared fill light.

[0120] Figure 2 A schematic diagram of the structure of a floor stain recognition device according to a third embodiment of the present application is shown. The floor stain recognition device is applied to a self-cleaning device. Figure 2 It can be seen that the floor stain identification device 300 includes:

[0121] A first recognition module 310 is used to obtain an RGB image of a field of view;

[0122] A second recognition module 320, configured to obtain a depth image of the field of view;

[0123] The image alignment module 330 is used to obtain the image common viewing area of ​​the RGB image and the depth image.

[0124] The coordinate calculation module 340 obtains the spatial coordinates of the stain area based on the image common viewing area.

[0125] In a possible implementation, the image alignment module 330 further includes obtaining a stained area based on the RGB image; obtaining a depth image containing only the ground based on the depth image; and aligning the stained area with the depth image containing only the ground to obtain the image common view area, wherein all pixel points of the stained area in the image common view area correspond one-to-one to all pixel points of the stained area in the depth image.

[0126] In a possible implementation manner, the coordinate calculation module 340 further includes obtaining the spatial coordinates of the stain area based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

[0127] In a possible implementation, the first recognition module 310 further includes: preprocessing the RGB image to obtain a preprocessed image; calling a trained stain recognition model, and inputting the preprocessed image into the stain recognition model to obtain the stain area.

[0128] In a possible implementation, the image alignment module 330 is further used to obtain a spectral image of the field of view, and obtain a stain area based on the spectral image; obtain an intersection of a stain area obtained based on the RGB image and a stain area obtained based on the spectral image, wherein the intersection is the selected stain area.

[0129] In a possible implementation manner, calling the trained stain recognition model to obtain the stain area further includes:

[0130] The preprocessed image is input into the stain recognition model, and the stain type and the stain area are obtained by executing a target detection algorithm, a semantic segmentation algorithm or an image segmentation algorithm, where the stain area is the coordinates of the stain in a pixel coordinate system.

[0131] In a possible implementation manner, the second recognition module 320 is further configured to cluster the spectrum of each pixel point in the spectral image to obtain a spectral cluster;

[0132] The spectrum recognition model is called, and each spectrum cluster is input into the spectrum recognition model to obtain stain features, wherein the stain features specifically include a stain area and a stain type.

[0133] In a possible implementation, the second recognition module 320 is further used to call the recognition category label model to make a consistency judgment between the stain type obtained based on the spectral recognition model and the stain type obtained based on the stain recognition model; if the two stain types are the same, then the stain type is selected; if the two stain types are different, then the stain type is not selected.

[0134] In a possible implementation, the coordinate calculation module 340 is used to convert the spatial coordinates of the stain area in the pixel coordinate system into the spatial coordinates of the stain area in the self-cleaning device coordinates; based on the real-time posture of the self-cleaning device, the spatial coordinates of the stain area in the self-cleaning device coordinates are converted into the spatial coordinates of the stain area in the world coordinate system.

[0135] It should be noted that the above-mentioned floor stain recognition device can implement the floor stain recognition method in the above-mentioned first embodiment one by one, and the technical features that are the same as the floor stain recognition method are not repeated here.

[0136] The present application also discloses a self-cleaning device of a fourth embodiment, which includes a self-cleaning device body and a controller for controlling the self-cleaning device body, wherein the controller adopts the floor stain recognition method of the first embodiment described above.

[0137] The above-mentioned device corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here. It can be understood by ordinary technicians in this field that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present invention.

[0138] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying ground stains, characterized in that: The method comprises: Get an RGB image of a field of view; Acquire a depth image of the field of view; Acquire an image common viewing area based on the RGB image and the depth image; The spatial coordinates of the stain area are acquired based on the image common viewing area.

2. The method for identifying floor stains according to claim 1, characterized in that: The acquiring of an RGB image of a field of view further includes: acquiring a stain area based on the RGB image.

3. The method for identifying floor stains according to claim 1, characterized in that: The acquiring of an RGB image of a field of view further comprises: An RGB stain area obtained based on the RGB image; Acquire a spectral image of the field of view, and acquire a spectral stain region based on the spectral image; An intersection of an RGB stain region obtained based on the RGB image and a spectral stain region obtained based on the spectral image is obtained, and the intersection is the stain region.

4. The method for identifying floor stains according to claim 2 or 3, characterized in that: The acquiring of the depth image of the field of view further comprises: A depth image containing only the ground is acquired based on the depth image.

5. The method for identifying floor stains according to claim 4, characterized in that: The step of obtaining the spatial coordinates of the stain area based on the image common viewing area further comprises: The stained area is aligned with the depth image containing only the ground to obtain the image common view area, in which all pixel points of the stained area in the image common view area correspond one-to-one to all pixel points of the stained area in the depth image.

6. The method for identifying floor stains according to claim 5, characterized in that: The step of obtaining the spatial coordinates of the stain area based on the image common viewing area further comprises: The spatial coordinates of the stain area are obtained based on the pixel depth values ​​and pixel positions of the pixel points in the stain area of ​​the image common viewing area.

7. The method for identifying floor stains according to claim 2, characterized in that: The obtaining of the stain area based on the RGB image further comprises: Preprocessing the RGB image to obtain a preprocessed image; The trained stain recognition model is called, and the preprocessed image is input into the stain recognition model to obtain the stain area.

8. The method for identifying floor stains according to claim 7, characterized in that: The calling of the trained stain recognition model to obtain the stain area further comprises: The preprocessed image is input into the stain recognition model, and the stain type and the stain area are obtained by executing a target detection algorithm, a semantic segmentation algorithm or an image segmentation algorithm, where the stain area is the coordinates of the stain in a pixel coordinate system.

9. A floor stain identification device, characterized in that: The device comprises: A first recognition module is used to obtain an RGB image of a field of view; A second recognition module, used to obtain a depth image of the field of view; Image alignment module, which obtains the image common viewing area based on RGB image and depth image; The coordinate calculation module obtains the spatial coordinates of the stain area based on the image common viewing area.

10. A self-cleaning device, characterized in that: It comprises a self-cleaning device body and a controller for controlling the self-cleaning device body, and the controller adopts the floor stain recognition method as described in any one of claims 1-8.

Images