Dirt detection device, dirt detection method, joint calibration method, and robot

Abstract

A dirt detection device, a dirt detection method, a joint calibration method, and a robot. The device comprises: one or more sensors arranged on the body of a robot, wherein each sensor comprises an optical sensing component, and / or an acoustic wave sensing component and / or an electromagnetic sensing component and is used for acquiring, by means of the optical sensing component, and / or the acoustic wave sensing component and / or the electromagnetic sensing component, information related to the ground of a region to be detected; and a processing module used for extracting feature information from the acquired information related to the ground of said region and implementing dirt detection on the basis of the feature information. By using the dirt detection device, automated and intelligent detection of dirt by the robot can be implemented while a reliable detection rate is ensured, and the present invention is simple and easy to implement.

Description

Dirt detection device, dirt detection method, combined calibration method and robot

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 29, 2023, with application number 202311850539.1, and the Chinese patent application filed with the China Patent Office on October 17, 2024, with application numbers 202411450054.8, 202411450058.6, 202411450046.3, and 202411449648.7. The contents of the above-mentioned Chinese patent applications are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and specifically, to a dirt detection device, a dirt detection method, a joint calibration method, and a robot. Background Art

[0003] In hotels, supermarkets, shopping malls, factories, hospitals, homes and other places, robots are gradually replacing manual cleaning tasks with their efficient, accurate and continuous working capabilities. The above working scenarios are complex and changeable. Robots perform cleaning tasks more often by global traversal and are unable to detect the dirtiness of the ground and perform targeted cleaning, which affects the effectiveness and intelligence of the robots' tasks.

[0004] Dirt detection technology is a technical means for detecting dirt and pollutants on the surface of objects or in the environment. It is widely used in manufacturing, service industries, medical care, and environmental protection, aiming to protect public health and maintain a clean environment. In related technologies, dirt detection is mainly achieved through the following methods: (1) manual detection; (2) auxiliary manual detection; and (3) infrared brush detection.

[0005] Among them, the use of manual detection to realize dirt detection requires a lot of manpower, and is not easy to detect accurately because of the great influence of the light environment; the use of auxiliary manual detection to realize dirt detection is to add fill light to the machine (for example, shining green light on the ground to increase the dust visibility effect). This method can improve the success rate of manual detection, but still requires human participation; the use of infrared brush plate detection to realize dirt detection is to add infrared detection equipment to the brush plate of the robot. If the ground is dirty, the brush plate will be dirtier, and the reflection of the infrared sensor on the brush plate will change, thereby realizing dirt detection. This detection method has a low detection rate, especially for scenes with low dirt levels or light-colored dirt such as clear water. It has no recognition ability.

[0006] Therefore, how to provide a simple and easy-to-implement automated dirt detection solution while ensuring a reliable detection rate is an urgent problem to be solved. Summary of the Invention

[0007] The main purpose of this application is to disclose a dirt detection device, a dirt detection method, a joint calibration method and a robot, so as to at least solve the problems in the related art such as the lack of a simple and easy-to-implement automatic dirt detection solution while ensuring a reliable detection rate.

[0008] According to one aspect of the present application, a dirt detection device is provided.

[0009] The dirt detection device according to the present application includes: one or more sensors arranged on the robot body, including: optical sensing components, and / or acoustic wave sensing components, and / or electromagnetic induction components, used to obtain information related to the ground of the area to be detected through the above-mentioned optical sensing components, and / or acoustic wave sensing components, and / or electromagnetic induction components; a processing module, used to extract feature information from the acquired information related to the ground of the area to be detected, and realize dirt detection based on the feature information.

[0010] According to another aspect of the present application, a dirt detection method for a dirt detection device is provided.

[0011] The dirt detection method of the dirt detection device according to the present application includes: performing a detection operation on a ground image acquired in real time to obtain one or more local areas; for each of the local areas, obtaining height information corresponding to each point in the local area based on the parallax of stereoscopic vision; performing a screening operation in each of the local areas to determine one or more points in the local area whose height information satisfies a predetermined range.

[0012] According to another aspect of the present application, a dirt detection method for a dirt detection device is provided.

[0013] The dirt detection method of the dirt detection device according to the present application includes: using a single camera to collect ground images in real time to obtain a monocular two-dimensional image; extracting features for depth estimation from the monocular two-dimensional image, and obtaining the depth value of each pixel point in the monocular two-dimensional image based on the features for depth estimation; obtaining depth gradient information based on the depth value of each pixel point in the monocular two-dimensional image; comparing the depth gradient information with a predetermined depth gradient threshold, and extracting a dirty area based on depth gradient information greater than or equal to the predetermined depth gradient threshold.

[0014] According to another aspect of the present application, a laser and dirt combined calibration method for a dirt detection device is provided.

[0015] The laser and dirt joint calibration method of the dirt detection device according to the present application includes: determining the dirt resolution, setting a grid corresponding to the dirt resolution at a ground position, placing a dirt detection device with a single camera or multiple cameras on the ground with a grid, determining the grid position of the one or more dirt points in the collected image, and constructing a first correspondence between the one or more dirt points and the grid; obtaining a light plane formed by a laser line emitted by a single line laser or a light plane formed by intersecting laser lines emitted by at least two line lasers among multiple line lasers, projecting it onto the grid map, determining the grid position of the light plane projection, and constructing a second correspondence between the light plane projection and the grid; drawing a map according to the first correspondence and the second correspondence to achieve joint calibration of laser and dirt.

[0016] According to yet another aspect of the present application, a robot is provided.

[0017] The robot according to the present application includes: one or more dirt detection devices as described in any one of the above items, wherein the one or more sensors of the dirt detection device are arranged in the front area and / or the rear area and / or the bottom area of ​​the body of the robot.

[0018] According to the present application, a dirt detection device, a dirt detection method, a joint calibration method and a robot are provided, which can realize the robot's automatic and intelligent detection of dirt while ensuring a reliable detection rate. The method is simple and easy to implement, thereby improving the robot's intelligence level and work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a structural block diagram of a dirt detection device according to an embodiment of the present application;

[0020] FIG2 is a structural block diagram of a dirt detection device according to a preferred embodiment of the present application;

[0021] FIG3 is a schematic structural diagram of a sensor module of a dirt detection device according to the first embodiment of the present application;

[0022] FIG4 is a schematic structural diagram of a sensor module of a dirt detection device according to a second embodiment of the present application;

[0023] FIG5 is a flow chart of a dirt detection method according to an embodiment of the present application;

[0024] FIG6 is a block diagram of a dual-camera stereo vision system according to a preferred embodiment of the present application;

[0025] FIG7 is a schematic diagram of constructing a dirt map based on projection of dirt pixels according to a preferred embodiment of the present application;

[0026] FIG8 is a schematic diagram of constructing a dirt map based on a plane projection of a dirt depth point cloud according to a preferred embodiment of the present application;

[0027] FIG9 is a flow chart of a dirt detection method according to an embodiment of the present application;

[0028] FIG10 is a block diagram of a system for extracting depth values ​​of pixels in a monocular two-dimensional image according to a preferred embodiment of the present application;

[0029] FIG11 is a flow chart of a laser and dirt combined calibration method according to an embodiment of the present application;

[0030] FIG12 is a schematic structural diagram of a robot according to Example 1 of the present application, in which a dirt detection device is provided in the front area of ​​the robot body;

[0031] Figure 13 is a structural schematic diagram of a robot in which a dirt detection device is provided at the bottom area of ​​the robot body according to Example 2 of the present application. DETAILED DESCRIPTION

[0032] The specific implementation of this application is described in detail below with reference to the accompanying drawings.

[0033] According to an embodiment of the present application, a dirt detection device is provided.

[0034] Figure 1 is a block diagram of a dirt detection device according to an embodiment of the present application. As shown in Figure 1, the dirt detection device includes: one or more sensors (Figure 1 shows N visual sensors, 10_1, 10_2, ..., 10_N) mounted on a robot body, including: optical sensing components, and / or acoustic sensing components, and / or electromagnetic sensing components, for acquiring information related to the ground surface of the area to be detected via the optical sensing components, and / or acoustic sensing components, and / or electromagnetic sensing components; and a processing module 12 for extracting feature information from the acquired information related to the ground surface of the area to be detected, and performing dirt detection based on the feature information.

[0035] The dirt detection device shown in FIG1 can realize the robot's automatic and intelligent detection of dirt while ensuring a reliable detection rate. It is simple and easy to implement, thereby improving the robot's intelligence level and work efficiency.

[0036] Preferably, the one or more sensors mentioned above may include but are not limited to at least one of the following: a camera sensor with a single camera or multiple cameras (for example, an ordinary camera without a fill light module, an ordinary camera with a fill light module, a structured light infrared camera, etc.), a TOF sensor, a laser sensor (for example, a two-dimensional laser sensor, a three-dimensional laser sensor), an ultrasonic sensor, an optical material recognition sensor, an electromagnetic induction sensor, and a spectral camera.

[0037] It should be noted that the sensors included in the above-mentioned dirt detection device are not limited to optical sensors that obtain information related to the ground in the above-mentioned area to be detected through the above-mentioned optical sensing components, such as ordinary cameras without fill light modules, ordinary cameras with fill light modules, structured light infrared cameras, TOF sensors, laser sensors and other sensors covering all optical modes; they are also not limited to ultrasonic sensors, material recognition sensors, etc. that obtain information related to the ground in the above-mentioned area to be detected through sound wave sensing components, and are not limited to material recognition sensors (for example, electromagnetic induction sensors) that obtain information related to the ground in the above-mentioned area to be detected through electromagnetic induction components.

[0038] The above-mentioned dirt detection device can not only use one sensor to detect information related to the ground to realize dirt detection, for example, using a single camera for dirt detection, but also use multiple sensors to detect information related to the ground to realize dirt detection, for example, using a multi-eye vision module for dirt detection.

[0039] The above-mentioned dirt detection device can realize dirt detection and obstacle detection. For example, the first type of dirt detection device may include: one or more line lasers, one or more camera sensors, and a processing module, wherein the one or more line lasers are used to emit laser lines to realize obstacle detection, and the one or more camera sensors are used to time-share multiplex and collect image information to realize obstacle detection and dirt detection respectively, and the processing module is used to execute the obstacle detection method and the dirt detection method based on the image information collected by the one or more camera sensors in time-share multiplexing; or, the second type of dirt detection device may include: multiple camera sensors and a processing module, wherein the multiple cameras are used to collect image information and can realize obstacle detection and dirt detection based on stereo vision, and the processing module is used to execute the obstacle detection method and the dirt detection method based on the image information collected by the multiple cameras.

[0040] The above two implementation modes are further described below.

[0041] FIG2 is a block diagram of the structure of a dirt detection device according to a preferred embodiment of the present application. As shown in FIG2 , the dirt detection device 2 may further include: one or more line lasers ( FIG2 shows n line lasers 20-1, 20-2, ..., 20-n), which are used to emit laser lines to detect obstacles, wherein, when the dirt detection device includes multiple line lasers, the laser lines emitted by at least two of the multiple line lasers intersect; the camera sensors included in the one or more sensors may include: one or more first cameras 22 with a first fill light module 220, and / or, one or more second cameras 24 without the first fill light module 220 ( FIG1 shows one first camera 22 with a first fill light module 220 and one second camera 24 without the first fill light module 220), wherein, The first camera 22 and / or the second camera 24 are used to collect image information in time-sharing multiplexing to respectively realize obstacle detection and dirt detection. When the camera sensor includes: the first camera 22, the first fill light module 220 is used to fill light for the first camera 22 that collects image information. When the camera sensor includes: the second camera, the dirt detection device further includes: one or more first fill light devices 26 (one first fill light device 26 is shown in Figure 1). The one or more first fill light devices 26 are used to fill light for the second camera 24 that collects image information; the processing module 28 is used to execute the obstacle detection method and the dirt detection method based on the image information collected by the first camera 22 and / or the second camera 24 in time-sharing multiplexing.

[0042] Figure 2 provides a dirt detection device with a simple structural design and capable of effectively identifying dirt. When the processing module in the dirt detection device executes the obstacle detection method and the dirt detection method, the first camera and / or the second camera can time-share multiplex the acquisition of image information. Therefore, the dirt detection device can detect both obstacle information and dirt information, thereby realizing the effective integration of the robot's obstacle avoidance function and dirt detection function.

[0043] Among them, the above-mentioned one or more first cameras with a first fill light module and / or one or more second cameras without a first fill light module can be infrared cameras; when the dirt detection device includes: the first camera, the first fill light module set in the first camera is an infrared fill light module, and the infrared fill light module corresponds to the same band as the infrared camera; when the dirt detection device includes: the second camera and the first fill light device, the first fill light device is an independently set infrared fill light lamp, and the infrared fill light corresponds to the same band as the infrared camera.

[0044] Preferably, in addition to the dirt detection device shown in Figure 2, there is another dirt detection device, wherein the camera sensor included in the one or more sensors includes: multiple third cameras with a second fill light module, and / or, multiple fourth cameras without the second fill light module, wherein the third camera and / or the fourth camera are used to collect image information and realize obstacle detection and dirt detection based on multi-eye stereo vision. When the camera sensor includes: the third camera, the second fill light module is used to fill light for the third camera that collects image information. When the camera sensor includes: the fourth camera, the dirt detection device also includes: one or more second fill light devices for filling light for the fourth camera that collects image information; the processing module is used to execute the obstacle detection method and the dirt detection method based on the image information collected by the third camera and / or the fourth camera.

[0045] Among them, obstacle detection and dirt detection based on multi-view stereo vision include but are not limited to the following methods: using multiple cameras to simultaneously obtain scene information, using stereo vision technology to calculate the depth information of objects in the scene, and then judging the distance and position of obstacles to achieve obstacle detection; obtaining the height information corresponding to each point in the ground area based on the parallax of stereo vision, and then performing a screening operation to determine one or more points in the above-mentioned ground area whose height information meets the predetermined range, obtaining dirt points, and achieving dirt detection.

[0046] Among them, the multiple third cameras with second fill light modules and / or the multiple fourth cameras without second fill light modules are infrared cameras; when the dirt detection device includes: the third camera, the second fill light module provided in the third camera is an infrared fill light module, and the infrared fill light module corresponds to the same band as the infrared camera; when the dirt detection device includes: the fourth camera and the second fill light device, the second fill light device is an independently provided infrared fill light lamp, and the infrared fill light corresponds to the same band as the infrared camera.

[0047] Preferably, the infrared camera can be a single-pass infrared camera. Since a single-band infrared fill light, infrared fill light module and infrared camera are used, the influence of ambient light can be eliminated, and the camera has reliable usability and anti-environmental interference performance.

[0048] The infrared single-pass camera can have a single-pass wavelength band of λ ± 20 nm, where λ is any wavelength value in the range [800 nm, 1000 nm]. The infrared single-pass camera can be installed at any angle in the range [-30°, 30°] relative to the horizontal direction; the field of view of the infrared single-pass camera can range from [10°, 160°].

[0049] Preferably, the horizontal coverage angle of the infrared fill light module is greater than the horizontal field of view angle of the infrared camera. Of course, the horizontal coverage angle of the infrared fill light module can also be less than or equal to the horizontal field of view angle of the infrared camera.

[0050] Preferably, the horizontal coverage angle of the infrared fill light is greater than the horizontal field angle of the infrared camera. Of course, the horizontal coverage angle of the infrared fill light can also be less than or equal to the horizontal field angle of the infrared camera.

[0051] The existing technology uses colored lasers and ordinary lenses, which are easily interfered with by ambient light and strong light, making them difficult to use in actual environments. In the present application, the first camera and the second camera in the first dirt detection device, or the third camera and the fourth camera in the second dirt detection device, can be set as infrared cameras, the fill light module in the first camera or the third camera can be set as an infrared fill light module, and the first fill light device or the second fill light device can be set as an infrared fill light lamp. The infrared fill light module or the infrared fill light lamp can emit light that is invisible to the human eye and illuminates the liquid dirt or solid dirt, which is then captured by the infrared camera.

[0052] Preferably, the above-mentioned first type of dirt detection device may include: a single line laser; the light plane formed by the infrared laser line emitted by the above-mentioned single line laser has an angle range of [10°, 150°], and the angle range between the light plane and the horizontal direction is [-30°, 0°].

[0053] Preferably, the first type of dirt detection device may also include: two line lasers spaced apart; the light planes formed by the infrared laser lines emitted by the two line lasers are both perpendicular to the ground; the laser lines emitted by the two line lasers intersect in front of the two line lasers, and the angle range of the intersection angle is [0°, 90°].

[0054] In the preferred implementation process, the above-mentioned first type of dirt detection device can also be provided with one or more line lasers. If multiple line lasers are provided, the laser lines emitted by at least two of the multiple line lasers are crossed. For example, 3D structured light technology usually adopts a cross-line laser solution, that is, two line lasers set on both sides are simultaneously stimulated to emit line lasers respectively, and the line lasers emitted by these two line lasers are crossed. Of course, non-intersecting bidirectional lasers can also be used to achieve obstacle detection. In the application of obstacle detection, 3D structured light technology can accurately measure the depth information of the surface of an object by emitting and receiving light, and then construct a three-dimensional model of the object. By analyzing these three-dimensional models, the position and shape of the obstacle can be effectively identified and located, thereby realizing obstacle detection. The use of cross-dual-line laser obstacle avoidance technology has the advantages of millimeter-level high precision, low cost, high stability and strong resistance to ambient light interference.

[0055] The laser emitted by the line laser can be visible light or invisible light, preferably infrared light. The wavelength of the infrared light can be the same as that of the infrared camera. For example, using a single wavelength infrared light can eliminate the influence of ambient light, ensuring reliable usability and resistance to environmental interference.

[0056] Preferably, the dirt detection device in the present application may further include: one or more fifth cameras with a third fill light module, and / or one or more sixth cameras without the third fill light module and one or more third fill light devices, wherein the fifth camera and the sixth camera are color cameras, which are used to collect color image information in a time-division multiplexing manner to assist in obstacle detection and dirt detection;

[0057] When the dirt detection device includes: the fifth camera, the third fill light module provided in the fifth camera may be a single-band or multi-band fill light module, for example, an 850-940nm band fill light module or a 940nm band fill light module. Of course, it may be set to a full-band fill light module. Preferably, the coverage angle of the single-band or multi-band fill light module may be greater than the field of view angle of the color camera. Of course, the coverage angle of the single-band or multi-band fill light module may also be less than or equal to the field of view angle of the color camera.

[0058] When the dirt detection device includes: the sixth camera and the third fill light device, the third fill light device is a single-band or multi-band fill light independently arranged on the housing of the dirt detection device, for example, an 850-940nm band fill light, or a 940nm band fill light is selected. Of course, it can be set to a full-band fill light. Preferably, the coverage angle of the single-band or multi-band fill light is greater than the field of view angle of the color camera. Of course, the coverage angle of the single-band or multi-band fill light can also be less than or equal to the field of view angle of the color camera.

[0059] The color camera can be installed at any angle within the range of [-30°, 30°] relative to the horizontal direction; the field of view of the color camera can be set to [10°, 180°].

[0060] The color camera is used to capture color images. The color camera can provide high-definition, rich-color images. In this application, the dirt detection device can also be equipped with a color camera. In some special scenarios, such as reflective scenarios, the color camera collects image information to assist in obstacle and dirt identification, more effectively realizing the obstacle avoidance and dirt detection functions. In the specific implementation process, the color camera can be set to one or more. The color camera can use a camera with a single-band or multi-band fill light module, or a camera without a single-band or multi-band fill light module, but it is necessary to add one or more independently set single-band or multi-band fill lights.

[0061] In a specific implementation, the line laser, the first camera and / or the second camera, the third camera and / or the fourth camera, the fifth camera and / or the sixth camera, the first fill light device, the second fill light device, and the third fill light device can be encapsulated by a shell made of plastic, metal or other materials to form a sensor module. The line laser, the first camera and / or the second camera, the third camera and / or the fourth camera, the fifth camera and / or the sixth camera, the first fill light device, the second fill light device, and the third fill light device can be embedded in the shell of the sensor module. A certain number of through holes are provided on the shell at positions corresponding to the above-mentioned devices so that the above-mentioned devices can emit light to the outside of the dirt detection device or receive light from the outside. The processing module of the dirt detection device can be arranged inside the shell of the above-mentioned sensor module. Of course, the processing module can also be arranged outside the shell of the above-mentioned sensor module and connected to the sensor module. For example, the processing module can be arranged on the mainboard, and the processing module on the mainboard is connected to the sensor module via a flexible printed circuit (FPC).

[0062] This application does not limit the position layout of the centerline laser, the first camera and / or the second camera, the third camera and / or the fourth camera, the fifth camera and / or the sixth camera, the first fill light device, the second fill light device, the third fill light device, and the processing module in the above-mentioned dirt detection device. Any position layout method is within the protection scope of this application.

[0063] The above preferred implementation is further described below in conjunction with the embodiments of FIG. 3 and FIG. 4 .

[0064] FIG3 is a schematic diagram of the structure of the sensor module of the dirt detection device according to the first preferred embodiment of the present application. As shown in FIG3 , the sensor module includes: a line laser 30 for emitting a single line laser; an infrared camera 32 for time-division multiplexing to collect image information for implementing dirt detection and obstacle detection, respectively; the infrared camera does not have an infrared fill light module; an infrared fill light 34 for providing fill light to the infrared camera 32, typically when the infrared camera 32 is used for dirt detection; a color camera 36 for time-division multiplexing to collect color image information to assist in obstacle detection and dirt detection; and a full-band fill light 38 for providing full-band fill light to the color camera 36. The line laser 30, infrared camera 32, infrared fill light 34, color camera 36, ​​and full-band fill light 38 are all encapsulated by a housing 39 of the dirt detection device.

[0065] The processing module of the dirt detection device can be arranged inside the sensor module, or can be arranged outside the sensor module and connected to the sensor module. The processing module is not shown in FIG3 .

[0066] It should be noted that Figure 3 is only one embodiment of the dirt detection device. This application does not limit the position layout of the centerline laser 30, infrared camera 32, infrared fill light 34, color camera 36, ​​and full-band fill light 38 packaged on the housing 39 of the above-mentioned dirt detection device. Any adjustment method of the position layout is within the protection scope of this application.

[0067] Figure 4 is a schematic diagram of the structure of a dirt detection device according to a second preferred embodiment of the present application. As shown in Figure 4, the dirt detection device includes: two line lasers 40_1 and 40_2 for emitting line lasers; an infrared camera 42 that collects image information in a time-sharing multiplexed manner, respectively, for dirt detection and obstacle detection; the infrared camera does not have an infrared fill light module; an infrared fill light 44 that provides fill light for the infrared camera 42, typically when the infrared camera 42 is used for dirt detection; a color camera 46 that collects color image information in a time-sharing multiplexed manner to assist in obstacle detection and dirt detection; and a full-band fill light 48 that provides full-band fill light for the color camera 46. The line lasers 40_1 and 40_2, the infrared camera 42, the infrared fill light 44, the color camera 46, and the full-band fill light 48 are all enclosed in a housing 49 of the dirt detection device. As shown in FIG4 , a line laser 40_1 and 40_2 are provided at each end of the housing 49 , and the two ends form a certain angle (greater than 0° and less than 90°) with the middle part of the housing 49 so that the laser lines emitted by the line lasers 40_1 and 40_2 intersect.

[0068] The processing module of the dirt detection device can be arranged inside the sensor module, or can be arranged outside the sensor module and connected to the sensor module. The processing module is not shown in FIG4 .

[0069] It should be noted that the dirt detection device shown in Figure 4 is only an embodiment. This application does not limit the position layout of the two line lasers 40_1 and 40_2, the infrared camera 42, the infrared fill light 44, the color camera 46, and the full-band fill light 48 in the above-mentioned dirt detection device packaged on the housing 49. Any adjustment method of the position layout is within the protection scope of this application.

[0070] Preferably, for the first type of dirt detection device, the first fill light module and / or the first fill light device are used to emit visible light or invisible light to the ground, wherein the first fill light module and / or the first fill light device are used to emit light of different brightness to the ground in every two adjacent frames, or to send light of constant brightness in the same band in all detection frames, or to send light of varying brightness in the same band in all detection frames, or to send light of constant brightness in multiple bands in all detection frames, or to send light of varying brightness in multiple bands in all detection frames.

[0071] Preferably, for the second type of dirt detection device, the second fill light module and / or the second fill light device are used to emit visible light or invisible light to the ground, wherein the first fill light module and / or the first fill light device are used to emit light of different brightness to the ground in every two adjacent frames, or to send light of constant brightness in the same band in all detection frames, or to send light of varying brightness in the same band in all detection frames, or to send light of constant brightness in multiple bands in all detection frames, or to send light of varying brightness in multiple bands in all detection frames.

[0072] The first fill light module 220 and / or the first fill light device 26 in the first dirt detection device, or the second fill light module and / or the second fill light device in the second dirt detection device, are not limited to LED light sources but may also be laser light sources, etc. Optionally, the light emitted by the first fill light module 220 and / or the first fill light device 26, or the second fill light module and / or the second fill light device, may also form speckle, which can enhance the texture effect, especially for ground surfaces with unclear textures, and further improve the camera's dirt detection capabilities.

[0073] There are several ways to do this:

[0074] (1) The first fill light module 220 and / or the first fill light device 26 in the first dirt detection device, or the second fill light module and / or the second fill light device in the second dirt detection device, can emit light of different brightness to the ground in every two adjacent frames (i.e., a flashing fill light method), that is, in every two adjacent frames, one frame emits light to the ground and the other frame does not emit light to the ground, or one frame emits light of higher intensity to the ground and the other frame emits light of lower intensity to the ground;

[0075] (2) The first fill light module 220 and / or the first fill light device 26 in the first dirt detection device, or the second fill light module and / or the second fill light device in the second dirt detection device, may also transmit light of constant brightness in the same wavelength band in all detection frames;

[0076] (3) The first fill light module 220 and / or the first fill light device 26 in the first dirt detection device, or the second fill light module and / or the second fill light device in the second dirt detection device, may also transmit light with varying brightness in the same wavelength band in all detection frames, wherein the brightness between multiple frames may be adjusted according to actual conditions;

[0077] (4) The first fill light module 220 and / or the first fill light device 26 in the first dirt detection device, or the second fill light module and / or the second fill light device in the second dirt detection device, may also transmit light of multiple wavelengths with constant brightness in all detection frames;

[0078] (5) The first fill light module 220 and / or the first fill light device 26 in the above-mentioned first dirt detection device, or the second fill light module and / or the second fill light device in the above-mentioned second dirt detection device, can also send multiple bands of constantly changing light in all detection frames, wherein the brightness between multiple frames can be adjusted according to actual conditions.

[0079] When the one or more sensors include: one or more camera sensors, the first fill light module 220 and / or the first fill light device 26 in the above-mentioned first dirt detection device, or the second fill light module and / or the second fill light device in the above-mentioned second dirt detection device, can respectively emit light in the manners (2)(3)(4)(5), and the above-mentioned processing module extracts gradient features whose gradient values ​​are greater than a predetermined gradient threshold from the above-mentioned ground image data, obtains dirt-related data based on the above-mentioned gradient features, and removes ground texture information or noise data caused by ground paving from the above-mentioned dirt-related data to obtain final dirt data.

[0080] When the one or more sensors include: one or more camera sensors, for the above-mentioned method (1), that is, the first fill light module and / or the first fill light device in the above-mentioned first dirt detection device, or the second fill light module and / or the second fill light device in the above-mentioned second dirt detection device, emit light of different brightness to the ground in every two adjacent frames or emit light of different bands to the ground in every two adjacent frames; then the processing module is used to extract gradient features with gradient values ​​greater than a predetermined gradient threshold from the ground image data collected by each camera, obtain dirt-related data corresponding to each camera sensor, match and verify the dirt-related data corresponding to a single camera sensor in two adjacent frames, or match and verify the dirt-related data in the overlapping area of ​​vision corresponding to multiple camera sensors in two adjacent frames, and obtain dirt data corresponding to a single camera sensor or multiple camera sensors when the gradient change is greater than the predetermined change threshold.

[0081] The following description takes a single or two camera sensors without a fill light module as examples.

[0082] For a single camera sensor, the camera sensor shoots the ground at a certain frequency to obtain monocular image data. The fill light module fills in the dirt objects on the ground in a flashing manner by emitting light of different brightness to the ground in every two adjacent frames (for example, one frame is bright and one frame is off; or one frame is bright and one frame is dim, etc.); or emits light of different bands to the ground in every two adjacent frames; the processing module obtains the ground image data taken by the camera sensor, extracts the obvious gradient features of the dirt objects on the ground due to the fill light, obtains the dirt data of the front and back frames of the monocular, matches and verifies the dirt data of the front and back frames, extracts the obvious gradient features of light and dark changes, and obtains the monocular dirt data.

[0083] For the two camera sensors, the above-mentioned two camera sensors shoot the ground at a certain frequency to obtain binocular image data, and the fill light module fills in the dirty objects on the ground in a flashing manner by emitting light of different brightness to the ground in every two adjacent frames (for example, one frame is bright, one frame is off; or one frame is bright, one frame is dimmed, etc.); or emits light of different bands to the ground in every two adjacent frames; the processing module obtains the ground image data taken by the above-mentioned two camera sensors, extracts the obvious gradient features of the dirty objects on the ground due to the fill light, and obtains the dirt-related data of the two frames before and after the binocular, matches and verifies the dirt-related data in the overlapping area of ​​the line of sight corresponding to the two camera sensors in the two adjacent frames, that is, compares the dirt-related data corresponding to the front and back frames, extracts the obvious gradient features of light and dark changes, and obtains binocular dirt data.

[0084] Preferably, the above-mentioned one or more sensors may also include: one or more TOF sensors, used to collect ground depth image data at a predetermined frequency respectively; the above-mentioned processing module, used to extract ground depth information from the above-mentioned depth image data, fit a plane according to the above-mentioned ground depth information, perform statistics on the above-mentioned ground depth information based on the fitted plane to obtain gradient feature information, extract gradient features with gradient values ​​greater than a predetermined gradient threshold, and obtain dirt data corresponding to each TOF sensor.

[0085] Preferably, the one or more sensors may also include: one or more laser sensors; the one or more laser sensors are used to detect the ground at a predetermined frequency to obtain ground depth information and ground reflected light intensity information; the processing module is used to fit the first plane or the first straight line according to the ground depth information, perform statistics on the ground depth information based on the fitted first plane or the first straight line to obtain first gradient characteristic information, fit the second plane or the second straight line according to the ground reflected light intensity information, perform statistics on the reflected light intensity information based on the fitted second plane or the second straight line to obtain second gradient characteristic information, calculate the depth noise information and / or the ground reflected light intensity information, and obtain dirt data by combining the first gradient characteristic information and the second gradient characteristic information.

[0086] Preferably, the one or more sensors may also include but are not limited to: one or more ultrasonic sensors and / or one or more optical material recognition sensors and / or one or more electromagnetic induction sensors and / or spectral cameras; wherein the one or more ultrasonic sensors are used to sense the material characteristics of the ground through optical sensing components and extract characteristic information related to the material, and perform ground material detection; the one or more ultrasonic sensors are used to sense the material characteristics of the ground through sound wave sensing components and extract characteristic information related to the material, and perform ground material detection; the one or more electromagnetic induction sensors are used to sense the material characteristics of the ground through electromagnetic induction components and extract characteristic information related to the material, and perform ground material detection; the one or more spectral cameras are used to collect spectral data in different wavelength ranges through optical sensing components and perform ground material detection; the processing module is used to judge the dirtiness based on the detection results of the material recognition sensor and / or spectral camera.

[0087] Preferably, the above-mentioned one or more sensors may also include: one or more sensors with optical sensing components; the above-mentioned one or more sensors with optical sensing components are used to collect light signals reflected from the ground after receiving light signals on the ground, wherein the above-mentioned light signals reflected from the ground include: diffuse reflection light signals and reflected light signals; the above-mentioned processing module is used to determine whether the ratio of the diffuse reflection light intensity to the reflected light intensity is greater than a predetermined intensity ratio threshold, and if it is greater than the above-mentioned predetermined intensity ratio threshold, determine whether the frequency of change of the reflected light intensity is greater than a first predetermined frequency threshold, and if it is greater than the above-mentioned first predetermined frequency threshold, determine whether the amplitude of change of the reflected light intensity exceeds a predetermined variation amplitude threshold, and if it is less than or equal to the above-mentioned first predetermined frequency threshold, determine that dirt is detected.

[0088] Preferably, the one or more sensors include: an ultrasonic sensor; the ultrasonic sensor is used to collect the ultrasonic signal reflected from the ground after receiving the ultrasonic signal on the ground; the processing module is used to determine whether the energy of the reflected ultrasonic signal exceeds a predetermined energy threshold, and if it is less than or equal to the predetermined energy threshold, determine whether the intensity change frequency of the ultrasonic signal energy reflected from the ground within the predetermined area currently adjacent to the robot is greater than a second predetermined frequency threshold, and if it is less than or equal to the second predetermined frequency threshold, determine that dirt is detected.

[0089] Preferably, the dirt detection device described in any of the above items may also include a deep learning module for constructing a deep learning model based on dirt feature training. When the robot is located in the detection area, the ground image data acquired by the one or more sensors is input into the deep learning model to detect dirt. Based on the dirt detection results, the dirt type and / or amount are determined. This improves the reliability and accuracy of detection, and enhances the robot's intelligence level and work efficiency. The deep learning module can be provided within the processing module, or alternatively, it can be provided independently of the processing module.

[0090] Combining the above deep learning modules, three methods can be used for dirt detection:

[0091] The first method: In the preferred implementation process, the above-mentioned deep learning module can also be combined with the dirt detection results of the above-mentioned various sensors and processing modules to jointly realize more specific dirt detection. Therefore, the use of the deep learning module can further assist the dirt detection device in achieving the purpose of dirt classification. For example, one or more sensors set on the robot body obtain information related to the ground of the area to be detected. The processing module extracts feature information based on the obtained information related to the ground, and determines that dirt is detected based on the feature information. Then, the deep learning module is combined to obtain the location information of the area where the dirt is located, and the type of dirt can also be distinguished, for example, solid, liquid, dust and other dirt types.

[0092] Second approach: In a preferred implementation, the deep learning module obtains a deep learning model based on dirt feature training. One or more sensors installed on the robot body obtain information related to the ground surface of the area to be detected. This information is then input into the deep learning model, thereby obtaining one or more preliminarily detected dirt areas and determining the dirt type and other information corresponding to each of the preliminarily detected dirt areas. For each preliminarily detected dirt area, a more accurate dirt area can be extracted based on gradient information, i.e., edges are determined by calculating the change in pixel values ​​in the image, thereby further extracting a more accurate dirt area from the preliminarily detected dirt area. This significantly improves the positioning accuracy of the dirt detection area. For each of the extracted dirt areas, height information corresponding to each point in the dirt area is obtained based on stereoscopic parallax. A screening operation is performed in each of the local areas to determine one or more points in the local area whose height information satisfies a predetermined range. This verifies the dirt area preliminarily detected by the deep learning model, thereby reliably ensuring the accuracy of dirt detection. Afterwards, the contamination confidence level of each of the one or more points can be calculated and the contamination detection result can be verified again based on the contamination confidence level, thereby reliably ensuring the accuracy of the contamination detection and further improving the precision of the contamination detection.

[0093] Third approach: In a preferred implementation, the deep learning module generates a deep learning model based on soiling feature training. When the robot is located in the detection area, the ground image data acquired by the one or more sensors is input into the deep learning model. Using the deep learning model, soiling is detected. This not only directly determines the location of the soiling area but also distinguishes soiling types, such as solid, liquid, or dust. Therefore, the deep learning module can be used alone to implement soiling detection.

[0094] According to an embodiment of the present application, a dirt detection method of a dirt detection device is also provided.

[0095] FIG5 is a flow chart of a contamination detection method of a contamination detection device according to Embodiment 1 of the present application. As shown in FIG5 , the contamination detection method of a contamination detection device according to any one of the above items includes:

[0096] Step S501: performing a detection operation on the ground image acquired in real time to obtain one or more local areas;

[0097] Step S503: for each of the local areas, obtaining height information corresponding to each point in the local area based on the parallax of stereoscopic vision;

[0098] Step S505: performing a screening operation in each of the local regions to determine one or more points in the local region whose height information satisfies a predetermined range.

[0099] In related technologies, the robot's dirt detection performance is affected by various factors such as the environment and ground type, and often suffers from false dirt detection or low detection rates in complex scenes. Using the method shown in Figure 5, a detection operation is first performed on the ground image acquired in real time to obtain one or more local areas (dirty-like areas, i.e., areas where dirt may exist); then, for each of the above local areas, the height information corresponding to each point in the local area is obtained based on the parallax of stereo vision; then, a screening operation is performed in each of the above local areas to determine one or more points in the local area whose height information meets the predetermined range (preliminarily determined to be the location points of dirt points). By detecting ground points that meet the predetermined height and using the above dirt detection method, dirt detection can be performed efficiently and reliably in real time in various complex scenes.

[0100] Preferably, in the above step S501, a detection operation is performed on the ground image acquired in real time, and obtaining one or more local areas can be achieved in the following manner: filtering the image acquired in real time, performing an image segmentation preprocessing operation on the filtered image, and obtaining the above one or more local areas.

[0101] Image segmentation is the process of dividing an image into specific regions with unique properties and identifying objects of interest. For example, image segmentation can be achieved based on gradient information. Specifically, the image's gradient information is used to extract and segment objects within the image. Edges are determined by calculating changes in pixel values ​​within the image, thereby achieving image segmentation and identifying areas that resemble dirt.

[0102] Preferably, in step S501, the detection operation is performed on the real-time acquired ground image to obtain one or more local areas. Alternatively, this can be achieved by inputting the acquired ground image information into a deep learning model, and using the deep learning model to obtain one or more preliminarily detected dirty areas. Of course, information such as the dirt type corresponding to each preliminarily detected dirty area can also be obtained simultaneously, such as solid dirt, liquid dirt, dust, etc. Subsequently, for each preliminarily detected dirty area, a more precise dirty area can be extracted based on gradient information. That is, edges can be determined by calculating changes in pixel values ​​in the image, thereby further extracting a more precise dirty area from the preliminarily detected dirty areas.

[0103] Preferably, for each of the above-mentioned local areas, obtaining the height information corresponding to each point in the local area based on the disparity of stereoscopic vision can further include: constructing a dual-camera stereoscopic vision system; for each point in each of the above-mentioned local areas, obtaining the coordinate information of the current point in the left and right cameras respectively, and calculating the disparity of stereoscopic vision based on the above-mentioned coordinate information; based on the above-mentioned disparity, calculating the height information of the current point in the left camera coordinate system or the right camera coordinate system.

[0104] The disparity of stereo vision can be calculated by the following formula: d = u1-u2, where d is the disparity of stereo vision, the coordinate information of the current point in one of the dual cameras is P1(u1, v1), and the coordinate information of the current point in the other camera is P2(u2, v2); the height information Z of the point in the left camera coordinate system or the right camera coordinate system can be calculated by the following formula c : Where b is the baseline distance, and f is the focal length of the left and right cameras.

[0105] Preferably, in step S501, if one or more local regions are obtained by using image gradient information to extract and segment objects in the image, thereby performing image segmentation to obtain dirt-like regions, then after dirt is detected, the above-mentioned deep learning module can be combined to perform image recognition on the ground image data captured by the camera, and based on the dirt detection results, dirt classification and other processing (e.g., solid dirt, liquid dirt, dust, etc.) can be implemented.

[0106] The above preferred embodiment is further described below with reference to the example of FIG. 6 .

[0107] In the preferred implementation process, a single camera or multiple cameras are used to acquire image data in real time, the original image is Gaussian filtered to achieve a smoothing effect and reduce noise interference, and the filtered image is preprocessed for image segmentation (for example, a gradient-based image segmentation algorithm, etc.) to obtain one or more local areas, namely the above-mentioned dirty areas.

[0108] Alternatively, one or more local regions can be obtained by inputting real-time image data from a single or multiple cameras into a deep learning model to obtain one or more preliminarily detected dirty regions. Information such as the dirt type corresponding to each preliminarily detected dirty region can also be obtained, such as solid dirt, liquid dirt, or dust. Subsequently, for each preliminarily detected dirty region, a more precise dirt region can be extracted based on gradient information.

[0109] Construct a dual-camera stereo vision system as shown in Figure 6, obtain the height information corresponding to each point in the local area based on the parallax of stereo vision, screen the above one or more local areas, and determine one or more points in the above local areas whose height information meets the predetermined range.

[0110] Specifically, as shown in Figure 6, the origins of the left and right image coordinate systems are at the intersections O1 and O2 of the camera optical axis and the plane. P represents a dirty spot on the ground, and its coordinates in the left and right camera images are P1(u1, v1) and P2(u2, v2). If the two cameras are mechanically calibrated on the same plane, the Y coordinates of point P will be the same, i.e., v1 = v2. Based on the geometric relationship of triangulation, it can be concluded that:

[0111] Among them, (x c ,y c , z c ) is the coordinate of P in the left camera coordinate system, b is the baseline distance, f is the focal length of the left and right cameras, (u1, v1) and (u2, v2) are the coordinates of point P in the left and right camera images respectively. The disparity can be expressed as:

[0112] From this, the coordinates of a certain point P in the space in the left camera coordinate system can be calculated (of course, the coordinates of a certain point P in the right camera coordinate system can also be calculated, and this embodiment takes the left camera as an example for explanation) are:

[0113] Therefore, for each of the above-mentioned local areas, the three-dimensional coordinates of each point in the area in the left camera coordinate system can be calculated based on the internal and external parameters of the camera. Then, the height information corresponding to each point (i.e., the Z-axis coordinate) can be used to screen out dirty spots. For example, under normal circumstances, regardless of light color, dark color, or patterned floor, the height of the flat ground is between 0mm and 1mm. The interference of various types of ground is eliminated according to the height, and one or more points that meet the predetermined range (for example, greater than or equal to 1mm and less than or equal to 10mm) are preliminarily determined to be dirty spots.

[0114] Preferably, after determining that the height information of one or more points in the above-mentioned local area meets the predetermined range, the following processing may also be included: according to the depth information and height information corresponding to each point determined in advance on the ground image, and the depth information and height information corresponding to the one or more points meeting the predetermined range, calculate and obtain the dirt confidence of each of the above-mentioned one or more points; compare the obtained each of the above-mentioned dirt confidences with the predetermined confidence threshold, and determine the point corresponding to the dirt confidence greater than or equal to the above-mentioned predetermined confidence threshold as a dirty point.

[0115] In the preferred implementation process, the contamination confidence S of each of the one or more points can be calculated using the following formula: d :

[0116] S d =(|d c ·Z c |-S g ) / S g , where S g =d g ·Z g , d c =x c ·y c , d c Indicates the depth information corresponding to any one of the above one or more points, x c The X-axis coordinate of the current point in the left camera coordinate system or the right camera coordinate system, c The Y-axis coordinate of any one of the above one or more points in the left camera coordinate system or the right camera coordinate system, c is the height information of any one of the above one or more points in the left camera coordinate system or the right camera coordinate system, d g and Z g They are respectively the depth information and height information corresponding to any one of the one or more points determined in advance on the non-dirt ground image.

[0117] Preferably, after determining that the height information in the local area satisfies one or more points within a predetermined range, the method further includes at least one of the following:

[0118] Determine the resolution of the pollution map, set a grid corresponding to the determined resolution on the ground, place one camera of a multi-camera camera or a single camera on the ground with the grid, determine the grid position of the pollution points, establish a correspondence between the pollution points and the grid, and draw a grid map based on the correspondence, that is, draw the pollution map using the pollution pixel projection method;

[0119] According to the depth information of the above-mentioned stain points, a plane projection is performed, and the position information of the above-mentioned stain points is converted into a grid coordinate system and a grid map is drawn using the following formula: g x =x c / r+w / 2,g y =y c / r+h / 2, where (g x , g y ) The coordinates of any of the above-mentioned dirty points in the grid coordinate system, r is the grid resolution, w and h are the length and width of the grid respectively, that is, the dirty map is drawn using the dirty depth point cloud plane projection method.

[0120] The above preferred embodiment is further described below with reference to the examples in FIG. 7 and FIG. 8 .

[0121] As shown in Figure 7, based on the dirty pixel projection method, the dirty pixels are projected onto the local grid in the camera coordinate system to confirm the actual detection range of dirt. For example, first confirm the resolution of the dirt map. If the dirt resolution is 5 cm, a 5 cm resolution grid can be prepared in advance; then lay the grid on a flat ground, place the camera on the flat grid ground, and confirm the dirty grid interval where each dirty pixel is located through the image. The black origin in Figure 7 represents the dirty pixel, and the shaded grid area represents the dirty grid area; then build a pixel-grid hash table to query the grid where the dirty pixel is located and draw the final grid map.

[0122] As shown in Figure 8, based on the plane projection method of the dirt depth point cloud, when the relative position of the independent camera module and the ground is fixed, the three-dimensional coordinates of all dirt points can be obtained according to the steps mentioned above. Therefore, plane projection can be performed according to the depth value (x coordinate, y coordinate) of each dirt point, and all dirt points can be converted to the grid coordinate system: g x =x c / r+w / 2 g y =y c / r+h / 2

[0123] Among them, (g x , g y ) is the coordinate of the dirt in the grid coordinate system, r is the grid resolution, w and h are the grid length and width respectively, so the grid coordinates of the dirt point centered in the camera coordinate system can be obtained.

[0124] According to an embodiment of the present application, a dirt detection method of a dirt detection device is also provided.

[0125] FIG9 is a flow chart of a dirt detection method of a dirt detection device according to the second embodiment of the present application. As shown in FIG9 , the dirt detection method includes:

[0126] Step S901: using a single camera to collect ground images in real time to obtain a monocular two-dimensional image;

[0127] Step S903: extracting features for depth estimation from the monocular two-dimensional image, and obtaining a depth value of each pixel in the monocular two-dimensional image based on the features for depth estimation;

[0128] Step S905: Obtaining depth gradient information based on the depth value of each pixel in the monocular two-dimensional image;

[0129] Step S907: Compare the depth gradient information with a predetermined depth gradient threshold, and extract the dirty area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.

[0130] In the related art, manual detection, assisted manual detection, infrared brush detection and other methods are used to realize dirt detection, which requires a lot of manpower and has a low detection rate. The dirt detection method shown in Figure 9 is used to extract the depth estimation features of the monocular two-dimensional image captured by a single camera, and obtain the depth value of each pixel in the above monocular two-dimensional image based on the above features for depth estimation. Then, the depth gradient information is obtained based on the depth value, and the dirt area is extracted based on the depth gradient information. The above method can realize automatic and intelligent detection of dirt without consuming a lot of manpower and effectively improve the dirt detection rate.

[0131] For monocular 2D images captured by a single camera, monocular depth estimation suffers from scale uncertainty because a single image cannot provide absolute distance information. However, in this application, dirt detection is not based on absolute depth values, but rather on relative depth information (e.g., depth gradient information of the fitted ground plane).

[0132] Preferably, in step S905, obtaining depth gradient information according to the depth value of each pixel in the monocular two-dimensional image may include at least one of the following:

[0133] fitting ground plane information according to the depth value of each pixel point in the monocular two-dimensional image, and performing statistics based on the fitted ground plane information to obtain depth gradient information;

[0134] Extract depth gradient information of a local area based on the depth value of each pixel in the monocular two-dimensional image.

[0135] That is, in the process of obtaining depth gradient information based on the depth values ​​of each pixel in the above-mentioned monocular two-dimensional image, two solutions can be used to achieve this. The first solution, starting from the perspective of the entire area, first fits the ground plane information of the entire area based on the depth values ​​of each pixel, and then performs statistics based on the fitted ground plane information to obtain depth gradient information. The second solution, starting from the perspective of the local area, first extracts the depth gradient information of the local area based on the depth values ​​of each pixel in the above-mentioned monocular two-dimensional image. Of course, in the specific implementation process, the above two solutions can be used in parallel to achieve contamination detection, or either solution can be used to achieve contamination detection.

[0136] Preferably, extracting features for depth estimation from the above-mentioned monocular two-dimensional image, and obtaining the depth value of each pixel in the above-mentioned monocular two-dimensional image based on the above-mentioned features for depth estimation can further include the following processing: using a feature extraction module in a pre-trained deep learning model to extract features for depth estimation from the above-mentioned monocular two-dimensional image, wherein the deep learning model includes: a fully convolutional neural network, and / or the back end of the deep learning model is also connected to a conditional random field; using the depth value acquisition module in the above-mentioned deep learning model to map the above-mentioned features for depth estimation to obtain the depth value of each pixel in the above-mentioned monocular two-dimensional image.

[0137] After obtaining the depth value of each pixel, the accuracy of depth estimation can be improved by combining appropriate processing operations, such as filtering out interference. For example, the network layer can be fully convolutionalized (for example, using a fully convolutional neural network (FCN)) and / or conditional random fields can be added to the back end of the deep learning model to remove noise and improve the overall accuracy of the depth estimation algorithm.

[0138] In the preferred implementation process, a monocular stereo vision solution is performed on a monocular two-dimensional image to extract ground depth information, as shown in FIG12 , specifically including the following steps:

[0139] Step 1: Use the feature extraction module in a pre-trained deep learning model (e.g., convolutional neural network (CNN) or residual network (ResNet), which can obtain the size, shape, and relative position of objects in the image) to extract features for depth estimation from the monocular 2D image, such as image edge features, image texture, or other features.

[0140] Step 2: The depth value acquisition module in the deep learning model maps the features used for depth estimation to obtain the depth value of each pixel in the monocular two-dimensional image. The deep learning model can use a fully convolutional neural network (FCN) and / or a conditional random field (CRF) can be connected to the back end of the deep learning model. Figure 10 shows that the deep learning model uses a fully convolutional neural network (FCN) and also connects a conditional random field (CRF) to the back end of the deep learning model.

[0141] Among them, the application of FCN in noise removal is mainly reflected in its full convolution characteristics. When removing noise, FCN learns the mapping relationship from noisy images to noise-free images by training a large number of data sets of noisy and noise-free images. This method performs well when dealing with image noise, especially when dealing with random noise. The application of conditional random fields (CRF) in noise removal focuses on optimizing the smoothness of labels. As a probabilistic graphical model, CRF can further refine the labeling results based on FCN and ensure a smooth transition of labels. By considering the spatial relationship between pixels, CRF can better handle the dependency of labels, thereby improving the accuracy and consistency of denoising results.

[0142] Preferably, before extracting features for depth estimation from the monocular two-dimensional image using a feature extraction module in a pre-trained deep learning model, the method may further include: optimizing the deep learning model using the following specific loss function L during training the deep learning model;

[0143] in, x represents the input vector, y represents the response variable, t represents the time parameter, ε represents the error vector, and max() represents the function that takes the maximum value. Represents the corresponding value of the input vector x and x at time t The L1 norm of the difference, Represents the corresponding value of the input vector ε and ε at time t The L1 norm of the difference, E x,y,t,ε [] means that under the given conditions of x, y, t, ε, and The expected maximum value of .

[0144] Among them, L1 norm is the sum of the absolute values ​​of each element in the vector. Given two vectors x = (x1, x2, ..., x n ) and x t =(x t1 , x t2 ,…,x tn ), and two error vectors ε=(ε1,ε2,…,ε n ) and ε t =(ε t1 ,ε t2 ,…,ε tn ), we can calculate xx separately t and ε-ε t The L1 norm of ||xx. t ||1=|x1-x t1 ∣+∣x2-x t2 ∣+…+∣x n -xtn ∣;||ε-ε t ||1=|ε1-ε t1 ∣+∣ε2-ε t2 ∣+…+∣ε n -ε tn ∣.

[0145] Preferably, after determining that dirt is detected, a deep learning method can also be used to perform image recognition on the ground image data obtained by the camera, and based on the dirt detection results, determine the dirt category (for example, solid dirt, or liquid dirt, etc.) and / or quantity.

[0146] According to an embodiment of the present application, a laser and dirt joint calibration method for a dirt detection device is also provided.

[0147] FIG11 is a flow chart of a laser and dirt combined calibration method according to an embodiment of the present application. As shown in FIG11 , the laser and dirt combined calibration method includes:

[0148] Step S1101: Determine a dirt resolution, set a grid corresponding to the dirt resolution at a ground position, place a dirt detection device having a single camera or multiple cameras on the ground, determine position information corresponding to one or more dirt points in the captured image, and establish a first correspondence between the one or more dirt points and the position information;

[0149] Step S1103: Projecting the acquired light plane formed by the laser line emitted by the single line laser or the light plane formed by the intersecting laser lines emitted by at least two line lasers among the multiple line lasers toward the ground, determining position information corresponding to the light plane projection, and establishing a second correspondence between the light plane projection and the position information;

[0150] Step S1105: Draw a map based on the first corresponding relationship and the second corresponding relationship to achieve joint calibration of laser and dirt.

[0151] In the related art, there is currently a lack of technical solutions for achieving joint laser and dirt calibration. Using the joint laser and dirt calibration method shown in Figure 11, a first correspondence is established between one or more dirt points and location information, and a second correspondence is established between the light plane projection formed by the laser line emitted by the line laser and the location information. A map is then drawn based on these first and second correspondences to achieve joint laser and dirt calibration. The resulting map can simultaneously determine obstacle and dirt information, effectively integrating the robot's obstacle avoidance and dirt detection functions.

[0152] Preferably, in step S1103, the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two line lasers among the multiple line lasers may include the following processing: setting a marking plate at different positions, and using a single camera or multiple cameras to capture and obtain multiple images of the above-mentioned marking plate, wherein the above-mentioned multiple images include: the projection of the laser line emitted by the single line laser on the above-mentioned marking plate, or the projection of the intersecting laser lines emitted by at least two line lasers among the multiple line lasers on the above-mentioned marking plate; extracting the projection center line of the laser line on the above-mentioned marking plate from the above-mentioned multiple images, wherein, when the extracted projection center line is the intersecting laser line, the projection center line of the intersecting laser line is extracted according to the projection center line of the intersecting laser line. According to the spatial position relationship, the light plane corresponding to the projection center line of the extracted laser line is determined; according to each extracted projection center line, the discrete points on the intersection line of the light plane and the marking plate plane are calculated, the discrete points are fitted, and the equations of multiple intersection lines are obtained; according to the equations of the multiple intersection lines, the normal vectors of all the light planes are obtained, and the optimal solution of the normal vector is determined from the normal vectors of all the light planes; in the equations of the multiple intersection lines, the straight lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold are eliminated to obtain the direction vectors corresponding to the remaining straight lines; the direction vectors corresponding to the remaining straight lines are used to construct equations to calculate the normal vector of the light plane formed by the laser line emitted by the single line laser or the light plane formed by the intersecting laser lines.

[0153] The following describes a preferred embodiment of simultaneously stimulating two line lasers disposed on both sides to emit line lasers respectively to form a cross line laser. The main steps include:

[0154] Step 1: Take multiple images;

[0155] For example, 3D structured light technology typically uses a cross-line laser scheme, where two line lasers set on either side are simultaneously stimulated to emit line lasers, and the line lasers emitted by these two line lasers intersect. A marker board is set 20 cm from the camera to capture multiple images. It should be noted that the distance between the calibration board and the camera should be greater than the distance between the intersection of the two line lasers and the camera. The marker board can be a flat checkerboard board, a circular board, an Apritag board, etc. It is necessary to ensure that the laser line projection on the marker board can be extracted in the captured image.

[0156] The laser emitted by the line laser may be visible light or invisible light, and preferably, infrared light may be used.

[0157] Step 2: Calibrate the parameters of the camera and marker board;

[0158] Using the corner points on the calibration plate, through the calibration algorithm in the prior art, for example, the Zhang Zhengyou calibration algorithm, the intrinsic parameters of the camera (focal length f, camera principal point coordinates (cx, cy)), the extrinsic parameters of the calibration plate (rotation matrix R and translation variable t) and the distortion coefficient of the camera (including radial distortion coefficient and tangential distortion coefficient) can be calibrated. This step can assist in correcting the image distortion of the camera and obtain accurate imaging parameters. Among them, the above-mentioned Zhang Zhengyou calibration algorithm is an effective method for multi-camera calibration, which uses a marker plate with internal reference points to detect at least 6 independent 2D-3D alignments to determine the relationship between each camera.

[0159] Step 3: Laser line extraction;

[0160] For example, the centerline of the laser line's projection on the marking plate is extracted from multiple captured images. Specifically, the image pixel values ​​are normalized to reduce the impact of varying lighting conditions. A gamma transform is then used to enhance the image contrast, making the laser line more prominent. Next, a Gaussian filter is applied to conform the image to a Gaussian distribution, thereby reducing noise interference. Finally, the Steger method is used to extract the centerline of the laser line's projection on the calibration plate, ensuring that the extracted line is accurate and continuous.

[0161] Step 4: Distinguish between left and right light planes;

[0162] Because the intersection of the two optical planes lies in front of the calibration plate and parallel to the imaging plane, the geometric spatial relationship indicates that the intersection of these two optical planes with the calibration plate is on either side of the image center and does not intersect. In other words, the optical plane corresponding to the projected centerline of the extracted laser line can be determined based on the spatial relationship of the intersecting laser lines. The intersection to the left of the image center corresponds to the optical plane formed by the line laser emitted by the right-hand line laser, while the intersection to the right of the image center corresponds to the optical plane formed by the line laser emitted by the left-hand line laser.

[0163] Step 5: Calibration of the light plane;

[0164] The accuracy of 3D structured light technology relies primarily on the precise calibration of the camera and light plane. Camera calibration determines its internal and external parameters to ensure that the captured image accurately reflects the geometric spatial characteristics of the real scene. Light plane calibration involves the precise position and direction of the laser line in three-dimensional space to ensure that the laser is correctly projected onto the target object. The specific steps include:

[0165] (1) The extracted laser center line is dedistorted using the camera's distortion coefficient.

[0166] (2) Use the camera's internal parameters (focal length f, camera principal point coordinates (cx, cy)) to convert the dedistorted laser centerline into a ray emitted from the camera. The details are as follows:

[0167] For each point (x, y) on the laser centerline after distortion, it can be converted into normalized camera coordinates through the pinhole model of the camera: X = (xc x ) / f Y=(yc y ) / f

[0168] Where f is the focal length of the camera, (c x , c y ) is the principal point coordinate of the camera;

[0169] Then convert the above points into rays in the camera coordinate system: r(t)=t·[X,Y,1] T

[0170] Here, t is a parameter.

[0171] (3) Combine the ray equations and the external parameters of the marker plate to calculate the discrete points on the intersection of the light plane and the marker plate plane. The details are as follows:

[0172] First, transform the ray in the camera coordinate system to the world coordinate system: r w (t)=R·r(t)+t, where the extrinsic parameters of the calibration plate include the rotation matrix R and the translation vector t.

[0173] Then, the plane equation of the calibration plate is n T p+d=0, where n is the plane normal vector, d is the distance to the origin, and p is the point on the calibration plate; replace the ray equation r w Substitute (t) = R·r(t) + t into the above calibration plate plane equation and solve for t;

[0174] Finally, after finding t, substitute it into r w (t), the discrete points on the intersection line of the light plane and the calibration plate plane can be calculated.

[0175] (4) To avoid cumulative errors, RANSAC is used to perform straight line fitting on the discrete points above to calculate the equation of the intersection line. Each calibration plate plane has only one intersection line with a certain light plane.

[0176] (5) According to the equation of the intersection line, for example, the equations of k straight lines are obtained on the light plane. Each straight line l i It can be expressed in vector form as: i =a i +td i

[0177] Among them, a i is the i-th point on the line, d i is the direction vector of the line, and t is the parameter.

[0178] The normal vector n of the light plane is perpendicular to the direction vector d of all lines i , that is, satisfying: n·d i =0

[0179] (6) First, use RANSAC to filter the normal vectors calculated from any two different direction vectors and obtain the optimal solution.

[0180] Specifically, we can arbitrarily select two straight line equations from the obtained k equations. Based on these two straight line equations, we can determine a light plane. Based on the different direction vectors of these two straight lines, we can calculate the normal vector of the light plane. That is, we can obtain the normal vector of the light plane by cross-multiplying the direction vectors of the two straight lines. Perform dot product operations on the normal vector and all the straight lines in the k straight lines. Based on the result of the dot product calculation, we can find the straight line with the best perpendicularity to the k straight lines as the optimal solution for the normal vector. For example, for a normal vector n i , the normal vector n i Perform dot product calculations with the direction vector of each of the k straight lines (if the two vectors being dot-producted are perpendicular, the dot product is 0), and obtain multiple dot product values. After calculating the absolute value of the obtained dot product values, accumulate them and obtain the cumulative value SUM i ,The above steps are executed repeatedly until the accumulated values ​​corresponding to the normal vectors of all light planes are calculated. The smallest accumulated value is selected from all accumulated values, and the normal vector corresponding to the smallest accumulated value is used as the optimal solution of the normal vector. The above scheme can further reduce the error.

[0181] (7) Among the k straight lines, the straight lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold (for example, 10%) are eliminated.

[0182] (8) According to the direction vectors corresponding to the remaining m straight lines in the k straight lines, the final light plane normal vector is obtained by simultaneous equations.

[0183] Where D = [d1, d2, ···, dm] T ,d1,d2,···dm are the direction vectors of the m remaining straight lines.

[0184] In the prior art, a separate calibration method is adopted for each laser in the dual lasers that emit intersecting laser lines, that is, the calibration of each laser needs to be set, measured and calculated separately. This method is not only time-consuming, but also requires the use of additional equipment to ensure the stability of the calibration process. This tedious calibration process places high demands on the precise installation and repeatability of the equipment, increasing the difficulty of operation. In the calibration process of the above-mentioned multi-line structured light obstacle avoidance module provided in the present application, when the projection center line of the intersecting laser lines is extracted, the light plane corresponding to the projection center line of the extracted laser line can be determined according to the spatial position relationship of the above-mentioned intersecting laser lines. Therefore, for multiple lasers that emit intersecting laser lines, the simultaneous calibration of the above-mentioned multiple lasers can be achieved.

[0185] In addition, the calibration methods in the prior art are sensitive to noise and are easily affected by factors such as changes in ambient light, equipment jitter, and measurement errors, resulting in inaccurate calibration results. Even small deviations will accumulate into large errors in the final system application, affecting the overall performance of the system. In this application, RANSAC is used to screen the normal vectors calculated from any two different direction vectors, and the optimal solution is obtained. Among the multiple intersection equations obtained based on the extracted projection center line, the straight lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold are eliminated. The final light plane normal vector is obtained based on the simultaneous equations of the direction vectors of the remaining straight lines that meet the predetermined conditions after elimination. The error processing mechanism provided by this application can intelligently analyze and correct various errors occurring during the calibration process, further improving the accuracy and reliability of the system.

[0186] Step 6: Realize the joint calibration of laser and dirt.

[0187] Specifically, a dirt detection device is used to perform a dirt detection operation. For example, the dirt detection device utilizes a camera and image processing technology (edge ​​detection algorithm, etc.) to detect dirt information on the ground. For example, the detected dirt information is projected onto a local grid in the camera coordinate system to confirm the actual detection range of dirt. For example, the dirt resolution is first confirmed. If the dirt resolution is 5cm, a 5cm resolution grid image (e.g., a checkerboard image) is prepared in advance. The grid image is then laid on a flat ground surface. After placing the camera on the flat checkerboard ground surface, the image is used to confirm the dirt grid interval in which each pixel is located. A correspondence between dirty pixels and position information is then constructed to query the grid where the dirty pixel is located.

[0188] The left and right light planes are projected toward the ground to determine the position information corresponding to the projection line, and then the correspondence between the laser projection information and the position information is constructed.

[0189] Based on the correspondence between dirty pixels and position information, as well as the correspondence between laser projection information and position information, a map is drawn, and the laser information and dirt information are jointly calibrated into the map. Using this map, obstacle information and dirt information can be determined simultaneously, realizing the effective integration of the robot's obstacle avoidance function and dirt detection function.

[0190] According to an embodiment of the present application, a robot is also provided.

[0191] The robot according to an embodiment of the present application includes: a dirt detection device as described in any one of the above items, wherein the one or more sensors of the dirt detection device are arranged in the front area and / or the rear area and / or the bottom area of ​​the body of the robot.

[0192] In a preferred implementation, one or more sensors in the dirt detection device described in any of the above items can be installed on the robot body in all directions and at multiple angles. Preferably, one or more sensors of the dirt detection device can be set in the front area and / or the rear area and / or the bottom area of ​​the robot body. The sensor set in the front area of ​​the body can detect the corresponding ground area. It can be set directly in front of the robot's forward direction or at a predetermined inclination angle toward the ground in the forward direction. For details, please refer to the example shown in Figure 12. Similarly, the sensor set in the rear area of ​​the body can be set directly behind the robot or at a predetermined inclination angle toward the ground behind the robot. The sensor set in the bottom area of ​​the robot body can be embedded in a groove at the bottom of the robot body to detect the ground. For details, please refer to the example shown in Figure 13.

[0193] At least one of the sensors disposed in the front region of the robot body is used to determine the dirt level based on the dirt detection result of the current dirt (e.g., dirt location information, dirt area information, dirt type information, etc.) before the robot moves to the current dirt location, so as to determine the dirt treatment strategy corresponding to the dirt level (different dirt levels correspond to different dirt treatment strategies); at least one of the sensors disposed in the rear region of the robot body is used to determine whether it is necessary to return to the current dirt location based on the dirt detection result of the current dirt after the robot performs the cleaning process corresponding to the dirt treatment strategy and leaves the current dirt location. Therefore, one or more sensors can be disposed in both the front region and the rear region of the robot body. This not only allows for early detection of dirt before the robot moves to the dirty area, but also allows for determination of whether the robot has effectively cleaned the dirt when the robot leaves after cleaning the dirt.

[0194] In summary, the above-mentioned embodiments provided in this application provide various implementations of a dirt detection device. A sensor mounted on a robot can obtain information related to the ground surface within the area to be detected through the optical sensing components, and / or acoustic sensing components, and / or electromagnetic sensing components. Feature information is extracted from the acquired ground-related information, and dirt detection is performed based on this feature information. This allows for automated, intelligent dirt detection by the robot while ensuring a reliable detection rate. This approach is simple and easy to implement, improving the robot's intelligence and work efficiency. Furthermore, based on the dirt detection results of the dirt detection device and the robot's position data, a dirt map is automatically created, and corresponding business decisions and processing are executed based on the clustered dirt areas, resulting in a wide range of applications.

[0195] The above disclosures are only a few specific embodiments of the present application. However, the present application is not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present application.

Claims

1. A dirt detection device, characterized in that, Comprising: One or more sensors disposed on the robot body, including: an optical sensing component, and / or an acoustic wave sensing component, and / or an electromagnetic induction component, for obtaining information related to the ground of the area to be detected through the optical sensing component, and / or the acoustic wave sensing component, and / or the electromagnetic induction component; A processing module, configured to extract feature information from the obtained information related to the ground of the area to be detected, and perform dirt detection based on the feature information.

2. The dirt detection device according to claim 1, wherein The one or more sensors include at least one of the following: a camera sensor having a single camera or multiple cameras, a TOF sensor, a laser sensor, an ultrasonic sensor, an optical material identification sensor, an electromagnetic induction sensor, a spectral camera.

3. The dirt detection device according to claim 1, wherein The dirt detection device further includes: one or more line lasers, configured to emit laser lines to implement obstacle detection, wherein when the dirt detection device includes multiple line lasers, at least two laser lines emitted by the multiple line lasers intersect; The camera sensor included in the one or more sensors includes: one or more first cameras having a first fill light module, and / or one or more second cameras without the first fill light module, wherein the first camera and / or the second camera are used for time-division multiplexing to collect image information to respectively implement obstacle detection and dirt detection. When the camera sensor includes the first camera, the first fill light module is used to fill light for the first camera that collects image information. When the camera sensor includes the second camera, the dirt detection device further includes: one or more first fill light devices, configured to fill light for the second camera that collects image information; The processing module is configured to execute an obstacle detection method and a dirt detection method based on the image information collected by the first camera and / or the second camera through time-division multiplexing.

4. The dirt detection device according to claim 1, wherein The camera sensor included in the one or more sensors includes: multiple third cameras having a second fill light module, and / or multiple fourth cameras without the second fill light module, wherein the third camera and / or the fourth camera are used for collecting image information and implementing obstacle detection and dirt detection based on multi-view stereo vision. When the camera sensor includes the third camera, the second fill light module is used to fill light for the third camera that collects image information. When the camera sensor includes the fourth camera, the dirt detection device further includes: one or more second fill light devices, configured to fill light for the fourth camera that collects image information; The processing module is configured to execute an obstacle detection method and a dirt detection method based on the image information collected by the third camera and / or the fourth camera.

5. The dirt detection device according to claim 3 or 4, wherein In the case where the camera sensor includes: one or more first cameras having a first fill light module, and / or one or more second cameras not having the first fill light module: The one or more first cameras having a first fill light module and / or one or more second cameras not having a first fill light module are infrared cameras; When the dirt detection device includes: the first camera, the first fill light module provided in the first camera is an infrared fill light module, and the wavelength band corresponding to the infrared fill light module is the same as the wavelength band corresponding to the infrared camera; When the dirt detection device includes: the second camera and the first fill light device, the first fill light device is an independently provided infrared fill light, and the wavelength band corresponding to the infrared fill light is the same as the wavelength band corresponding to the infrared camera; In the case where the camera sensor includes: a plurality of third cameras having a second fill light module, and / or a plurality of fourth cameras not having the second fill light module: The plurality of third cameras having a second fill light module and / or a plurality of fourth cameras not having a second fill light module are infrared cameras; When the dirt detection device includes: the third camera, the second fill light module provided in the third camera is an infrared fill light module, and the wavelength band corresponding to the infrared fill light module is the same as the wavelength band corresponding to the infrared camera; When the dirt detection device includes: the fourth camera and the second fill light device, the second fill light device is an independently provided infrared fill light, and the wavelength band corresponding to the infrared fill light is the same as the wavelength band corresponding to the infrared camera.

6. The dirt detection device according to claim 3 or 4, wherein The dirt detection device further includes: one or more fifth cameras having a third fill light module, and / or one or more sixth cameras not having the third fill light module and one or more third fill light devices, wherein the fifth camera and the sixth camera are color cameras, and are used for time-division multiplexing to collect color image information to assist in realizing obstacle detection and dirt detection; When the dirt detection device includes: the fifth camera, the third fill light module provided in the fifth camera is a single-band or multi-band fill light module; When the dirt detection device includes: the sixth camera and the third fill light device, the third fill light device is a single-band or multi-band fill light provided independently on the housing of the dirt detection device.

7. The dirt detection device according to claim 3 or 4, wherein When the dirt detection device includes the first supplementary lighting module and / or the first supplementary lighting device, the first supplementary lighting module and / or the first supplementary lighting device are configured to emit visible light or non-visible light to the ground. Among them, the first supplementary lighting module and / or the first supplementary lighting device are configured to emit light with different brightness to the ground for every two adjacent frames, or emit light with a constant brightness in the same band for all detection frames, or emit light with a changing brightness in the same band for all detection frames, or emit light with a constant brightness in multiple bands for all detection frames, or emit light with a changing brightness in multiple bands for all detection frames; When the dirt detection device includes the second supplementary lighting module and / or the second supplementary lighting device, the second supplementary lighting module and / or the second supplementary lighting device are configured to emit visible light or non-visible light to the ground. Among them, the first supplementary lighting module and / or the first supplementary lighting device are configured to emit light with different brightness to the ground for every two adjacent frames, or emit light with a constant brightness in the same band for all detection frames, or emit light with a changing brightness in the same band for all detection frames, or emit light with a constant brightness in multiple bands for all detection frames, or emit light with a changing brightness in multiple bands for all detection frames.

8. The dirt detection device according to claim 7, wherein When the dirt detection device includes the first supplementary lighting module and / or the first supplementary lighting device, the first supplementary lighting module and / or the first supplementary lighting device are configured to emit light with different brightness to the ground for every two adjacent frames or emit light with different bands to the ground for every two adjacent frames; When the dirt detection device includes the second supplementary lighting module and / or the second supplementary lighting device, the second supplementary lighting module and / or the second supplementary lighting device are configured to emit light with different brightness to the ground for every two adjacent frames or emit light with different bands to the ground for every two adjacent frames; The processing module is configured to extract gradient features with gradient values greater than a predetermined gradient threshold from the ground image data collected by each camera, obtain dirt-related data corresponding to each camera sensor, match and verify the dirt-related data corresponding to a single camera sensor for two adjacent frames before and after, or match and verify the dirt-related data within the line-of-sight overlapping area corresponding to multiple camera sensors for two adjacent frames before and after. When the gradient change amount is greater than a predetermined change amount threshold, obtain the dirt data corresponding to a single camera sensor or multiple camera sensors.

9. The dirt detection device according to any one of claims 1 to 8, characterized in that, Further comprising: A deep learning module, configured to train based on dirt features to construct a deep learning object detection model. When the robot is located in the area to be detected, input the ground image data obtained by the one or more sensors into the deep learning object detection model to detect dirt, and determine the dirt category and / or quantity based on the dirt detection result.

10. A method for detecting dirt of a dirt detection device according to any one of claims 1 to 9, characterized in that, Comprising: Perform a detection operation on the ground image obtained in real time to obtain one or more local areas; For each of the local regions, height information corresponding to each point in the local region is respectively obtained based on the parallax of stereo vision; A screening operation is respectively performed in each of the local regions to determine one or more points in the local region whose height information satisfies a predetermined range.

11. The dirt detection method according to claim 10, wherein For each of the local regions, respectively obtaining height information corresponding to each point in the local region based on the parallax of stereo vision includes: Construct a binocular camera stereo vision system; For each point in each of the local regions, coordinate information of the current point in the left and right cameras is obtained respectively, and the parallax of stereo vision is calculated according to the coordinate information; According to the parallax, the height information of the current point in the left camera coordinate system or the right camera coordinate system is calculated.

12. The dirt detection method according to claim 11, wherein The parallax d of stereo vision is calculated by the following formula: d = u1 - u2, where the coordinate information of the current point in one camera of the binocular camera is P1(u1, v1), and the coordinate information of the current point in the other camera of the binocular camera is P2(u2, v2); Calculate the height information Z of this point in the left camera coordinate system or the right camera coordinate system through the following formula c :[[]]END]] Where b is the baseline spacing and f is the focal length of the left and right cameras.

13. The dirt detection method according to claim 10, characterized in that, After determining one or more points in the local region whose height information satisfies a predetermined range, it further includes: According to the depth information and height information corresponding to each point previously determined on the ground image, and the depth information and height information corresponding to the one or more points that satisfy the predetermined range, the dirt confidence of each point among the one or more points is calculated and obtained; The obtained dirt confidences are respectively compared with a predetermined confidence threshold, and the points corresponding to the dirt confidences greater than or equal to the predetermined confidence threshold are determined as dirt spots.

14. The dirt detection method according to any one of claims 10 to 13, characterized in that After determining one or more points in the local region whose height information satisfies a predetermined range, it further includes at least one of the following: Determine the resolution of the dirt map, set the grid corresponding to the determined resolution at the ground position, place the dirt detection device with multiple cameras or a single camera on the ground where the grid is set, determine the grid position where the dirt spot is located, construct the corresponding relationship between the dirt spot and the grid, and draw a grid map according to the corresponding relationship; Perform a planar projection based on the depth information of the stain, and convert the position information of the stain to the grid coordinate system and draw a grid map through the following formula: g x = x c / r + w / 2, g y = y c / r + h / 2, where (g x , g y ) are the coordinates of any point in the stain in the grid coordinate system, r is the grid resolution, and w and h are the length and width of the grid respectively.

15. A method for detecting dirt of a dirt detection device according to any one of claims 1 to 9, characterized in that, It includes: Use a single camera to collect ground images in real time and obtain monocular two-dimensional images; Extract features for depth estimation from the monocular two-dimensional images, and obtain the depth values of each pixel point in the monocular two-dimensional images according to the features for depth estimation; According to the depth values of each pixel point in the monocular two-dimensional images, depth gradient information is obtained; The depth gradient information is compared with a predetermined depth gradient threshold, and the dirt regions are extracted according to the depth gradient information greater than or equal to the predetermined depth gradient threshold.

16. The dirt detection method according to claim 15, characterized in that, Obtaining depth gradient information according to the depth values of each pixel point in the monocular two-dimensional images includes at least one of the following: According to the depth values of each pixel point in the monocular two-dimensional images, the ground plane information is fitted, and based on the fitted ground plane information, statistics are performed to obtain the depth gradient information; Extract the depth gradient information of the local area according to the depth values of each pixel point in the monocular two-dimensional image.

17. The dirt detection method according to claim 15 or 16, characterized in that, Extracting features for depth estimation from the monocular two-dimensional image and obtaining the depth values of each pixel point in the monocular two-dimensional image according to the features for depth estimation includes: Using the feature extraction module in the pre-trained deep learning model to extract features for depth estimation from the monocular two-dimensional image; Using the depth value acquisition module in the deep learning model to map the features for depth estimation to obtain the depth values of each pixel point in the monocular two-dimensional image, where the deep learning model includes: a fully convolutional neural network, and / or, a conditional random field is further connected to the backend of the deep learning model.

18. A method for jointly calibrating laser and dirt of a dirt detection device according to any one of claims 1 to 9, characterized in that Including: Determine the dirt resolution, set the grid corresponding to the dirt resolution at the ground position, place the dirt detection device with a single camera or multiple cameras on the ground, determine the position information corresponding to the one or more dirt spots in the collected image, and construct the first correspondence between the one or more dirt spots and the position information; Obtain the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two of the multiple line lasers, project it in the direction towards the ground, determine the position information corresponding to the light plane projection, and construct the second correspondence between the light plane projection and the position information; Draw a map according to the first correspondence and the second correspondence to realize the joint calibration of the laser and the dirt.

19. The laser and dirt combined calibration method according to claim 18, wherein Obtaining the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two of the multiple line lasers includes: Set marker boards at different positions, and use a single camera or multiple cameras to take multiple images of the marker boards, where the multiple images include: the projection of the laser line emitted by a single line laser on the marker board, or the projection of the intersecting laser lines emitted by at least two of the multiple line lasers on the marker board; Respectively extract the projection center lines of the laser lines on the marker board from the multiple images. When extracting the projection center lines of the intersecting laser lines, determine the light plane corresponding to the extracted projection center lines of the laser lines according to the spatial position relationship of the intersecting laser lines; According to each extracted projection center line, calculate the discrete points on the intersection line of the light plane and the marker board plane, and fit the discrete points to obtain the equations of multiple intersection lines; According to the equations of the multiple intersection lines, obtain the normal vectors of all the light planes, and determine the optimal solution of the normal vectors from the normal vectors of all the light planes; In the equations of the multiple intersection lines, eliminate the straight lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds the predetermined deviation threshold to obtain the direction vectors corresponding to the remaining straight lines; Use the direction vectors corresponding to the remaining straight lines to construct an equation, and calculate the normal vector of the light plane formed by the laser line emitted by the single line laser or the light plane formed by the intersecting laser lines.

20. A robot, characterized in that, Including: One or more dirt detection devices as described in any one of claims 1 to 9, wherein the one or more sensors of the dirt detection device are disposed in the front area of the upper body and / or the rear area of the body and / or the bottom area of the body of the robot.

Images