Dirt detection method, device and equipment
By extracting ground depth gradient information using a monocular camera and a deep learning model, this technology solves the problems of high manpower consumption and low detection rate in existing dirt detection methods, achieving automated and intelligent dirt detection and improving detection accuracy and robot cleaning efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INDEMIND TECH CO LTD
- Filing Date
- 2024-10-17
- Publication Date
- 2026-04-17
AI Technical Summary
Current technologies for detecting dirt require a lot of manpower and have a low detection rate, making it impossible to achieve targeted cleaning.
A single camera is used to acquire ground images in real time. Depth gradient information is extracted through a deep learning model, and the dirty area is determined by combining monocular visual geometric relationships, thus achieving automated and intelligent detection.
It can improve the dirt detection rate, enhance the cleaning efficiency and intelligence of robots, and enable early detection of dirty areas without requiring a large number of manpower.
Smart Images
Figure CN121883347A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and equipment for detecting dirt and grime. Background Technology
[0002] In environments such as hotels, supermarkets, shopping malls, factories, hospitals, and homes, robots are gradually replacing human cleaners due to their efficient, accurate, and continuous working capabilities. However, these work scenarios are complex and changeable. Robots often perform cleaning tasks by traversing the entire floor, making it impossible to detect the degree of dirt on the ground and clean accordingly. This affects the effectiveness and intelligence of the robots in performing their tasks.
[0003] Currently, robots on the market use a full-scene traversal cleaning method to achieve cleaning tasks. This means that the robot is controlled to traverse the entire target work scene without discrimination to achieve cleaning of the work scene.
[0004] Dirt detection technology is a technical means used to detect dirt and pollutants on the surface of objects or in the environment. It is widely used in manufacturing, service industry, medical and health care and environmental protection, aiming to protect public health and maintain environmental cleanliness. Among related technologies, dirt detection is mainly achieved through the following methods: (1) manual detection; (2) auxiliary manual detection; (3) infrared brush plate detection;
[0005] There are several methods for detecting dirt. Manual detection requires significant manpower and is prone to inaccuracy due to the significant influence of lighting conditions. Assisted manual detection involves adding supplemental lighting to the machine (e.g., shining green light on the ground to enhance dust visibility), which improves the success rate but still requires manual intervention. Infrared brush detection involves adding infrared sensors to the robot's brush. If the ground is dirty, the brush will be even dirtier, altering the reflection from the infrared sensor and thus detecting the dirt. However, this method has a low detection rate, particularly in lightly soiled areas or with light-colored dirt like clear water. Furthermore, it only detects dirt when the robot has reached or passed through the area, lacking the ability to detect dirt before it reaches the target area and thus hindering targeted cleaning strategies. Summary of the Invention
[0006] The main objective of this invention is to disclose a method, apparatus, and equipment for detecting dirt, so as to at least solve the problems in related technologies such as the need for a large amount of manpower and the low detection rate in achieving dirt detection.
[0007] According to one aspect of the present invention, a method for detecting dirt and grime is provided.
[0008] The dirt detection method according to the present invention includes: acquiring ground images in real time using a single camera to obtain a monocular two-dimensional image; extracting features for depth estimation from the monocular two-dimensional image, and obtaining the depth value of each pixel in the monocular two-dimensional image based on the features for depth estimation; obtaining depth gradient information based on the depth value of each pixel in the monocular two-dimensional image; comparing the depth gradient information with a predetermined depth gradient threshold, and extracting the dirt area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
[0009] According to another aspect of the present invention, a dirt detection device is provided.
[0010] The dirt detection device according to the present invention includes: an acquisition module for acquiring ground images in real time using a single camera to obtain a monocular two-dimensional image; a first acquisition module for extracting features for depth estimation from the monocular two-dimensional image and obtaining the depth value of each pixel in the monocular two-dimensional image based on the features for depth estimation; a second acquisition module for obtaining depth gradient information based on the depth value of each pixel in the monocular two-dimensional image; and an extraction module for comparing the depth gradient information with a predetermined depth gradient threshold and extracting the dirt area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
[0011] According to another aspect of the present invention, a dirt detection device is provided.
[0012] The dirt detection device according to the present invention comprises: one or more line lasers for emitting laser lines, wherein, when the dirt detection device comprises multiple line lasers, at least two of the multiple line lasers emit laser lines that intersect; one or more cameras with supplementary lighting modules, and / or one or more cameras without supplementary lighting modules and one or more supplementary lighting devices, wherein the one or more cameras with supplementary lighting modules and / or the one or more cameras without supplementary lighting modules are used to time-division multiplex the acquisition of image information when the processor executes the obstacle detection method and the dirt detection method, and the supplementary lighting modules and / or the supplementary lighting devices are used to provide supplementary lighting to the cameras acquiring the image information; and a processor is used to implement the obstacle detection method based on the laser lines emitted and reflected by the single or multiple line lasers, and to execute the dirt detection method as described above based on the image information acquired by the one or more cameras with supplementary lighting modules and / or the one or more cameras without supplementary lighting function.
[0013] According to the present invention, for a monocular two-dimensional image captured by a single camera, depth estimation features are extracted, and the depth value of each pixel in the monocular two-dimensional image is obtained based on the depth estimation features. Then, the depth gradient information is obtained by fitting ground plane information based on the depth values, and the dirty area is extracted based on the depth gradient information. By using the above method for dirt detection, dirt detection can be automated and intelligent, without requiring a lot of manpower, and the dirt detection rate is effectively improved. Attached Figure Description
[0014] Figure 1 This is a flowchart of a dirt detection method according to an embodiment of the present invention;
[0015] Figure 2 This is a structural block diagram of a system for extracting depth values of individual pixels in a monocular two-dimensional image according to a preferred embodiment of the present invention;
[0016] Figure 3 This is a flowchart of a dirt detection method according to a preferred embodiment of the present invention;
[0017] Figure 4 This is a structural block diagram of a dirt detection device according to an embodiment of the present invention;
[0018] Figure 5 This is a structural block diagram of a dirt detection device according to a preferred embodiment of the present invention;
[0019] Figure 6 This is a structural block diagram of a dirt detection device according to an embodiment of the present invention. Detailed Implementation
[0020] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.
[0021] According to an embodiment of the present invention, a method for detecting dirt and grime is provided.
[0022] Figure 1 This is a flowchart of a dirt detection method according to an embodiment of the present invention. Figure 1 As shown, the dirt detection method includes:
[0023] Step S101: Use a single camera to acquire ground images in real time to obtain monocular two-dimensional images;
[0024] Step S103: Extract features for depth estimation from the above monocular two-dimensional image, and obtain the depth value of each pixel in the above monocular two-dimensional image based on the features for depth estimation.
[0025] Step S105: Obtain depth gradient information based on the depth values of each pixel in the monocular two-dimensional image above;
[0026] Step S107: Compare the above depth gradient information with a predetermined depth gradient threshold, and extract the dirty area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
[0027] In related technologies, methods such as manual inspection, assisted manual inspection, and infrared brush plate inspection are used to detect dirt, which require a lot of manpower and have a low detection rate. Figure 1 The dirt detection method shown extracts depth estimation features from a monocular 2D image captured by a single camera, obtains the depth value of each pixel in the monocular 2D image based on the depth estimation features, then obtains depth gradient information based on the depth values, and finally extracts the dirt region based on the depth gradient information. Using the above method, dirt can be detected automatically and intelligently without requiring a lot of manpower, and the dirt detection rate is effectively improved.
[0028] It should be noted that monocular depth estimation predicts scene depth information using a single image. For a 2D image m, a function F is needed to calculate its corresponding depth d. This process can be written as: d = F(m), where the depth information d represents the actual distance between each pixel in the 2D image projected from the 3D object and the monocular camera.
[0029] For monocular 2D images captured by a single camera, monocular depth estimation suffers from scale uncertainty because a single image cannot provide absolute distance information. However, in this application, dirt detection is not based on absolute depth values, but rather on dirt extraction based on relative depth information (e.g., depth gradient information from fitted ground plane information).
[0030] Preferably, in step S105 above, obtaining the depth gradient information based on the depth values of each pixel in the monocular two-dimensional image may include at least one of the following:
[0031] Based on the depth values of each pixel in the monocular two-dimensional image, ground plane information is fitted, and depth gradient information is obtained by statistical analysis based on the fitted ground plane information.
[0032] Based on the depth values of each pixel in the monocular two-dimensional image, the depth gradient information of the local region is extracted.
[0033] In the process of obtaining depth gradient information based on the depth values of each pixel in the aforementioned monocular 2D image, two schemes can be used. The first scheme, from a global perspective, first uses the depth values of each pixel to fit the ground plane information of the entire region, and then performs statistical analysis on the fitted ground plane information to obtain the depth gradient information. The second scheme, from a local region perspective, first uses the depth values of each pixel in the monocular 2D image, and then extracts the depth gradient information of the local region based on these pixel values. Of course, in specific implementations, both schemes can be used in parallel for dirt detection, or either scheme can be used for dirt detection.
[0034] Preferably, extracting features for depth estimation from the monocular two-dimensional image and obtaining the depth value of each pixel in the monocular two-dimensional image based on the features for depth estimation may further include the following processing: using a feature extraction module in a pre-trained deep learning model to extract features for depth estimation from the monocular two-dimensional image; using a depth value acquisition module in the deep learning model to map the features for depth estimation and obtain the depth value of each pixel in the monocular two-dimensional image.
[0035] In the preferred implementation process, monocular stereo vision calculation is performed on the monocular two-dimensional image to extract ground depth information, such as... Figure 2 As shown, the specific steps include:
[0036] Step 1: Use the feature extraction module in a pre-trained deep learning model (e.g., Convolutional Neural Network (CNN), ResNet, etc., where the deep learning model can obtain the size, shape and relative position of objects in the image) to extract features for depth estimation from the monocular 2D image, such as image edge features, image texture or other features.
[0037] Step 2: The depth value acquisition module in the deep learning model maps the features used for depth estimation to obtain the depth value of each pixel in the monocular two-dimensional image.
[0038] Preferably, before extracting features for depth estimation from the monocular two-dimensional image using the feature extraction module in the pre-trained deep learning model, the method may further include: optimizing the deep learning model using the following specific loss function L during the training process.
[0039] in, x represents the input vector, y represents the response variable, t represents the time parameter, ε represents the error vector, and max() represents the function that takes the maximum value. This represents the input vector x and its corresponding value at time t. The L1 norm of the difference This represents the input vector ε and its corresponding value at time t. The L1 norm of the difference, E x,y,t,ε [] indicates that, given x, y, t, ε, and The expected maximum value.
[0040] The L1 norm is the sum of the absolute values of the elements in a vector. Given two vectors x = (x1, x2, ..., xn),... n ) and x t =(x t1 x t2 , ..., x tn ), and two error vectors ε = (ε1, ε2, ..., ε n ) and ε t =(ε t1 ,ε t2 ,…,ε tn ), can calculate xx separately t and ε-ε t The L1 norm. ||xx t ||1=∣x1-x t1 |+|x2-x t2 |+…+|x n -x tn |;||ε-ε t ||1=∣ε1-ε t1 ∣+∣ε2-ε t2 ∣+…+∣ε n -ε tn |
[0041] In the optimal implementation process, when training a deep learning model, a loss function can be used to update the model parameters. That is, training a deep learning model can be viewed as solving for the model parameters, with parameter updates relying on the gradient of the loss function to provide the optimization direction. The loss function measures the difference between the model's predictions and the true values; it is a non-negative real-valued function. The smaller the loss function, the better the model's robustness. The training process of a deep learning model is an optimization problem, with the goal of minimizing the loss function. By continuously adjusting the model's parameters, the loss function gradually decreases, and the model's predictive ability gradually improves.
[0042] Of course, the loss function described above can be constructed and used. To optimize deep learning models, you can also use L1 loss function, L2 loss function, etc.
[0043] Preferably, the deep learning model includes: a fully convolutional neural network, and / or, a conditional random field is connected to the backend of the deep learning model.
[0044] In the preferred implementation process, after obtaining the depth values of each pixel, appropriate processing operations can be combined to improve the accuracy of depth estimation by filtering out interference. For example, the network layers can be fully convolutional (e.g., using a fully convolutional neural network FCN), and / or a conditional random field can be added to the back end of the deep learning model to remove noise and improve the overall accuracy of the depth estimation algorithm. That is, a fully convolutional neural network FCN can be used, or a conditional random field can be added to the back end of the deep learning model, or, as... Figure 2 As shown, while employing a fully convolutional neural network (FCN), a conditional random field is also connected to the backend of the deep learning model.
[0045] The application of Fully Convolutional Networks (FCNs) in noise removal is mainly reflected in their fully convolutional characteristics. When removing noise, FCNs learn the mapping relationship from noisy images to noise-free images by training on a large dataset of both noisy and noise-free images. This method performs well in handling image noise, especially in dealing with random noise.
[0046] The application of Conditional Random Fields (CRFs) in noise removal focuses on optimizing the smoothness of labeling. As a probabilistic graphical model, CRFs can further refine the labeling results based on FCNs, ensuring a smooth transition between labels. By considering the spatial relationships between pixels, CRFs can better handle the dependencies between labels, thereby improving the accuracy and consistency of the denoising results.
[0047] Preferably, after extracting the dirty area based on depth gradient information greater than or equal to a predetermined depth gradient threshold, the following processing may be included: determining the height information corresponding to each point in the dirty area based on monocular visual geometric relationships; performing a filtering operation in the dirty area respectively to determine one or more points in the sub-area whose height information meets the predetermined range.
[0048] The height information corresponding to each point in the aforementioned dirty area, determined based on monocular visual geometric relationships, includes:
[0049] The height value z corresponding to each point in the above-mentioned dirty area is calculated using the following formula:
[0050]
[0051] Where v0 is the position of the horizontal plane at the height of a single camera in the monocular 2D image, v1 and v2 are the top and bottom positions of the dirt spot in the monocular 2D image, and the height of a single camera perpendicular to the ground is h.
[0052] In the preferred implementation process, the soiled areas extracted in step S109 can be further verified and screened. Height information can eliminate interference from various floor types. For example, in this case, regardless of whether the floor is light, dark, or patterned, the height of the flat surface will fall within a certain range, such as greater than 0 mm and less than or equal to 1 mm. Therefore, based on the height values corresponding to each point in the soiled area, it is possible to effectively distinguish between soiled and unsoiled conditions. Specifically:
[0053] v0 represents the position of the horizontal plane at the height of the monocular camera in the monocular 2D image, while v1 and v2 represent the top and bottom positions of the dirt points in the monocular 2D image. The height of the monocular camera perpendicular to the ground is h. Based on the geometric relationship of monocular vision, the above formula can be used to calculate the height value z of the dirt points in the image. Then, dirt points are filtered out using the height information z, ensuring that the detected dirt height is within a certain range, for example, greater than 1 mm and less than or equal to 10 mm. For the previously detected dirty areas, the height information corresponding to each point in the dirty area is used for further judgment and filtering, which can improve the accuracy and precision of dirt detection.
[0054] The following combination Figure 3 The preferred embodiments described above are further described below.
[0055] Figure 3 This is a flowchart of a dirt detection method according to a preferred embodiment of the present invention. Figure 3 As shown, this method for detecting dirt includes the following processes:
[0056] Step S301: Use a single camera to acquire ground images in real time to obtain monocular two-dimensional images;
[0057] The camera mentioned above can be a camera with a supplementary lighting module or a camera without a supplementary lighting module. If a camera without a supplementary lighting module is used, one or more supplementary lighting devices (e.g., supplementary lights) can be set to illuminate the ground when the camera captures ground images. Preferably, the camera can be an infrared single-pass camera.
[0058] The aforementioned fill light can use a flashing fill light method, that is, to emit light of different brightness to the ground in every two adjacent frames. Of course, this also includes: in every two adjacent frames, one frame emits light of a certain brightness and the other frame does not emit light.
[0059] The aforementioned supplementary lights can also adopt a single-band constant brightness supplementary lighting method, that is, sending light of constant brightness in the same band to the ground in all detection frames.
[0060] The aforementioned fill lights can also employ a multi-band constant brightness fill light method, that is, emitting light of different bands to the ground in every two adjacent frames.
[0061] The aforementioned supplementary lights can emit both visible and non-visible light.
[0062] Step S303: The feature extraction module in the fully convolutional neural network FCN extracts features for depth estimation from the above monocular two-dimensional image, such as image edge features, image texture, or other features;
[0063] Specifically, during the training of the aforementioned deep learning model, a specific loss function L is used to optimize the deep learning model.
[0064] FCN learns the mapping relationship from noisy images to noiseless images by training on a large dataset of noisy and noiseless images, thereby improving the ability to process image noise.
[0065] Step S305: The depth value acquisition module in FCN maps the features used for depth estimation to obtain the depth value of each pixel in the monocular two-dimensional image.
[0066] The backend of the aforementioned deep learning model also incorporates a Conditional Random Field (CRF). As a probabilistic graphical model, CRF can further refine the labeling results based on FCN, ensuring a smooth transition of labels. By considering the spatial relationships between pixels, CRF can better handle the dependencies between labels, thereby improving the accuracy and consistency of the denoising results.
[0067] Step S307: Based on the depth values of each pixel in the above monocular two-dimensional image, the ground plane information is obtained by fitting.
[0068] Step S309: Based on the fitted ground plane information above, perform statistics to obtain depth gradient information;
[0069] Step S311: Compare the above depth gradient information with a predetermined depth gradient threshold, and extract the dirty area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
[0070] Step S313: Determine the height information corresponding to each point in the above-mentioned dirty area based on the monocular visual geometric relationship;
[0071] Step S315: Perform a screening operation in the above-mentioned dirty areas respectively to determine one or more points in the above-mentioned sub-areas whose height information meets the predetermined range (e.g., greater than 1 mm and less than or equal to 10 mm).
[0072] Therefore, in the dirty area extracted in step S311, step S315 further filters out dirty spots using height information z. This improves the accuracy and precision of dirt detection.
[0073] According to an embodiment of the present invention, a dirt detection device is also provided.
[0074] Figure 4 This is a structural block diagram of a dirt detection device according to an embodiment of the present invention. Figure 4 As shown, the dirt detection device includes: a data acquisition module 40, used to acquire ground images in real time using a single camera to obtain a monocular two-dimensional image; a first acquisition module 42, used to extract features for depth estimation from the monocular two-dimensional image and obtain the depth value of each pixel in the monocular two-dimensional image based on the features for depth estimation; a second acquisition module 44, used to obtain depth gradient information based on the depth value of each pixel in the monocular two-dimensional image; and an extraction module 46, used to compare the depth gradient information with a predetermined depth gradient threshold and extract the dirty area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
[0075] use Figure 4 The dirt detection device shown in the figure has an acquisition module that extracts depth estimation features from a monocular 2D image captured by a single camera, and obtains the depth value of each pixel in the monocular 2D image based on the depth estimation features. Then, it obtains depth gradient information based on the depth value, and extracts the dirt area based on the depth gradient information. Using the above method, dirt can be detected automatically and intelligently without consuming a lot of manpower, and the dirt detection rate can be effectively improved.
[0076] In the specific implementation process, the second acquisition module 44 can adopt two schemes to achieve dirt detection. As mentioned above, the first scheme is to fit the ground plane information of the whole region based on the depth value of each pixel in the monocular two-dimensional image, and then perform statistics based on the fitted ground plane information to obtain the depth gradient information; the second scheme is to extract the depth gradient information of the local region based on the depth value of each pixel in the monocular two-dimensional image. Of course, the above two schemes can also be implemented in parallel.
[0077] Preferably, the first acquisition module 42 includes a deep learning model, which further includes a feature extraction module for extracting features for depth estimation from the monocular two-dimensional image; and a depth value acquisition module for mapping the features for depth estimation to obtain the depth value of each pixel in the monocular two-dimensional image. See details [link to relevant documentation]. Figure 2 .
[0078] Preferably, such as Figure 5As shown, the above-mentioned device may further include: a determining module 48, used to determine the height information corresponding to each point in the above-mentioned dirty area based on monocular visual geometric relationship; and a filtering module 50, used to perform filtering operations in the above-mentioned dirty area respectively, and determine one or more points in the above-mentioned sub-area whose height information meets the predetermined range.
[0079] It should be noted that the preferred implementation method for the combination of the modules in the above-mentioned dirt detection device is described in the corresponding reference. Figures 1 to 3 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.
[0080] According to an embodiment of the present invention, a dirt detection device is also provided.
[0081] Figure 6 This is a structural block diagram of a dirt detection device according to an embodiment of the present invention. Figure 6 As shown, the dirt detection device 6 includes: one or more line lasers 60 for emitting laser lines to implement an obstacle detection method, wherein, when the dirt detection device includes multiple line lasers, at least two of the multiple line lasers emit laser lines that intersect; one or more cameras 62 with supplementary lighting modules, and / or one or more cameras 64 without supplementary lighting modules and one or more supplementary lighting devices 66, wherein the one or more cameras 62 with supplementary lighting modules and / or the one or more cameras 64 without supplementary lighting modules are used to time-division multiplex the acquired image information when the processor executes the obstacle detection method and the dirt detection method, and the supplementary lighting modules and / or the supplementary lighting devices 66 are used to provide supplementary lighting to the cameras acquiring image information; a processor 68 is used to execute the obstacle detection method and the dirt detection method described above based on the time-division multiplexing of the acquired image information from the one or more cameras 62 with supplementary lighting modules and / or the one or more cameras 64 without supplementary lighting modules.
[0082] It should be noted that the aforementioned dirt detection device may simultaneously include one or more cameras 62 with supplementary lighting modules and one or more cameras 64 without supplementary lighting modules. Alternatively, the aforementioned dirt detection device may include only one or more cameras 62 with supplementary lighting modules, or only one or more cameras 64 without supplementary lighting modules. All of the above embodiments are within the protection scope of this invention. For the aforementioned cameras 62 with supplementary lighting modules, there is no need to provide a separate supplementary lighting device 66 to illuminate the camera 62. For the aforementioned cameras 64 without supplementary lighting modules, since the camera 64 does not have a self-illuminating function, a separate supplementary lighting device 66 is required to illuminate the camera 64.
[0083] Figure 6An example is shown that includes only a camera 64 without a fill light module, a fill light device 66, and two line lasers 60_1 and 60_2.
[0084] The processor 68 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.
[0085] When the aforementioned processor 68 is working, it executes as follows: Figures 1 to 3 The dirt detection method in the illustrated embodiment.
[0086] It should be noted that specific details regarding the aforementioned dirt detection equipment can be found in the relevant references. Figures 1 to 3 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.
[0087] In summary, by utilizing the above-described embodiments of this invention, the dirt detection method of this invention, based on monocular stereo vision calculation and employing a deep learning model, acquires ground depth gradient information, enabling automated and intelligent dirt detection without requiring significant manpower and effectively improving the dirt detection rate. Furthermore, after extracting the dirty area, the height information corresponding to each point in the dirty area is determined based on monocular visual geometric relationships. Based on this height information, one or more dirty points meeting a predetermined range are further screened and verified, improving the accuracy and precision of dirt detection. This scheme helps robots achieve automated and intelligent dirt detection, and allows for early detection of dirt before the robot reaches the dirty area, improving the robot's intelligence level and work efficiency.
[0088] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A method for detecting dirt and grime, characterized in that, include: A single camera is used to capture ground images in real time, obtaining monocular two-dimensional images; Features for depth estimation are extracted from the monocular 2D image, and the depth value of each pixel in the monocular 2D image is obtained based on the features for depth estimation. Depth gradient information is obtained based on the depth values of each pixel in the monocular two-dimensional image; The depth gradient information is compared with a predetermined depth gradient threshold, and the dirty area is extracted based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
2. The method according to claim 1, characterized in that, Based on the depth values of each pixel in the monocular two-dimensional image, the depth gradient information obtained includes at least one of the following: Based on the depth values of each pixel in the monocular two-dimensional image, ground plane information is fitted, and depth gradient information is obtained by statistical analysis based on the fitted ground plane information. Based on the depth values of each pixel in the monocular two-dimensional image, the depth gradient information of the local region is extracted.
3. The method according to claim 1 or 2, characterized in that, Extracting features for depth estimation from the monocular 2D image, and obtaining the depth value of each pixel in the monocular 2D image based on the features for depth estimation, includes: The feature extraction module in a pre-trained deep learning model is used to extract features for depth estimation from the monocular two-dimensional image. The depth value acquisition module in the deep learning model is used to map the features used for depth estimation to obtain the depth value of each pixel in the monocular two-dimensional image.
4. The method according to claim 3, characterized in that, Before extracting features for depth estimation from the monocular two-dimensional image using the feature extraction module in a pre-trained deep learning model, the method further includes: During the training of the deep learning model, the following specific loss function L is used to optimize the deep learning model; in, x represents the input vector, y represents the response variable, t represents the time parameter, ε represents the error vector, and max() represents the function that takes the maximum value. This represents the input vector x and its corresponding value at time t. The L1 norm of the difference This represents the input vector ε and its corresponding value at time t. The L1 norm of the difference, E x,y,t,ε [] indicates that, given x, y, t, ε, and The expected maximum value.
5. The method according to claim 3, characterized in that, The deep learning model includes: a fully convolutional neural network, and / or, a conditional random field as the backend of the deep learning model.
6. The method according to claim 1, characterized in that, After extracting the contaminated region based on depth gradient information that is greater than or equal to a predetermined depth gradient threshold, the process also includes: The height information corresponding to each point in the dirty area is determined based on the geometric relationship of monocular vision. A screening operation is performed in each of the dirty areas to determine one or more points in the sub-area whose height information meets a predetermined range.
7. The method according to claim 6, characterized in that, Determining the height information corresponding to each point in the dirty area based on monocular visual geometric relationships includes: The height value z corresponding to each point in the dirty area is calculated using the following formula: Where v0 is the position of the horizontal plane at the height of a single camera in the monocular 2D image, v1 and v2 are the top and bottom positions of the dirt spot in the monocular 2D image, and the height of a single camera perpendicular to the ground is h.
8. A dirt detection device, characterized in that, include: The acquisition module is used to acquire ground images in real time using a single camera to obtain monocular two-dimensional images; The first acquisition module is used to extract features for depth estimation from the monocular two-dimensional image, and to obtain the depth value of each pixel in the monocular two-dimensional image based on the features for depth estimation. The second acquisition module is used to obtain depth gradient information based on the depth values of each pixel in the monocular two-dimensional image; The extraction module is used to compare the depth gradient information with a predetermined depth gradient threshold, and extract the dirty area based on the depth gradient information that is greater than or equal to the predetermined depth gradient threshold.
9. The apparatus according to claim 8, characterized in that, The first acquisition module includes: a deep learning model, and the deep learning model further includes: The feature extraction module is used to extract features for depth estimation from the monocular two-dimensional image; The depth value acquisition module is used to map the features used for depth estimation to obtain the depth value of each pixel in the monocular two-dimensional image.
10. The apparatus according to claim 8 or 9, characterized in that, Also includes: The determination module is used to determine the height information corresponding to each point in the dirty area based on monocular visual geometric relationships; A filtering module is used to perform filtering operations in the dirty areas respectively, and determine one or more points in the sub-area whose height information meets a predetermined range.
11. A dirt detection device, characterized in that, include: One or more line lasers are used to emit laser lines to achieve an obstacle detection method, wherein when the dirt detection device includes multiple line lasers, at least two of the multiple line lasers emit laser lines that intersect. One or more cameras with a supplementary lighting module, and / or one or more cameras without a supplementary lighting module and one or more supplementary lighting devices, wherein the one or more cameras with a supplementary lighting module and / or the one or more cameras without a supplementary lighting module are used to time-division multiplex the acquired image information when the processor executes the obstacle detection method and the dirt detection method, and the supplementary lighting module and / or the supplementary lighting device are used to provide supplementary lighting to the cameras that acquire image information; A processor is configured to perform an obstacle detection method and a dirt detection method as described in any one of claims 1 to 7, based on image information acquired in a time-division multiplexing manner from one or more cameras with supplementary lighting modules and / or one or more cameras without supplementary lighting modules.