A camera dirt detection method, apparatus, device, medium and program product

Abstract

This invention discloses a method, apparatus, device, medium, and program product for detecting dirt in a camera. The method includes: acquiring real-time frame images of a camera under test; performing dirt detection on the camera under test using a multi-dimensional dirt detection strategy based on the real-time frame images to obtain multi-dimensional dirt detection results; and determining the target dirt detection result of the camera under test based on the multi-dimensional dirt detection results. This invention can identify dirt from multiple different dimensions, effectively avoiding the situation where environmental occlusion is misjudged as dirt occlusion when a single-dimensional anomaly is caused by environmental obstruction. It can also adapt to various types of dirt, avoiding missed dirt detection and significantly improving the detection accuracy of camera dirt.

Description

Technical Field

[0001] This invention relates to the field of self-moving device technology, and in particular to a method, apparatus, self-moving device, computer-readable storage medium, and computer program product for detecting dirt in a camera. Background Technology

[0002] Currently, self-moving devices (such as yard robots) are widely used in daily life to automate various functions, including snow removal, lawn mowing, and leaf blowing. These devices are typically equipped with cameras, but in complex working environments, these cameras can become obstructed. This obstruction can be caused by dirt such as snow, water, moisture, mud, or dust, or by environmental factors like strong light or tall grass. If this dirt and obstruction are not addressed promptly, it will continuously affect the accuracy and stability of the self-moving device's visual algorithms. Therefore, it is necessary to perform timely and accurate detection of dirt on the self-moving device's cameras.

[0003] However, existing camera occlusion detection methods have a high false detection rate for camera dirt. For example, existing technologies are prone to misinterpreting environmental disturbances such as tree shadows, sudden changes in light, flying insects, or flat walls as dirt obstructing the camera, resulting in low detection accuracy for camera dirt. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and program product for detecting dirt in cameras, in order to solve the technical problem of low detection accuracy of dirt in existing technologies.

[0005] To address the aforementioned technical problems, a first aspect of this invention provides a method for detecting dirt in a camera, comprising: Acquire real-time frame images from the camera under test; Based on the real-time frame image, a multi-dimensional dirt detection strategy is used to detect dirt in the camera under test, and multi-dimensional dirt detection results are obtained. Based on the multidimensional dirt detection results, the target dirt detection result of the camera under test is determined.

[0006] A second aspect of the present invention provides a camera dirt detection device, comprising: The camera image acquisition module is used to acquire real-time frame images from the camera under test. A multi-dimensional dirt detection module is used to perform dirt detection on the camera under test based on the real-time frame image and adopt a multi-dimensional dirt detection strategy to obtain multi-dimensional dirt detection results. The target dirt detection result determination module is used to determine the target dirt detection result of the camera under test based on the multi-dimensional dirt detection result.

[0007] A third aspect of the present invention provides a self-moving device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the camera dirt detection method according to any one of the first aspects.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the camera dirt detection method according to any one of the first aspects.

[0009] A fifth aspect of the present invention provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the camera dirt detection method described in any of the first aspects.

[0010] Compared with the prior art, the beneficial effect of the embodiments of the present invention is that by using a multi-dimensional dirt detection strategy based on real-time frame images to detect dirt in the camera under test, dirt can be identified from multiple different dimensions. This effectively avoids the situation where environmental occlusion is misjudged as dirt occlusion when a single-dimensional anomaly is caused by environmental occlusion. It can also adapt to various types of dirt, avoid the situation of missed dirt detection, and significantly improve the detection accuracy of dirt in cameras. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the camera dirt detection method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the camera dirt detection device in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the self-moving device in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0013] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0014] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used in this application are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" used in this application includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0015] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0016] Please see Figure 1 The first aspect of this invention provides a method for detecting dirt in a camera, comprising the following steps S1 to S3: Step S1: Acquire real-time frame images from the camera under test; Step S2: Based on the real-time frame image, a multi-dimensional dirt detection strategy is used to detect dirt on the camera under test to obtain multi-dimensional dirt detection results. Step S3: Based on the multidimensional dirt detection results, determine the target dirt detection result of the camera under test.

[0017] Specifically, for self-moving devices, these can be yard robots such as snow-sweeping robots or lawn-mowing robots. Typically, cameras are installed at different locations on these devices to capture images of scenes from different directions, enabling them to handle tasks such as obstacle avoidance and navigation when moving in different directions. Since dirt or obstruction of the camera will directly result in occluded areas in the captured image, this embodiment acquires real-time frame images of the camera under test to promptly detect such obstructions. It is understood that at any given moment, this embodiment can use one, two, or all cameras on the self-moving device as the camera under test; this embodiment does not specifically limit the camera under test.

[0018] Furthermore, considering that some environmental disturbances only exhibit single-dimensional anomalies—for example, strong light only presents brightness anomalies caused by changes in light and shadow, and flat wall obstructions only present edge loss issues—if dirt identification is performed solely based on a single dimension, environmental obstructions are easily misjudged as dirt obstructions on the camera, leading to frequent false detections in extreme environments. Therefore, this embodiment employs a multi-dimensional dirt detection strategy based on real-time frame images to detect dirt on the camera under test. This approach detects dirt from multiple dimensions, avoiding misjudging environmental obstructions as dirt obstructions on the camera and effectively detecting various types of dirt, preventing missed detections. The resulting multi-dimensional dirt detection result indicates whether dirt anomalies exist under multiple dimensions.

[0019] Furthermore, by integrating multi-dimensional dirt detection results, the system can effectively distinguish between dirt-induced occlusion and non-dirty occlusion when camera obstruction is present. Non-dirty occlusion can be caused by environmental factors, such as physical obstacles like rocks or grass obstructing the view of a mobile device entering a narrow space. Dirt-induced occlusion, on the other hand, is caused by dirt on the lens, such as mud or dust. The target dirt detection results for the camera under test can be categorized as no obstruction, dirt-induced occlusion, or non-dirty occlusion.

[0020] The camera dirt detection method provided in this invention uses a multi-dimensional dirt detection strategy based on real-time frame images to detect dirt in the camera under test. This allows for dirt identification from multiple different dimensions, effectively avoiding the misjudgment of environmental occlusion as dirt occlusion when a single-dimensional anomaly is caused by environmental occlusion. It can also adapt to various types of dirt, avoiding missed dirt detection and significantly improving the detection accuracy of camera dirt.

[0021] In one optional embodiment, the step of performing dirt detection on the camera under test based on the real-time frame image using a multi-dimensional dirt detection strategy to obtain multi-dimensional dirt detection results specifically includes: Based on the real-time frame image, at least two of the following are performed on the camera under test: edge information anomaly detection, brightness deviation anomaly detection, and block feature detection, to obtain edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results; Based on the edge information anomaly detection results, the brightness deviation anomaly detection results, and the block feature detection results, the multidimensional dirt detection results are determined.

[0022] Specifically, the multi-dimensional dirt detection strategy in this embodiment includes at least two of edge information anomaly detection, brightness deviation anomaly detection, and block feature detection. In an optional embodiment, after acquiring the real-time frame image of the camera under test, at least two of edge information anomaly detection, brightness deviation anomaly detection, and block feature detection can be performed simultaneously. For example, on devices with limited computing resources, only the combination of "edge information anomaly detection" and "brightness deviation anomaly detection" or only the combination of "edge information anomaly detection" and "block feature detection" can be enabled, while on devices with sufficient computing resources, "edge information anomaly detection," "brightness deviation anomaly detection," and "block feature detection" can be enabled simultaneously.

[0023] It is worth noting that dirt on the camera (such as mud, dust, etc.) can blur the image, resulting in blurred edges and reduced edge information. Therefore, edge information anomaly detection can identify the blurred image caused by dirt occlusion. In bright working environments, if the camera is obstructed by dirt, the captured image will have localized areas of reduced brightness. Brightness deviation anomaly detection can effectively detect the localized decrease in image brightness caused by dirt occlusion. However, for dirt occlusion caused by thin dust or light mud, the resulting changes in edge information and brightness deviation are not obvious and are easily missed. Therefore, this embodiment uses block feature detection to achieve effective identification of local dirt, thus complementing edge information anomaly detection and brightness deviation anomaly detection, and improving the accuracy of dirt detection.

[0024] Furthermore, the edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results provide results on whether there are suspected dirt anomalies from different dimensions. Therefore, by combining the detection results from various dimensions, a multi-dimensional dirt detection result is formed.

[0025] In one optional embodiment, the method specifically performs edge information anomaly detection on the camera under test through the following steps to obtain the edge information anomaly detection result: The real-time frame image and a preset number of previous historical frame images are grayscaled to obtain a first grayscale image corresponding to the real-time frame image and a second grayscale image corresponding to each of the historical frame images. The first grayscale image and each of the second grayscale images are divided into blocks to determine at least two first block regions in the first grayscale image and at least two second block regions in each of the second grayscale images; wherein the number of first block regions is the same as the number of second block regions, and each first block region has a corresponding second block region. Edge detection is performed on each of the first block regions and each of the second block regions to determine the number of first edge pixels in the first block region and the number of second edge pixels in the second block region; Based on the number of first edge pixels and the total number of first pixels in the first block region, and the number of second edge pixels and the total number of second pixels in the second block region, determine the proportion of first edge pixels in each of the first block regions and the current average proportion of edge pixels; Based on the first edge pixel ratio and the average edge pixel ratio, detect abnormal edge information regions in each of the first block regions; The edge information anomaly detection result is determined based on the number of edge information anomaly regions and a preset threshold for the number of edge information anomaly regions.

[0026] Specifically, for real-time frame images acquired from the camera under test, to facilitate edge information anomaly detection, the resolution needs to be uniformly set to a fixed size, for example, a uniform resolution of 640×480. Then, the current real-time frame image and a preset number of previous historical frame images are grayscaled to reduce color interference. The grayscale expression is: Gray = 0.299R + 0.587G + 0.114B, where Gray is the grayscale value, R is the red component value, G is the green component value, and B is the blue component value. Optionally, the first grayscale image corresponding to the real-time frame image and the second grayscale images corresponding to each historical frame image are further subjected to Gaussian filtering. For example, a 5×5 convolution kernel is used to remove high-frequency noise, such as noise caused by sudden changes in illumination or pixel noise. Optionally, in this embodiment, the standard deviation σ of the Gaussian filter is set to 1.5 to avoid excessive blurring of the true edges.

[0027] Furthermore, to improve the precision of edge information detection, this embodiment performs block processing on the first grayscale image and each of the second grayscale images. For example, the first grayscale image and each of the second grayscale images are uniformly divided into 16 equally sized block regions. Taking a grayscale image with a resolution of 640×480 as an example, the size of each block region is 160×120. It is worth noting that during the block processing, the blocks must be divided according to pixel coordinates, ensuring that there is no overlap between the block regions, and then the coordinate range of each block region is recorded. For the first grayscale image and each of the second grayscale images, the number of first block regions in the first grayscale image is the same as the number of second block regions in the second grayscale image, and each first block region has a corresponding second block region, which facilitates the analysis of the edge information changes of the current real-time frame image.

[0028] Furthermore, this embodiment employs the Canny operator to perform edge detection on each first block region and each second block region to identify their respective edge pixels and count the number of edge pixels. Optionally, in this embodiment, the Canny low threshold is set to 20 and the Canny high threshold is set to 80. That is, pixels with a gradient intensity value greater than 80 are considered strong edge pixels and are directly retained, while pixels with a gradient intensity value between 20 and 80 are considered weak edge pixels, and only pixels connected to strong edge pixels are retained. Pixels with a gradient intensity value less than 20 are considered non-edge pixels and are directly suppressed or discarded, judged as noise or useless information. Since dirt such as dust and mud can blur image edges, causing originally obvious strong edges to degenerate into continuous weak edges, this embodiment can effectively capture these damaged edge features.

[0029] Furthermore, the ratio of the number of first edge pixels in the first segmented region to the total number of first pixels can be used to determine the proportion of first edge pixels in the first segmented region; similarly, the ratio of the number of second edge pixels in the second segmented region to the total number of second pixels can be used to determine the proportion of second edge pixels in the second segmented region. In an optional embodiment, this embodiment pre-initializes a first-in-first-out queue with a length equal to the preset number of image analyses. The number of image analyses is the total number of real-time frame images and all historical frame images, such as a total of 50 frames. This queue is used to store the edge pixel proportion of each segmented region in real time. After each frame image is calculated, the edge pixel proportion of each segmented region is written into the queue sequentially. When the queue is detected to be full, the earliest frame data is discarded, keeping the queue length always the same as the number of image analyses.

[0030] Furthermore, the average edge pixel ratio is calculated for each first block region and its corresponding second block regions to obtain the average edge pixel ratio for each first block region. Considering that dirt occlusion will cause edge blurring, resulting in a significant decrease in the edge pixel ratio, this embodiment obtains the current edge pixel ratio threshold based on the average edge pixel ratio and a preset edge information reduction ratio threshold. For example, if the edge information reduction ratio is 0.3, meaning that the edge information can be reduced by a maximum of 30%, then the edge pixel ratio threshold is the average edge pixel ratio × 0.7. When the first edge pixel ratio of a certain first block region is less than the edge pixel ratio threshold, the block region is determined to be suspected of being dirty and is considered an area with abnormal edge information.

[0031] Furthermore, if the number of abnormal edge information regions is greater than or equal to the preset threshold for the number of abnormal edge information regions, the determination of abnormal edge information is triggered. Optionally, the threshold for the number of abnormal edge information regions is set to 50% of the total number of the first block region, thereby avoiding misjudgment of noise in a single block region and avoiding missed judgment due to an excessively high threshold.

[0032] In one optional embodiment, the method specifically performs brightness deviation anomaly detection on the camera under test through the following steps to obtain the brightness deviation anomaly detection result: The real-time frame image is preprocessed to determine multiple grayscale regions corresponding to the real-time frame image; Based on the grayscale value of each pixel in each of the segmented grayscale image regions, the brightness deviation value of each of the segmented grayscale image regions is obtained; Clustering is performed on the brightness deviation values ​​of each of the said segmented grayscale regions to detect abnormal brightness deviation regions in each of the said segmented grayscale regions; The brightness deviation anomaly detection result is determined based on the current brightness deviation threshold, the preset threshold for the number of abnormal brightness deviation regions, the brightness deviation value and the number of the abnormal brightness deviation regions.

[0033] Specifically, this embodiment first preprocesses the real-time frame image, including resolution unification, grayscale conversion, and block segmentation. Since brightness deviation anomaly detection can be performed simultaneously with edge information anomaly detection, the resolution unification and grayscale conversion can directly reuse the grayscale image obtained during edge information anomaly detection, thereby reducing computational load. For block segmentation, to ensure the statistical validity of subsequent clustering, the size of the block grayscale image regions in this embodiment is uniform and the number is greater than or equal to 32. In an optional embodiment, the real-time frame image is divided into 64 block grayscale image regions.

[0034] Furthermore, by statistically analyzing the grayscale values ​​(0~255) of each pixel in each grayscale image region and averaging them, the average brightness value of each grayscale image region can be obtained, denoted as the brightness feature value of the grayscale image region. Optionally, the average brightness value is of floating-point data type, retaining two decimal places. Based on the average brightness value of each grayscale image region, the global brightness mean or neighborhood brightness mean of all grayscale image regions can be calculated, and then the brightness deviation value of each grayscale image region can be calculated using the following expression: ; in, This is the brightness deviation value, used to measure the degree of deviation between the brightness value of a certain grayscale region and the global brightness average or the brightness average of its neighborhood. This represents the total number of pixels in the divided grayscale image region. Let be the brightness value of the i-th pixel in the segmented grayscale image region; It can be the global average brightness or the neighborhood average brightness. The neighborhood average brightness can be obtained by averaging the average brightness values ​​of the neighboring grayscale regions adjacent to the current grayscale region.

[0035] Furthermore, since the brightness deviation values ​​generated by dirty areas such as dark mud and dark sewage are significantly higher than those of normal imaging areas, this embodiment employs K-means (K=2) clustering to split the one-dimensional set of brightness deviation values ​​into two clusters. Two initial values ​​are randomly selected or empirically chosen as low-deviation centers. and high deviation center Understandably, low-bias centers are used to identify low-bias regions, representing areas in the image with relatively "normal" or "consistent" brightness distribution. These areas have small differences in brightness compared to the global brightness mean or the brightness mean of their neighboring areas, conforming to natural light and shadow transitions. In clustering, this class has a smaller centroid (center value), representing clean lens areas or background environments. High-bias centers are used to identify high-bias regions, representing areas in the image with "abnormal" or "abrupt" brightness performance. These blocks exhibit significant isolated fluctuations in brightness relative to the global brightness mean or the brightness mean of their neighboring areas. In clustering, this class has a significantly higher centroid (center value) than the low-bias class, representing areas obscured by mud (dark spots), water stains (bright reflections), or dust. Then, the grayscale region of each block is calculated. arrive and The distance, if If the value is positive, the corresponding grayscale region is classified as "low-biased"; otherwise, it is classified as "high-biased". The average of all values ​​in these two categories is recalculated as the new centroid. and Repeat the above process until the centroid position no longer changes or the maximum number of iterations is reached, for example, the number of iterations ≥ 10 or the change in the class center < 0.1.

[0036] After clustering, the grayscale image regions classified as "high deviation" are considered as brightness deviation anomaly regions. Then, based on a brightness deviation threshold and a preset threshold for the number of brightness deviation anomaly regions, the brightness deviation value and quantity of the current brightness deviation anomaly region are further judged. If the brightness deviation value of the current brightness deviation anomaly region is greater than the brightness deviation threshold, and the quantity is greater than or equal to the threshold for the number of brightness deviation anomaly regions, then a brightness deviation anomaly determination is triggered. It is understood that because the light conditions in different environments vary, the overall brightness of the captured image also varies. Therefore, the brightness deviation threshold in this embodiment is set based on actual environmental light variations and is not a fixed preset value. Optionally, the threshold for the number of brightness deviation anomaly regions in this embodiment is set to 20% of the total number of grayscale image regions to avoid misjudging sporadic brightness deviation anomaly regions.

[0037] In one alternative embodiment, the brightness deviation threshold is determined based on the standard deviation of the brightness deviation values ​​corresponding to all the brightness deviation values ​​of the segmented grayscale regions.

[0038] Specifically, this embodiment introduces global standard deviation verification when detecting abnormal brightness deviation. That is, the brightness deviation threshold is determined based on the standard deviation of the brightness deviation values ​​of all block grayscale regions. Optionally, the brightness deviation threshold is set to twice the standard deviation of the global brightness deviation value, thereby enhancing the adaptability to changes in ambient light. It is worth noting that under uniform light (such as on a cloudy day or a flat lawn), the standard deviation of the global brightness deviation value will be very small. At this time, even if there are only slight dust spots on the lens, they will be detected because they exceed the extremely small standard deviation of the global brightness deviation value. However, under complex light (such as strong tree shadows or dusk), the standard deviation of the global brightness deviation value will be very large. Real-time calculation can automatically raise the judgment threshold to prevent normal natural light and shadow fluctuations from being misjudged as lens dirt.

[0039] In one optional embodiment, the method specifically performs block feature detection on the camera under test through the following steps to obtain the block feature detection result: The real-time frame image is divided into at least two monitoring regions, and the monitoring regions are divided into blocks according to a preset number of blocks to determine multiple monitoring blocks corresponding to each monitoring region. Edge detection is performed on each of the monitoring blocks to determine the edge pixels in each monitoring block; Based on the average gradient value of the edge pixels in the monitoring block and the preset benchmark gradient value, the block feature value of each monitoring block is obtained; Based on the block feature value and the preset block feature threshold, the number of dirty blocks in the monitoring area is detected; Based on the number of dirty blocks and a preset threshold for the number of dirty blocks, the block feature detection result is determined.

[0040] Specifically, to ensure the precision of block feature detection, this embodiment first divides the real-time frame image into at least two monitoring regions. For example, the original real-time frame image or its grayscale image is divided into two equally sized monitoring regions along the horizontal midline. Of course, it can also be divided into four or eight monitoring regions, etc., depending on actual needs. This embodiment does not make a specific limitation here. Then, the number of block divisions is preset according to the monitoring importance, and the monitoring region is further divided into blocks. For example, the monitoring region is further evenly subdivided into 8 monitoring blocks of 2×4. The number of monitoring blocks divided in each monitoring region can be the same or different. This embodiment does not make a specific limitation here.

[0041] Furthermore, in this embodiment, edge detection is performed on each monitoring block using the Canny operator, and the average gradient value of the edge pixels in the monitoring block is determined. Then, the block feature value of each monitoring block is calculated using the following expression: Block feature value = 1 - (average gradient value / baseline gradient value), where the baseline gradient value is the average gradient value of the edge pixels when there is no dirt occlusion, which is obtained through offline calibration.

[0042] Furthermore, if the block feature value is greater than or equal to a preset block feature threshold, the corresponding monitoring block is determined to be a dirty block. Optionally, the block feature threshold is set to 0.003, which is the minimum detectable feature value of dirt determined through offline testing, corresponding to the edge degradation degree of thin dust or slight mud. The number of dirty blocks in each monitoring area is counted. Then, based on the dirty block number threshold, if the number of dirty blocks in the current monitoring area is greater than or equal to the dirty block number threshold, the determination of block feature anomaly is triggered. At this time, the coordinate values ​​of each dirty block need to be recorded. Optionally, in this embodiment, the dirty block number threshold is set to 50% of the total number of monitoring blocks, thereby balancing missed detections and false detections.

[0043] In one optional embodiment, determining the multidimensional dirt detection result based on the edge information anomaly detection result, the brightness deviation anomaly detection result, and the block feature detection result specifically includes: Based on the edge information anomaly detection results, the brightness deviation anomaly detection results, and the block feature detection results, determine the current number of anomaly dimension triggers; Based on the number of abnormal dimension triggers, the dirt confidence score of the camera under test is calculated, and the dirt confidence score is used as the multidimensional dirt detection result: The determination of the target dirt detection result of the camera under test based on the multi-dimensional dirt detection result specifically includes: When the dirt confidence level is greater than or equal to the preset dirt confidence level threshold, the target dirt detection result is determined to be that there is dirt occlusion.

[0044] Specifically, after obtaining the edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results, the number of current anomaly dimension triggers can be determined based on the results that are determined to be abnormal. For example, when there are edge information anomaly detection results and brightness deviation anomaly detection results, the number of anomaly dimension triggers is 2, that is, the triggered anomaly dimensions are edge information anomaly and brightness deviation anomaly; when there is only block feature detection result, the number of anomaly dimension triggers is 1, that is, the triggered anomaly dimension is only block feature anomaly. This embodiment will not elaborate further here.

[0045] Based on the current number of abnormal dimension triggers, the dirt confidence score of the camera under test is calculated. This dirt confidence score is used to characterize the possibility of dirt occlusion, and then the dirt confidence score is used as the multidimensional dirt detection result. Optionally, for detection items determined to be abnormal among edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results, the corresponding abnormal image region and dirt confidence score can be used as the multidimensional dirt detection result simultaneously.

[0046] Since the dirt confidence level directly represents the possibility that the camera under test is obstructed by dirt, the higher the dirt confidence level, the greater the possibility that the camera under test is obstructed by dirt. In this embodiment, a dirt confidence level threshold is set, for example, to 0.8, 0.83, 0.85, 0.78, etc. This embodiment does not make a specific limitation. When the dirt confidence level is greater than or equal to the dirt confidence level threshold, the target dirt detection result is determined to be obstructed by dirt.

[0047] In one optional embodiment, calculating the dirt confidence level of the camera under test based on the number of abnormal dimension triggers specifically includes: The dirt confidence level is determined based on the preset dirt confidence level corresponding to the number of abnormal dimension triggers; Alternatively, when the number of abnormal dimension triggers is at least two, determine the number of overlapping abnormal regions in the real-time frame image or the current percentage of abnormal image frames of the camera under test; wherein, the number of overlapping abnormal regions is the number of image regions in the real-time frame image where at least two of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results are abnormal; the percentage of abnormal image frames is the percentage of images in the real-time frame image and a preset number of historical frame images before it where at least one of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results is abnormal; The dirt confidence level is determined based on the preset dirt confidence level corresponding to the number of overlapping abnormal regions or the preset dirt confidence level corresponding to the proportion of abnormal image frames. Alternatively, when the number of triggered abnormal dimensions is at least two, a preset dirt probability score is obtained corresponding to the at least two triggered abnormal dimension conditions; wherein, the abnormal dimension conditions include edge information abnormality, brightness deviation abnormality, and block feature abnormality; The dirt confidence level is calculated based on the confidence weights corresponding to various preset abnormal dimensions and the preset dirt probability score.

[0048] Specifically, the calculation method for the dirt confidence level of the camera under test includes: The first method: directly setting the confidence level based on the number of abnormal dimension triggers. It's understood that a higher number of abnormal dimension triggers indicates a greater likelihood of dirt occlusion, thus resulting in a higher dirt confidence level. In this embodiment, a corresponding dirt confidence level can be preset for different numbers of abnormal dimension triggers. For example, for a case with 1 abnormal dimension trigger, the dirt confidence level is set to low, such as 30%~40%; for a case with 2 abnormal dimension triggers, the dirt confidence level is set to medium-high, such as 70%~80%; and for a case with 3 abnormal dimension triggers, the dirt confidence level is set to the highest, such as greater than 90%. This embodiment does not impose specific limitations here.

[0049] The second approach: When at least two anomaly dimensions are triggered, the proportion of images that are anomalous (e.g., 50 frames in total, including real-time and historical frames) is detected based on at least one of the following: edge information anomaly detection result, brightness deviation anomaly detection result, and block feature detection result. A higher proportion indicates a persistent anomaly across multiple frames, suggesting a higher likelihood of dirt occlusion. For example, if only 5 frames out of 50 frames show anomalies, it's likely sporadic interference, more likely due to environmental occlusion. However, if more than 40 frames show anomalies, it indicates persistent interference, resulting in a higher dirt confidence level. Different dirt confidence levels can be preset for different proportions of anomalous frames; a higher proportion results in a higher dirt confidence level.

[0050] Furthermore, it can detect the number of image regions in a real-time frame image where at least two of the detection results for edge information anomalies, brightness deviation anomalies, and block feature anomalies are abnormal. For example, if a region in a real-time frame image exhibits both edge information anomalies and brightness deviation anomalies, it indicates that these two anomalous dimensions resonate at the same physical location. This spatial consistency greatly eliminates false alarms from a single dimension, such as eliminating brightness anomalies caused solely by changes in light and shadow, or edge loss caused solely by a flat wall obstruction. Different dirt confidence levels can be pre-set for different numbers of overlapping anomalous regions; the larger the number of overlapping anomalous regions, the higher the corresponding dirt confidence level.

[0051] The third approach: When at least two abnormal dimensions are triggered, this embodiment first identifies at least two of the currently triggered abnormal dimension situations, namely at least two of edge information abnormality, brightness deviation abnormality, and block feature abnormality. Then, based on at least two of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results, a corresponding preset dirt probability score is determined. For example, for edge information abnormality, if the edge information decreases by 40%, the dirt probability score is set to 0.7; for brightness deviation abnormality, its dirt probability score can be set to 0.8, etc. This embodiment does not make specific limitations here. Furthermore, this embodiment also sets corresponding confidence weights for different abnormal dimension situations. For example, the confidence weight corresponding to edge information abnormality is set to 0.6, the confidence weight corresponding to brightness deviation abnormality is set to 0.3, and the confidence weight corresponding to block feature abnormality is set to 0.1, etc. This embodiment does not make specific limitations here. By using the corresponding confidence weights to perform a weighted summation of each preset dirt probability score, the corresponding dirt confidence can be obtained.

[0052] The confidence calculation method described above can effectively address complex dirt detection scenarios, such as tree shadow shaking scenarios and sudden light changes. By considering different dimensions to calculate the dirt confidence, the possibility of dirt occlusion can be quantified, which can effectively avoid false detection problems caused by considering only a single abnormal dimension.

[0053] In one optional embodiment, calculating the dirt confidence level of the camera under test based on the number of abnormal dimension triggers further includes: When the number of abnormal dimension triggers is one, continue to detect whether a preset number of future frame images after the real-time frame image trigger at least one of the abnormal dimension situations; When the abnormal dimension situation is detected to be triggered by the future frame image, the number of abnormal dimension triggers is updated based on the abnormal dimension situation triggered after the real-time frame image, and the dirt confidence is calculated based on the updated number of abnormal dimension triggers.

[0054] Specifically, if the current number of abnormal dimension triggers is only one, i.e., only one of the edge information anomaly detection result, brightness deviation anomaly detection result, and block feature detection result is abnormal, this embodiment will not immediately calculate the dirt confidence score and use it as the basis for the final dirt detection. Instead, it will enter a "pending confirmation" state and continue to detect whether at least one abnormal dimension situation is triggered in a preset number of future frame images after the real-time frame image. For example, it will continue to detect whether the next 10 future frame images trigger abnormal dimension situations. If at least one abnormal dimension situation is detected in a future frame image, the number of abnormal dimension triggers in the pending confirmation state will be updated based on the triggered abnormal dimension situation. For example, assuming that initially only the edge information anomaly detection result is abnormal, i.e., the number of abnormal dimension triggers is 1, and the brightness deviation anomaly is triggered in the next 10 frames, then the number of abnormal dimension triggers will be updated to 2. However, if only the edge information anomaly is triggered in the next 10 frames, the number of abnormal dimension triggers will still be 1. Then, the dirt confidence score will be calculated based on the updated number of abnormal dimension triggers. This detection method can effectively distinguish between transient interference (such as a flying insect, which may only cause a single change in edge or brightness deviation) and persistent dirt (such as mud adhesion, which will cause continuous and synchronous errors in both edge and brightness deviation dimensions), thereby improving the accuracy of dirt detection.

[0055] In an optional embodiment, before determining that the target dirt detection result indicates the presence of dirt obstruction, the method further includes: The dirt confidence level is corrected using the depth map data of the camera under test to obtain a corrected dirt confidence level. Based on the corrected dirt confidence level and the dirt confidence level threshold, the target dirt detection result is determined.

[0056] Specifically, in order to solve the problem that a completely black area in the line-of-sight void is easily misdetected as dirt, this embodiment further uses depth map data to correct the dirt confidence. It is worth noting that the line-of-sight void area can be understood as the area in the image where the algorithm cannot extract effective features due to physical environmental factors, or where the binocular camera cannot complete pixel matching, including: (1) Depth hole area: due to the baseline limitation of the binocular camera or light and shadow occlusion, the area formed at a very close distance (less than the minimum blind zone) or behind the occluded object where the depth value cannot be calculated. (2) Feature missing scene: for example, facing a pure white wall, extremely pure glass, or a completely dark night, in these areas, the image grayscale gradient is almost zero, the edge detection operator cannot capture any edges, and it looks like it is occluded. Therefore, this embodiment uses whether the corresponding area indicated by the depth map data has a depth value to further distinguish between the dirt occlusion area and the line-of-sight void area, thereby further improving the accuracy of dirt detection. If the corrected dirt confidence level is still greater than or equal to the preset dirt confidence level threshold, then the situation is ultimately characterized as dirt occlusion.

[0057] In one optional embodiment, the step of correcting the dirt confidence score using the depth map data of the camera under test to obtain a corrected dirt confidence score specifically includes: The real-time frame image is determined to contain at least two of the following abnormal image regions: the edge information anomaly detection result, the brightness deviation anomaly detection result, and the block feature detection result. Based on the depth map data of the camera under test, the percentage of invalid pixels in the depth map corresponding to each abnormal image region is obtained; When the proportion of invalid pixels in the depth map is greater than or equal to a preset threshold for the proportion of invalid pixels in the depth map, and the position of the abnormal image region remains unchanged within a preset detection time, the dirt confidence level is increased. When the proportion of invalid pixels in the depth map is less than the threshold of the proportion of invalid pixels in the depth map, or when the position of the abnormal image region changes within the preset detection time, the abnormal image region is determined to be an environmental occlusion region, and the dirt confidence level is reduced.

[0058] Specifically, this embodiment identifies suspected dirty image regions beforehand through edge information anomaly detection, brightness deviation anomaly detection, and block feature detection. For cases triggered by only a single anomaly dimension, the confidence level for dirtiness is usually low and it is not definitively classified as dirt occlusion. Therefore, there is no need for further filtering using depth map data. Thus, this embodiment first determines abnormal image regions in the real-time frame image where at least two of the edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results are abnormal, meaning at least two anomaly dimensions have been triggered. Since the dirtiness confidence level is usually high in this case, these abnormal image regions have been preliminarily identified as dirty regions. To avoid misjudgment due to them actually being empty regions at view distance, it is necessary to obtain the percentage of invalid pixels in the depth map of these abnormal image regions based on depth map data.

[0059] It is worth noting that the camera under test in this embodiment is a binocular camera. A depth map can be generated by calculating the parallax between the images captured by the left and right cameras. The following situations will lead to the determination of invalid depth map values: (1) Matching failure: The stereo matching algorithm attempts to find the corresponding pixel in the left image in the right image. If it cannot find it (for example, because the local area of ​​the lens is covered by mud, resulting in local blurring, or facing a white wall without texture), the stereo matching algorithm will mark a specific value (usually 0 or a maximum value) at that position. This is an invalid value of the depth map.

[0060] (2) Low confidence: The binocular matching algorithm usually outputs a confidence score. If the texture of a certain area is too poor, the noise of the matching result is extremely high. The system will remove it according to the preset quality threshold, which is the invalid value of the depth map.

[0061] (3) Distance out of range: If the object is too close to the camera, i.e., enters the blind zone, such as within 0.2 meters, or if the object is too far from the camera, i.e. exceeds the range, it will also be marked as an invalid value of the depth map.

[0062] Therefore, if the abnormal image region corresponds to too many invalid pixels in the depth map (i.e., the proportion of invalid pixels in the depth map is greater than or equal to the preset threshold for the proportion of invalid pixels in the depth map), and the position of the abnormal image region in the image coordinate system remains unchanged within the preset detection time, it indicates that it is attached to the camera under test and belongs to dirt occlusion. This further increases the dirt confidence level, ensuring that the camera under test can be effectively determined to have dirt occlusion. If the abnormal image region has a stable depth value, such as a depth value of 0.5 meters, it indicates that there is a white wall in front of it, and it is determined to be a normal environmental occlusion area, not a dirt occlusion area. In this case, the abnormal image region needs to be filtered and the dirt confidence level reduced. If all abnormal image regions are determined to be normal environmental occlusion areas and not dirt occlusion areas, it is necessary to ensure that the dirt confidence level is reduced to below the preset dirt confidence threshold, thereby ensuring that the camera under test is not mistakenly judged to have dirt occlusion. Alternatively, if the position of the abnormal image region changes in the image coordinate system within a preset detection time, considering that the surrounding environment will also change as the mobile device moves, such as from a white wall to an area with grass, if the original image occlusion is caused by environmental interference, the position of the abnormal image region in the image coordinate system will usually change. Unlike dirt that is always attached to the camera, the abnormal image region can also be identified as an environmental occlusion region in this case, and the current dirt confidence level can be reduced.

[0063] In an optional embodiment, the method further includes: When the target dirt detection result indicates that there is dirt occlusion or no dirt occlusion, one or more occluded areas in the real-time frame image are obtained based on the multi-dimensional dirt detection result and the depth map data of the camera under test. Perform occlusion optimization processing on the occluded area; The occlusion optimization processing operation includes at least one of the following: The occluded area is masked; the invalid point cloud data mapped to the 3D space by the occluded area is filtered; and the occluded area is set as a false obstacle area.

[0064] Specifically, in this embodiment, when the target dirt detection result indicates either dirt occlusion or non-dirt occlusion, an occlusion optimization process needs to be performed on each currently occluded area. It can be understood that this embodiment distinguishes between dirt occlusion and non-dirt occlusion by using a preset dirt confidence threshold. If the dirt confidence is greater than or equal to the threshold, dirt occlusion exists; conversely, if the dirt confidence is greater than 0 but less than the threshold, non-dirt occlusion exists. This occlusion optimization process prevents the risk of the entire self-moving device malfunctioning due to partial occlusion.

[0065] It is worth noting that, for cases with dirt occlusion, the multi-dimensional dirt detection results can identify one or more abnormal image regions determined by edge information anomaly detection, brightness deviation anomaly detection, and block feature detection. Combining this with depth map data allows for the differentiation between dirt-occluded and environmentally occluded regions. Similarly, for cases without dirt occlusion, the multi-dimensional dirt detection results are still needed to identify abnormal image regions. These regions may be triggered by one or more of these anomalies, or by at least two. For image regions that trigger at least two anomalies, depth map data is used to further detect whether they are environmentally occluded regions, thereby identifying one or more currently occluded regions. In this embodiment, the occluded region can be entirely dirt-occluded, entirely environmentally occluded, or may include both. This embodiment does not impose specific limitations.

[0066] In the occlusion optimization process, each occluded region needs to be masked. That is, a dynamic mask is generated at the algorithm layer to block out invalid data in the occluded region, so that it does not participate in the processing or the calculation of processing parameters.

[0067] For situations where there is dirt or obstruction, it is necessary to filter out the invalid point cloud data mapped to the 3D space from the obstructed area. This invalid point cloud data refers to the point cloud data corresponding to the obstructed area. This is because when the camera under test is obstructed by dirt, binocular matching generates a large amount of disordered noise. The 3D coordinate point cloud data converted from this noise will appear as if there is a transparent obstacle at close range directly in front. Once the navigation algorithm receives a signal of a very close obstacle, it will immediately trigger emergency obstacle avoidance logic, such as stopping or rotating in place to find a way out. This would prevent the self-moving device from moving due to obstacles that do not exist. Therefore, filtering this invalid point cloud data can prevent interference from false obstacles, avoid abnormal movement of the self-moving device, and also avoid the backend SLAM (Simultaneous Localization and Mapping) algorithm from consuming a large amount of unnecessary computing resources by processing invalid point cloud data. It is worth noting that this embodiment only filters the invalid point cloud data mapped to the 3D space from the obstructed area, but it is necessary to retain the point cloud data of other undisturbed areas to ensure the normal operation of the self-moving device. In addition, raw point cloud data sent before the invalid point cloud data is filtered needs to be truncated and discarded directly at the end of the data stream.

[0068] Furthermore, in cases where there is dirt or obstruction, this embodiment also sets the dirty or obstructed area within the obstructed area as a false obstacle area, so that the self-moving device no longer attempts to bypass this false obstacle, but instead maintains its original driving trajectory after confirming safety through other sensors.

[0069] In an optional embodiment, the method further includes: Based on the current machine motion state and the orientation of the camera under test, the dirt detection frequency of the camera under test is adjusted; wherein, the machine motion state includes forward state, backward state and turning state.

[0070] Specifically, to balance the dirt detection frequency with system resource consumption, this embodiment adjusts the dirt detection frequency of the camera under test based on the current machine movement state and the orientation of the camera under test. It is understood that the machine movement state directly determines the dominant position of the camera under test in the current environment. For example, when the machine is moving forward, and the orientation of the camera under test is also in the direction corresponding to the forward movement, if the camera under test is obstructed by dirt, it will severely affect the normal movement of the mobile device in the forward movement state. Therefore, it is necessary to increase the dirt detection frequency of the camera under test to detect whether it is obstructed by dirt in a timely manner. Conversely, if the orientation of the camera under test is not in the direction corresponding to the forward movement state, such as if it is a rear camera, then even if the camera under test is obstructed by dirt in the forward movement state, it will not significantly affect the normal movement of the mobile device in the forward movement state. Therefore, it is necessary to reduce its dirt detection frequency to save overall computing resources of the device.

[0071] In one optional embodiment, adjusting the dirt detection frequency of the camera under test based on the current machine motion state and the orientation of the camera under test specifically includes: When the machine's motion state is detected to be consistent with the orientation of the camera under test, the dirt detection frequency of the camera under test is increased to a preset high-frequency detection frequency. When the machine's motion state is detected to be inconsistent with the orientation of the camera under test, the dirt detection frequency of the camera under test is reduced to a preset low-frequency detection frequency; wherein, the high-frequency detection frequency is greater than the low-frequency detection frequency. Alternatively, based on the machine's motion state and the orientation of the camera under test, determine the correlation coefficient of the current motion state of the camera under test; Based on the dirt confidence level and the preset dirt confidence level ambiguity range, the risk correction coefficient of the current camera under test is determined; wherein, the dirt confidence level ambiguity range is 0.4~0.6; Based on the product of the preset basic dirt detection frequency of the camera under test, the motion state correlation coefficient, and the risk correction coefficient, the target detection frequency of the camera under test is determined, and the dirt detection frequency of the camera under test is adjusted to the target detection frequency.

[0072] Specifically, this embodiment adjusts the dirt detection frequency of the camera under test from two dimensions. Firstly, this embodiment can synchronously adjust the dirt detection frequency of the camera under test based on the machine's movement state. It is worth noting that in the forward movement state, the focus is on monitoring the forward-facing cameras with the same orientation. At this time, the forward-facing cameras should be in a high-frequency detection mode because they are the primary obstacle avoidance position, while the left and right cameras with inconsistent orientations and the rear camera should be in a low-frequency polling mode to save computing resources. In the backward movement state, to prevent collisions with obstacles, the primary obstacle avoidance position is the rear camera. Therefore, the focus is on monitoring the rear camera with the same orientation, increasing its dirt detection frequency to a high-frequency detection frequency, while the left and right cameras with inconsistent orientations and the forward camera should be in a low-frequency polling mode to save computing resources. In the turning state, the camera with the same orientation as the turning direction becomes the primary obstacle avoidance position, so its dirt detection frequency is increased to a high-frequency detection frequency, while the other cameras with inconsistent orientations are in a low-frequency polling mode, and their dirt detection frequency is reduced to a low-frequency detection frequency. This embodiment does not specifically limit the high-frequency and low-frequency detection frequencies; they can be set according to the requirements of obstacle avoidance sensitivity.

[0073] On the other hand, this embodiment can also adjust the dirt detection frequency based on the orientation of the camera under test and the current dirt confidence level. First, a basic dirt detection frequency for each camera is preset. For example, the forward-facing camera, as the main navigation camera, should have the highest basic dirt detection frequency among all cameras, while cameras facing other directions only serve as obstacle avoidance aids most of the time, so their basic dirt detection frequencies can be set lower. This embodiment does not impose specific limitations here. Then, the target detection frequency of the camera under test is calculated using the following expression: ; in, Target detection frequency; The frequency of basic dirt detection; The motion state correlation coefficient can be understood as follows: when the machine's motion state and the orientation of the camera under test are consistent, it indicates that the camera under test is correlated with the machine's motion state, and the motion state correlation coefficient should be set to a higher value, such as greater than or equal to 2, to increase the target detection frequency; while when the machine's motion state and the orientation of the camera under test are inconsistent, it indicates that the camera under test is not correlated with the machine's motion state, and the motion state correlation coefficient should be set to a lower value, such as 0.5, to reduce the target detection frequency and save computing resources. The risk correction coefficient is used to determine the ambiguity of the dirt confidence level in this embodiment, which is 0.4 to 0.6. If the current dirt confidence level is within this ambiguity range, it indicates that the current dirt confidence level is in the medium range, and the determination of dirt occlusion is relatively ambiguous. Therefore, it is necessary to increase the risk correction coefficient to enable the camera under test to accumulate dirt detection data from more frames in a shorter time for determination. For example, the initial value of the risk correction coefficient is 1, and when the dirt confidence level is within this ambiguity range, it is increased to 1.5.

[0074] Once the target detection frequency is determined, the dirt detection frequency of the camera under test can be directly adjusted to that target detection frequency.

[0075] In an optional embodiment, the method further includes: When the target dirt detection result indicates that there is dirt occlusion, the texture features, shape features, and color features of the occluded area are extracted; Based on the texture features, shape features, and color features, a dirt classification model is used to classify the dirt in the occluded area and determine the dirt category of the occluded area. The dirt classification model is obtained by training a support vector machine model using sample images with dirt category labels.

[0076] Specifically, when the target dirt detection result indicates dirt occlusion, this embodiment extracts texture, shape, and color features of the occluded area to further clarify the current dirt category. Texture features are obtained by extracting local binary pattern features; for example, mud typically has complex textures, while water stains or simple dust occlusions have smoother textures. Shape features are obtained by calculating the roundness, aspect ratio, and edge roughness of the dirt's connected components. Color features are obtained by extracting the hue and saturation of the dirt area; for example, mud is often dark or brown, while water stains produce bright specular reflections. The extracted texture, shape, and color features are then concatenated into a high-dimensional feature vector.

[0077] Furthermore, the high-dimensional feature vector is input into a pre-trained dirt classification model for dirt classification. Specifically, the dirt classification model maps the high-dimensional feature vector to a high-dimensional space using a radial basis function kernel to find the optimal classification hyperplane, and then directly outputs the specific dirt category label. The dirt classification model is obtained by training a support vector machine model with sample images bearing dirt category labels. For example, the sample images can be labeled mud images, water droplet images, floating and sinking images, weed-covered images, etc., and this embodiment does not impose specific limitations.

[0078] Please see Figure 2 A second aspect of the present invention provides a camera dirt detection device 100, comprising: The camera image acquisition module 11 is used to acquire real-time frame images of the camera under test; The multi-dimensional dirt detection module 12 is used to perform dirt detection on the camera under test based on the real-time frame image and adopt a multi-dimensional dirt detection strategy to obtain multi-dimensional dirt detection results. The target dirt detection result determination module 13 is used to determine the target dirt detection result of the camera under test based on the multi-dimensional dirt detection result.

[0079] In an optional embodiment, the multi-dimensional dirt detection module 12 is used to perform dirt detection on the camera under test based on the real-time frame image using a multi-dimensional dirt detection strategy, and obtain multi-dimensional dirt detection results, specifically including: Based on the real-time frame image, edge information anomaly detection, brightness deviation anomaly detection, and block feature detection are performed on the camera under test to obtain edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results. Based on the edge information anomaly detection results, the brightness deviation anomaly detection results, and the block feature detection results, the multidimensional dirt detection results are determined.

[0080] In an optional embodiment, the multi-dimensional dirt detection module 12 specifically performs edge information anomaly detection on the camera under test through the following steps to obtain the edge information anomaly detection result: The real-time frame image and a preset number of previous historical frame images are grayscaled to obtain a first grayscale image corresponding to the real-time frame image and a second grayscale image corresponding to each of the historical frame images. The first grayscale image and each of the second grayscale images are divided into blocks to determine at least two first block regions in the first grayscale image and at least two second block regions in each of the second grayscale images; wherein the number of first block regions is the same as the number of second block regions, and each first block region has a corresponding second block region. Edge detection is performed on each of the first block regions and each of the second block regions to determine the number of first edge pixels in the first block region and the number of second edge pixels in the second block region; Based on the number of first edge pixels and the total number of first pixels in the first block region, and the number of second edge pixels and the total number of second pixels in the second block region, determine the proportion of first edge pixels in each of the first block regions and the current average proportion of edge pixels; Based on the first edge pixel ratio and the average edge pixel ratio, detect abnormal edge information regions in each of the first block regions; The edge information anomaly detection result is determined based on the number of edge information anomaly regions and a preset threshold for the number of edge information anomaly regions.

[0081] In an optional embodiment, the multi-dimensional dirt detection module 12 specifically performs brightness deviation anomaly detection on the camera under test through the following steps to obtain the brightness deviation anomaly detection result: The real-time frame image is preprocessed to determine multiple grayscale regions corresponding to the real-time frame image; Based on the grayscale value of each pixel in each of the segmented grayscale image regions, the brightness deviation value of each of the segmented grayscale image regions is obtained; Clustering is performed on the brightness deviation values ​​of each of the said segmented grayscale regions to detect abnormal brightness deviation regions in each of the said segmented grayscale regions; Based on the current brightness deviation threshold, the preset threshold for the number of abnormal brightness deviation regions, the brightness deviation value and the number of the abnormal brightness deviation regions, the brightness deviation anomaly detection result is determined; The brightness deviation threshold is determined based on the standard deviation of the brightness deviation values ​​of all the segmented grayscale regions.

[0082] In an optional embodiment, the multidimensional dirt detection module 12 specifically performs block feature detection on the camera under test through the following steps to obtain the block feature detection result: The real-time frame image is divided into at least two monitoring regions, and the monitoring regions are divided into blocks according to a preset number of blocks to determine multiple monitoring blocks corresponding to each monitoring region. Edge detection is performed on each of the monitoring blocks to determine the edge pixels in each monitoring block; Based on the average gradient value of the edge pixels in the monitoring block and the preset benchmark gradient value, the block feature value of each monitoring block is obtained; Based on the block feature value and the preset block feature threshold, the number of dirty blocks in the monitoring area is detected; Based on the number of dirty blocks and a preset threshold for the number of dirty blocks, the block feature detection result is determined.

[0083] In an optional embodiment, the multidimensional dirt detection module 12 is used to determine the multidimensional dirt detection result based on the edge information anomaly detection result, the brightness deviation anomaly detection result, and the block feature detection result, specifically including: Based on the edge information anomaly detection results, the brightness deviation anomaly detection results, and the block feature detection results, determine the current number of anomaly dimension triggers; Based on the number of abnormal dimension triggers, the dirt confidence score of the camera under test is calculated, and the dirt confidence score is used as the multidimensional dirt detection result; The target dirt detection result determination module 13 is used to determine the target dirt detection result of the camera under test based on the multi-dimensional dirt detection result, specifically including: When the dirt confidence level is greater than or equal to the preset dirt confidence level threshold, the target dirt detection result is determined to be that there is dirt occlusion.

[0084] In an optional embodiment, the multidimensional dirt detection module 12 is used to calculate the dirt confidence level of the camera under test based on the number of abnormal dimension triggers, specifically including: The dirt confidence level is determined based on the preset dirt confidence level corresponding to the number of abnormal dimension triggers; Alternatively, when the number of abnormal dimension triggers is at least two, determine the number of overlapping abnormal regions in the real-time frame image or the current percentage of abnormal image frames of the camera under test; wherein, the number of overlapping abnormal regions is the number of image regions in the real-time frame image where at least two of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results are abnormal; the percentage of abnormal image frames is the percentage of images in the real-time frame image and a preset number of historical frame images before it where at least one of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results is abnormal; The dirt confidence level is determined based on the preset dirt confidence level corresponding to the number of overlapping abnormal regions or the preset dirt confidence level corresponding to the proportion of abnormal image frames. Alternatively, when the number of triggered abnormal dimensions is at least two, a preset dirt probability score is obtained corresponding to the at least two triggered abnormal dimension conditions; wherein, the abnormal dimension conditions include edge information abnormality, brightness deviation abnormality, and block feature abnormality; The dirt confidence level is calculated based on the confidence weights corresponding to various preset abnormal dimensions and the preset dirt probability score.

[0085] In an optional embodiment, the multidimensional dirt detection module 12 is used to calculate the dirt confidence level of the camera under test based on the number of abnormal dimension triggers, and specifically further includes: When the number of abnormal dimension triggers is one, continue to detect whether a preset number of future frame images after the real-time frame image trigger at least one of the abnormal dimension situations; When the abnormal dimension situation is detected to be triggered by the future frame image, the number of abnormal dimension triggers is updated based on the abnormal dimension situation triggered after the real-time frame image, and the dirt confidence is calculated based on the updated number of abnormal dimension triggers.

[0086] In an optional embodiment, before determining that the target dirt detection result indicates the presence of dirt obstruction, the target dirt detection result determination module 13 is further configured to: The dirt confidence level is corrected using the depth map data of the camera under test to obtain a corrected dirt confidence level. Based on the corrected dirt confidence level and the dirt confidence level threshold, the target dirt detection result is determined.

[0087] In an optional embodiment, the target dirt detection result determination module 13 is used to correct the dirt confidence level using the depth map data of the camera under test, to obtain a corrected dirt confidence level, specifically including: The real-time frame image is determined to contain at least two of the following abnormal image regions: the edge information anomaly detection result, the brightness deviation anomaly detection result, and the block feature detection result. Based on the depth map data of the camera under test, the percentage of invalid pixels in the depth map corresponding to each abnormal image region is obtained; When the proportion of invalid pixels in the depth map is greater than or equal to a preset threshold for the proportion of invalid pixels in the depth map, and the position of the abnormal image region remains unchanged within a preset detection time, the dirt confidence level is increased. When the proportion of invalid pixels in the depth map is less than the threshold of the proportion of invalid pixels in the depth map, or when the position of the abnormal image region changes within the preset detection time, the abnormal image region is determined to be an environmental occlusion region, and the dirt confidence level is reduced.

[0088] In an optional embodiment, the device is further configured to: When the target dirt detection result indicates that there is dirt occlusion or no dirt occlusion, one or more occluded areas in the real-time frame image are obtained based on the multi-dimensional dirt detection result and the depth map data of the camera under test. Perform occlusion optimization processing on the occluded area; The occlusion optimization processing operation includes at least one of the following: The occluded area is masked; the invalid point cloud data mapped to the 3D space by the occluded area is filtered; and the occluded area is set as a false obstacle area.

[0089] In an optional embodiment, the device is further configured to: Based on the current machine motion state and the orientation of the camera under test, the dirt detection frequency of the camera under test is adjusted; wherein, the machine motion state includes forward state, backward state and turning state.

[0090] In one optional embodiment, the device is used to adjust the dirt detection frequency of the camera under test based on the current machine motion state and the orientation of the camera under test, specifically including: When the machine's motion state is detected to be consistent with the orientation of the camera under test, the dirt detection frequency of the camera under test is increased to a preset high-frequency detection frequency. When the machine's motion state is detected to be inconsistent with the orientation of the camera under test, the dirt detection frequency of the camera under test is reduced to a preset low-frequency detection frequency; wherein, the high-frequency detection frequency is greater than the low-frequency detection frequency. Alternatively, based on the machine's motion state and the orientation of the camera under test, determine the correlation coefficient of the current motion state of the camera under test; Based on the dirt confidence level and the preset dirt confidence level ambiguity range, the risk correction coefficient of the current camera under test is determined; wherein, the dirt confidence level ambiguity range is 0.4~0.6; Based on the product of the preset basic dirt detection frequency of the camera under test, the motion state correlation coefficient, and the risk correction coefficient, the target detection frequency of the camera under test is determined, and the dirt detection frequency of the camera under test is adjusted to the target detection frequency.

[0091] In an optional embodiment, the device is further configured to: When the target dirt detection result indicates that there is dirt occlusion, the texture features, shape features, and color features of the occluded area are extracted; Based on the texture features, shape features, and color features, a dirt classification model is used to classify the dirt in the occluded area and determine the dirt category of the occluded area. The dirt classification model is obtained by training a support vector machine model using sample images with dirt category labels.

[0092] The camera dirt detection device 100 provided in this embodiment of the invention detects dirt in the camera under test by adopting a multi-dimensional dirt detection strategy based on real-time frame images. This enables dirt discrimination from multiple different dimensions, effectively avoiding the situation where environmental occlusion is misjudged as dirt occlusion when a single-dimensional anomaly is caused by environmental occlusion. It can also adapt to various types of dirt, avoiding the situation of missed dirt detection, and significantly improving the detection accuracy of camera dirt.

[0093] Please see Figure 3 The third aspect of the present invention provides a self-moving device 200, including a memory 22, a processor 21, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the camera dirt detection method described in any embodiment of the first aspect.

[0094] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the self-moving device 200.

[0095] The self-moving device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the self-moving device 200 and does not constitute a limitation on the self-moving device 200. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the self-moving device 200 may also include input / output devices, network access devices, buses, etc.

[0096] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 21 can be any conventional processor 21. The processor 21 is the control center of the self-moving device 200, connecting all parts of the self-moving device 200 via various interfaces and lines.

[0097] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the self-moving device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0098] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the camera dirt detection method described in any embodiment of the first aspect.

[0099] A fifth aspect of the present invention provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the camera dirt detection method described in any embodiment of the first aspect.

[0100] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0101] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting dirt in a camera, characterized in that, include: Acquire real-time frame images from the camera under test; Based on the real-time frame image, a multi-dimensional dirt detection strategy is used to detect dirt in the camera under test, and multi-dimensional dirt detection results are obtained. Based on the multidimensional dirt detection results, the target dirt detection result of the camera under test is determined.

2. The camera dirt detection method as described in claim 1, characterized in that, The step of using a multi-dimensional dirt detection strategy to detect dirt on the camera under test based on the real-time frame image to obtain multi-dimensional dirt detection results specifically includes: Based on the real-time frame image, at least two of the following are performed on the camera under test: edge information anomaly detection, brightness deviation anomaly detection, and block feature detection, to obtain edge information anomaly detection results, brightness deviation anomaly detection results, and block feature detection results; Based on the edge information anomaly detection results, the brightness deviation anomaly detection results, and the block feature detection results, the multidimensional dirt detection results are determined.

3. The camera dirt detection method as described in claim 2, characterized in that, The method specifically performs edge information anomaly detection on the camera under test through the following steps to obtain the edge information anomaly detection result: The real-time frame image and a preset number of previous historical frame images are grayscaled to obtain a first grayscale image corresponding to the real-time frame image and a second grayscale image corresponding to each of the historical frame images. The first grayscale image and each of the second grayscale images are divided into blocks to determine at least two first block regions in the first grayscale image and at least two second block regions in each of the second grayscale images; wherein the number of first block regions is the same as the number of second block regions, and each first block region has a corresponding second block region. Edge detection is performed on each of the first block regions and each of the second block regions to determine the number of first edge pixels in the first block region and the number of second edge pixels in the second block region; Based on the number of first edge pixels and the total number of first pixels in the first block region, and the number of second edge pixels and the total number of second pixels in the second block region, determine the proportion of first edge pixels in each of the first block regions and the current average proportion of edge pixels; Based on the first edge pixel ratio and the average edge pixel ratio, detect abnormal edge information regions in each of the first block regions; The edge information anomaly detection result is determined based on the number of edge information anomaly regions and a preset threshold for the number of edge information anomaly regions.

4. The camera dirt detection method as described in claim 2, characterized in that, The method specifically performs brightness deviation anomaly detection on the camera under test through the following steps to obtain the brightness deviation anomaly detection result: The real-time frame image is preprocessed to determine multiple grayscale regions corresponding to the real-time frame image; Based on the grayscale value of each pixel in each of the segmented grayscale image regions, the brightness deviation value of each of the segmented grayscale image regions is obtained; Clustering is performed on the brightness deviation values ​​of each of the said segmented grayscale regions to detect abnormal brightness deviation regions in each of the said segmented grayscale regions; Based on the current brightness deviation threshold, the preset threshold for the number of abnormal brightness deviation regions, the brightness deviation value and the number of the abnormal brightness deviation regions, the brightness deviation anomaly detection result is determined; The brightness deviation threshold is determined based on the standard deviation of the brightness deviation values ​​of all the segmented grayscale regions.

5. The camera dirt detection method as described in claim 2, characterized in that, The method specifically performs block feature detection on the camera under test through the following steps to obtain the block feature detection result: The real-time frame image is divided into at least two monitoring regions, and the monitoring regions are divided into blocks according to a preset number of blocks to determine multiple monitoring blocks corresponding to each monitoring region. Edge detection is performed on each of the monitoring blocks to determine the edge pixels in each monitoring block; Based on the average gradient value of the edge pixels in the monitoring block and the preset benchmark gradient value, the block feature value of each monitoring block is obtained; Based on the block feature value and the preset block feature threshold, the number of dirty blocks in the monitoring area is detected; Based on the number of dirty blocks and a preset threshold for the number of dirty blocks, the block feature detection result is determined.

6. The camera dirt detection method as described in claim 2, characterized in that, The determination of the multidimensional dirt detection result based on the edge information anomaly detection result, the brightness deviation anomaly detection result, and the block feature detection result specifically includes: Based on the edge information anomaly detection results, the brightness deviation anomaly detection results, and the block feature detection results, determine the current number of anomaly dimension triggers; Based on the number of abnormal dimension triggers, the dirt confidence score of the camera under test is calculated, and the dirt confidence score is used as the multidimensional dirt detection result; The determination of the target dirt detection result of the camera under test based on the multi-dimensional dirt detection result specifically includes: When the dirt confidence level is greater than or equal to the preset dirt confidence level threshold, the target dirt detection result is determined to be that there is dirt occlusion.

7. The camera dirt detection method as described in claim 6, characterized in that, The calculation of the dirt confidence level of the camera under test based on the number of abnormal dimension triggers specifically includes: The dirt confidence level is determined based on the preset dirt confidence level corresponding to the number of abnormal dimension triggers; Alternatively, when the number of abnormal dimension triggers is at least two, determine the number of overlapping abnormal regions in the real-time frame image or the current percentage of abnormal image frames of the camera under test; wherein, the number of overlapping abnormal regions is the number of image regions in the real-time frame image where at least two of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results are abnormal; the percentage of abnormal image frames is the percentage of images in the real-time frame image and a preset number of historical frame images before it where at least one of the edge information abnormality detection results, brightness deviation abnormality detection results, and block feature detection results is abnormal; The dirt confidence level is determined based on the preset dirt confidence level corresponding to the number of overlapping abnormal regions or the preset dirt confidence level corresponding to the proportion of abnormal image frames. Alternatively, when the number of triggered abnormal dimensions is at least two, a preset dirt probability score is obtained corresponding to the at least two triggered abnormal dimension conditions; wherein, the abnormal dimension conditions include edge information abnormality, brightness deviation abnormality, and block feature abnormality; The dirt confidence level is calculated based on the confidence weights corresponding to various preset abnormal dimensions and the preset dirt probability score.

8. The method for detecting dirt in a camera as described in claim 7, characterized in that, The calculation of the dirt confidence level of the camera under test based on the number of abnormal dimension triggers specifically includes: When the number of abnormal dimension triggers is one, continue to detect whether a preset number of future frame images after the real-time frame image trigger at least one of the abnormal dimension situations; When the abnormal dimension situation is detected to be triggered by the future frame image, the number of abnormal dimension triggers is updated based on the abnormal dimension situation triggered after the real-time frame image, and the dirt confidence is calculated based on the updated number of abnormal dimension triggers.

9. The camera dirt detection method as described in claim 6, characterized in that, Before determining that the target dirt detection result indicates the presence of dirt obstruction, the method further includes: The real-time frame image is determined to contain at least two of the following abnormal image regions: the edge information anomaly detection result, the brightness deviation anomaly detection result, and the block feature detection result. Based on the depth map data of the camera under test, the percentage of invalid pixels in the depth map corresponding to each abnormal image region is obtained; When the proportion of invalid pixels in the depth map is greater than or equal to a preset threshold for the proportion of invalid pixels in the depth map, and the position of the abnormal image region remains unchanged within a preset detection time, the dirt confidence level is increased. When the proportion of invalid pixels in the depth map is less than the threshold of the proportion of invalid pixels in the depth map, or when the position of the abnormal image region changes within the preset detection time, the abnormal image region is determined to be an environmental occlusion region, and the dirt confidence level is reduced.

10. The method for detecting dirt in a camera as described in claim 1, characterized in that, The method further includes: When the target dirt detection result indicates that there is dirt occlusion or no dirt occlusion, one or more occluded areas in the real-time frame image are obtained based on the multi-dimensional dirt detection result and the depth map data of the camera under test. Perform occlusion optimization processing on the occluded area; The occlusion optimization processing operation includes at least one of the following: The occluded area is masked; the invalid point cloud data mapped to the 3D space by the occluded area is filtered; and the occluded area is set as a false obstacle area.

11. The camera dirt detection method as described in claim 6, characterized in that, The method further includes: Based on the current machine motion state and the orientation of the camera under test, the dirt detection frequency of the camera under test is adjusted; wherein, the machine motion state includes forward state, backward state and turning state.

12. The camera dirt detection method as described in claim 11, characterized in that, The adjustment of the dirt detection frequency of the camera under test based on the current machine motion state and the orientation of the camera under test specifically includes: When the machine's motion state is detected to be consistent with the orientation of the camera under test, the dirt detection frequency of the camera under test is increased to a preset high-frequency detection frequency. When the machine's motion state is detected to be inconsistent with the orientation of the camera under test, the dirt detection frequency of the camera under test is reduced to a preset low-frequency detection frequency; wherein, the high-frequency detection frequency is greater than the low-frequency detection frequency. Alternatively, based on the machine's motion state and the orientation of the camera under test, determine the correlation coefficient of the current motion state of the camera under test; Based on the dirt confidence level and the preset dirt confidence level ambiguity range, the risk correction coefficient of the current camera under test is determined; wherein, the dirt confidence level ambiguity range is 0.4~0.6; Based on the product of the preset basic dirt detection frequency of the camera under test, the motion state correlation coefficient, and the risk correction coefficient, the target detection frequency of the camera under test is determined, and the dirt detection frequency of the camera under test is adjusted to the target detection frequency.

13. The camera dirt detection method as described in claim 10, characterized in that, The method further includes: When the target dirt detection result indicates that there is dirt occlusion, the texture features, shape features, and color features of the occluded area are extracted; Based on the texture features, shape features, and color features, a dirt classification model is used to classify the dirt in the occluded area and determine the dirt category of the occluded area. The dirt classification model is obtained by training a support vector machine model using sample images with dirt category labels.

14. A camera dirt detection device, characterized in that, include: The camera image acquisition module is used to acquire real-time frame images from the camera under test. A multi-dimensional dirt detection module is used to perform dirt detection on the camera under test based on the real-time frame image and adopt a multi-dimensional dirt detection strategy to obtain multi-dimensional dirt detection results. The target dirt detection result determination module is used to determine the target dirt detection result of the camera under test based on the multi-dimensional dirt detection result.

15. A self-moving device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the camera dirt detection method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the camera dirt detection method according to any one of claims 1 to 13.

17. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the steps of the camera dirt detection method according to any one of claims 1 to 13.

Images