Dirt detection method of camera, self-operation equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to a camera smudginess detection method, self-operation equipment and a storage medium, and the method comprises the steps: obtaining a binocular image, collected by a binocular camera, of an operation area; inputting the binocular image into an image anomaly detection model for identifying whether an abnormal region influencing the image quality exists in the image, and obtaining a smudginess category of the abnormal region; according to the method, the depth information of the abnormal area is acquired, whether actual smudginess of the smudginess category exists in the abnormal area or not is judged according to the depth information, a smudginess verification result is obtained, and compared with the prior art, automatic detection of the smudginess of the camera is achieved, and the accuracy of automatic detection can be ensured.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting dirt in a camera, a self-operating device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of technology, the use of autonomous equipment for automated operations can greatly improve work efficiency. For example, lawnmower robots, as a type of autonomous equipment, integrate technologies such as motion control, multi-sensor fusion, and path planning, and are widely used in the maintenance of home lawns and the mowing of large lawns. Among these, the camera, as one of the important sensors for environmental perception in autonomous equipment, is usually directly exposed to the environment, making it susceptible to dirt and interference. Summary of the Invention

[0003] This invention provides a method for detecting dirt in a camera, an automated device, and a computer-readable storage medium, which enables automated detection of dirt in a camera and ensures the accuracy of the automated detection.

[0004] In a first aspect, the present invention provides a method for detecting dirt in a camera, comprising: Acquire binocular images of the work area captured by a binocular camera; The binocular images are input into an image anomaly detection model to identify whether there are abnormal regions in the images that affect image quality and to obtain the type of dirt in the abnormal regions. Obtain the depth information of the abnormal area, determine whether there is actual dirt of the abnormal area based on the depth information, and obtain the dirt verification result.

[0005] Optionally, in one embodiment, the image anomaly detection model includes a classification sub-model and a detection sub-model. The binocular image is input into the image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and to obtain the dirt category of the abnormal regions, including: The binocular images are input into the classification sub-model to identify whether there are abnormal regions in the binocular images that are naturally dirty or not naturally dirty. If an abnormal region with natural dirt is identified in the binocular image, the binocular image is input into the detection sub-model to obtain the natural dirt category of the abnormal region.

[0006] Optionally, in one embodiment, determining whether there is actual dirt of a certain type in the abnormal area based on depth information to obtain a dirt verification result includes: The depth porosity of abnormal areas is calculated based on depth information. If the depth porosity reaches the first porosity threshold, a dirt verification result indicating the presence of natural dirt categories in the abnormal area is obtained.

[0007] Optionally, in one embodiment, if the depth porosity reaches a first porosity threshold, a dirt verification result indicating the presence of natural dirt categories in the abnormal area is obtained, including: If the depth porosity reaches the first porosity threshold, then the depth change information of the edge of the abnormal area is obtained based on the depth information. If the depth change information matches the preset depth change information corresponding to the natural dirt category, then the dirt verification result indicating the presence of the natural dirt category in the abnormal area is obtained.

[0008] Optionally, in one embodiment, after calculating the depth porosity of the abnormal region based on the depth information, the method further includes: If the porosity is less than the second porosity threshold, a dirt verification result is obtained indicating that there is actual non-natural dirt in the abnormal area.

[0009] Optionally, in one embodiment, after determining whether there is actual dirt of a certain type in the abnormal area based on the depth information and obtaining the dirt verification result, the method further includes: Obtain the dirt verification result set of all binocular images corresponding to the work area within the first time window of the binocular image; If the percentage of actual dirt in the dirt verification results indicating abnormal areas that are of the natural dirt category reaches the first percentage threshold, then a dirt cleaning operation will be performed.

[0010] Optionally, in one embodiment, performing a dirt cleaning operation includes: Identify the target dirt cleaning operation corresponding to the natural dirt category and execute the target dirt cleaning operation.

[0011] Optionally, in one embodiment, the dirt detection method for a camera provided in this aspect further includes: During the cleaning process, abnormal areas are marked as non-obstacle avoidance detection areas.

[0012] Optionally, in one embodiment, after inputting the binocular images into a classification sub-model to identify abnormal regions in the binocular images that contain natural or unnatural dirt, the method further includes: If an abnormal area with unnatural dirt is identified in the binocular image, the recognition result set of all binocular images corresponding to the work area within the second time window of the binocular image is obtained. If the proportion of non-natural dirt in the identification result set reaches the second proportion threshold, then the non-natural dirt is determined to be a man-made obstruction.

[0013] Optionally, in one embodiment, before inputting the binocular images into the image anomaly detection model, the method further includes: Obtain the brightness histogram of the binocular image; Statistical analysis of brightness distribution information from brightness histograms, and determination of the target scene type of the work area based on brightness distribution information; Preprocess the stereo images according to the preprocessing strategy corresponding to the target scene type.

[0014] Secondly, the self-operating device provided by the present invention includes a binocular camera, a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the dirt detection method of the camera provided by the present invention.

[0015] Thirdly, the computer-readable storage medium provided by the present invention stores a computer program, which, when executed by a processor, implements the dirt detection method for a camera provided by the present invention.

[0016] The dirt detection scheme for cameras provided by this invention acquires binocular images of the work area captured by a binocular camera; inputs the binocular images into an image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and obtains the dirt category of the abnormal regions; obtains the depth information of the abnormal regions, and determines whether there is actual dirt of the specified category in the abnormal regions based on the depth information, thus obtaining a dirt verification result. In this way, by classifying the dirt categories of abnormal regions, manual visual inspection is eliminated, achieving automated dirt detection for cameras. Furthermore, the detection and classification results of the image anomaly detection model are verified using depth information, ensuring the accuracy of automated detection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a dirt detection method for a camera provided in an embodiment of the present invention; Figure 2 yes Figure 1 Detailed process diagram of S120; Figure 3 This is a schematic diagram illustrating the merging of the left and right eye images by channel in an embodiment of the present invention; Figure 4 yes Figure 1 Detailed process diagram of S130; Figure 5 yes Figure 4Detailed process diagram of S1302; Figure 6 This is another schematic flowchart of the dirt detection method for a camera provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the process of dirt detection by the lawnmower robot in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of the self-operating device provided in an embodiment of the present invention; Figure 9 This is an example diagram of the product form of the self-operating equipment provided in the embodiments of the present invention. Detailed Implementation

[0019] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] This invention provides a method, device, self-operating equipment, and storage medium for detecting dirt in a camera. The dirt detection method can be executed by the dirt detection device or by a self-operating equipment integrating the dirt detection device. The method involves: acquiring a binocular image of the working area captured by a binocular camera; inputting the binocular image into an image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and obtaining the dirt category of the abnormal region; acquiring the depth information of the abnormal region, and determining whether there is actual dirt of the specified category in the abnormal region based on the depth information, thereby obtaining a dirt verification result.

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a dirt detection method for a camera provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the procedure for this dirt detection method can be as follows: S110. Acquire binocular images of the work area captured by the binocular camera.

[0028] Autonomous operating equipment refers to intelligent devices that can autonomously complete specific tasks through sensors, algorithms, and control systems without continuous human operation. Its core feature is the closed-loop capability of "environmental perception-decision control-task execution", such as lawn mowing robots, agricultural unmanned vehicles, and sweeping robots.

[0029] A binocular camera is an imaging system that simulates human vision. It consists of two horizontally arranged cameras and is a key sensor for various intelligent devices, including autonomous devices.

[0030] Taking a lawnmower robot as an example of an autonomous operating device, a binocular camera can be exposed on the robot's body, allowing the robot to capture binocular images of the work area. These images can then be used for operation control and movement control. For instance, the lawnmower robot can use the binocular images captured by the camera to construct a depth map of the lawn, and then use this depth map to construct a 3D point cloud of the lawn for mowing control. For example, the 3D point cloud can be used to identify the height of the grass blades to determine whether mowing is needed; or, for example, the 3D point cloud can be used to identify the terrain undulations of the lawn to control the walking speed according to the slope, and so on.

[0031] It is understandable that if dirt is present on the binocular camera and is not detected and removed in time, it will affect the quality of the binocular images captured by the camera, and thus affect the normal operation of the self-operating equipment. Based on this, the present invention provides a dirt detection method suitable for binocular cameras.

[0032] The self-operating equipment first acquires binocular images of the operating area captured by the binocular camera. The binocular images can include left and right eye images.

[0033] S120. Input the binocular image into the image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality and obtain the dirt category of the abnormal region.

[0034] It should be noted that, in this embodiment of the invention, the image anomaly detection model consists of a classification sub-model and a detection sub-model. The classification sub-model is configured to take an image that may include anomaly regions that affect image quality due to dirt as input, and to output a predicted anomaly type indicating the anomaly region in the image. The anomaly type includes natural dirt and unnatural dirt. Natural dirt refers to anomaly objects that are attached to the binocular camera due to natural reasons and interfere with the binocular camera's shooting, such as water droplets, mud spots, rain fog, etc. Unnatural dirt refers to anomaly objects that are not attached to the binocular camera but interfere with the binocular camera's shooting, such as man-made protective covers, stickers, or the frame of the operating equipment itself that is damaged.

[0035] The classification sub-model can be constructed using lightweight convolutional neural networks such as SqueezeNet, MobileNet, and ShuffleNet, and trained in a supervised manner. The specific implementation can be determined by those skilled in the art based on actual needs, and no specific limitations are imposed here. In this embodiment of the invention, by inputting binocular images into the classification sub-model, the classification results of the sub-model include three categories: normal, naturally soiled, and unnaturally soiled. The classification sub-model is trained using supervised learning, and the training data includes a large number of fused image samples with labeled categories, outputting the predicted category.

[0036] The detection sub-model is activated after the classification sub-model determines that the image is "naturally dirty" or "unnaturally dirty". It is used to accurately detect the location and type of abnormal regions in the image. It can be obtained by transferring the pre-trained target detection model to labeled sample binocular images.

[0037] For example, it can be implemented as follows: The object detection model based on YOLOv8 is used as the base model for training the detection sub-model. Its model structure includes the following components: Backbone network: A backbone network is a network used to extract image features. Its main function is to transform the original input image into a multi-layer feature map for use in subsequent detection tasks.

[0038] Neck network: The neck network is used to combine feature maps from different levels to generate feature maps with different scale features, thereby improving detection accuracy.

[0039] The detection head consists of three different output layers, each responsible for detecting targets of different scales (i.e., abnormal regions).

[0040] During the training phase, sample binocular images covering different categories of anomalous objects (such as water droplets, mud spots / stains, leaf occlusion, steam, light spots / reflections, etc.) are collected. For each set of sample binocular images, they are aligned and merged by channel to obtain a sample channel-merged image. Each sample channel-merged image is labeled to obtain training labels including ground truth bounding boxes indicating anomalous regions and anomalous object categories. Each sample channel-merged image and its corresponding training label are used as a set of samples to input into the object detection model for training until a preset stopping condition is met, resulting in a detection sub-model. The configuration of the preset stopping condition is not specifically limited here. For example, the preset stopping condition can be configured as: the object detection model converges, or the number of parameter update epochs for the object detection model reaches a preset number of epochs, such as 150 epochs.

[0041] The detection sub-model outputs the location information (boundary box coordinates) of each abnormal region and the abnormal category label.

[0042] Accordingly, please refer to Figure 2 The binocular images are input into an image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and to obtain the type of dirt in the abnormal regions, which may include: S1201. Input the binocular image into the classification sub-model to identify whether there are abnormal areas of natural or unnatural dirt in the binocular image. S1202. If an abnormal region with natural dirt is detected in the binocular image, the binocular image is input into the detection sub-model to obtain the natural dirt category of the abnormal region.

[0043] After acquiring binocular images of the work area from the binocular camera, the self-operating equipment first aligns the left and right images, then merges the aligned left and right images by channel to obtain a channel-merged image. The channel-merged image is then input into a classification sub-model, which identifies whether there are abnormal areas in the binocular images that affect image quality due to natural or unnatural dirt.

[0044] If an abnormal region of natural dirt is identified in the binocular image, the channel-merged image is then input into the detection sub-model. The detection sub-model further detects the location of the abnormal region and the specific type of natural dirt (such as water droplets, mud spots / stains, etc.). For example, please refer to... Figure 3 If both the left and right images are RGB three-channel images, then the channel-merged image obtained by merging the left and right images by channels will include six channels.

[0045] The above process first uses a classification sub-model for preliminary classification. When the classification sub-model identifies abnormal areas of natural dirt in the binocular image, the detection sub-model further confirms the structural location of the possible natural dirt, thus verifying the classification results and forming a decision loop.

[0046] It should be noted that in other embodiments, if abnormal areas with unnatural dirt are detected in the binocular image, the binocular image can be input into the detection sub-model for secondary detection, or the binocular image can be left uninputted into the detection sub-model for secondary detection.

[0047] For example, for a binocular image, if the classification sub-model classifies the image as "natural dirt" and the detection sub-model detects the image as "no natural dirt", then the classification sub-model is deemed to have misclassified the image, and its classification result is corrected to "normal" without further processing. For a binocular image, if the classification sub-model classifies the image as "natural dirt" and the detection sub-model detects the image as "branches and leaves" indicating an abnormal region, then the image is determined to be foreground occlusion, which is a misclassification by the classification sub-model. The classification result is then corrected to "normal" and no further processing is performed. For a binocular image, if the classification result obtained by the classification sub-model is "unnatural dirt" and the detection result obtained by the detection sub-model indicates that the object category of the abnormal region is "spot / reflection", then the classification sub-model is judged to be correct, but the abnormal region is actually caused by spot / reflection, and no further processing is performed. For a binocular image, assuming the classification result obtained by the classification sub-model is "natural dirt" and the detection result obtained by the detection sub-model indicates that the object category of the abnormal region is "water droplet", then the classification sub-model is determined to be correct, and the object category "water droplet" matches natural dirt. Accordingly, the natural dirt category of the abnormal region is determined to be "water droplet", and subsequent processing is carried out.

[0048] S130. Obtain the depth information of the abnormal area, determine whether there is actual dirt of the dirt type in the abnormal area based on the depth information, and obtain the dirt verification result.

[0049] Among them, depth information is used to describe the physical distance from the scene point corresponding to the pixel to the binocular camera.

[0050] In this embodiment of the invention, the self-operating device also acquires the depth information of the abnormal area, and according to the configured dirt verification strategy, determines whether there is actual dirt of the type of dirt detected by the image anomaly detection model in the abnormal area based on the depth information of the abnormal area, and obtains the dirt verification result accordingly.

[0051] For example, the self-operating device can reuse the binocular images captured by the binocular camera to construct a depth map corresponding to the working area, or it can directly obtain a depth map corresponding to the working area through a depth camera; after obtaining the depth map corresponding to the working area, the self-operating device further obtains the depth information of the abnormal area from the depth map.

[0052] The following example illustrates how to construct a depth map corresponding to the work area using binocular images captured by a self-operating device multiplexed by a binocular camera: The self-operating device first acquires the bidirectional parallax of the same scene point in the working area across the left and right eye images, and determines the target area with consistent bidirectional parallax. It can be understood that a point in the working area can be projected onto both the left and right eye images; that is, each of the left and right eye images contains a pixel representing the same scene point. The bidirectional parallax of the same scene point in the left and right eye images includes the parallax from the left to the right eye image and the parallax from the right to the left eye image. In other words, it represents the offset distance of the scene point in the left eye image relative to the same scene point in the right eye image in the image coordinates, and the offset distance of the scene point in the right eye image relative to the same scene point in the left eye image in the image coordinates.

[0053] Then, the self-operating equipment calculates the depth information of each pixel in the target area based on the bidirectional parallax of each pixel, thereby obtaining a depth map of the operating area, which can be represented as: ; in, Indicates coordinates as The depth information of the pixels, B represents the baseline of the dual-camera system, that is, the physical distance between the two cameras, and f represents the focal length of the camera. The baseline B and the focal length f are inherent parameters of the camera. Indicates coordinates as The disparity value of the pixels.

[0054] Finally, the self-operating equipment cuts out the depth sub-map corresponding to the abnormal area from the depth map, thereby obtaining the depth information of the abnormal area.

[0055] As shown above, after obtaining the depth information of the abnormal area, it is further determined whether there is actual natural dirt in the abnormal area based on the depth information, so as to verify whether the dirt actually exists and obtain the dirt verification result, thereby reducing the false judgment rate.

[0056] Please refer to Figure 4 Based on depth information, it determines whether there is actual natural dirt in the abnormal area, and obtains the dirt verification result, including: S1301. Calculate the depth porosity of abnormal areas based on depth information; S1302. If the depth porosity reaches the first porosity threshold, the actual dirt verification result indicating the presence of natural dirt categories in the abnormal area is obtained.

[0057] It should be noted that "hole pixels" refer to pixel regions lacking effective depth information. The direct cause is that natural dirt (such as water droplets or mud spots) adhering to the surface of the binocular camera interferes with its capture, preventing the matching of the same scene point between the left and right images. Consequently, disparity values ​​cannot be calculated, and therefore effective depth information cannot be obtained. Based on this, this embodiment of the invention uses hole pixels to verify whether the abnormal regions identified by the image anomaly detection model contain actual dirt of the natural dirt category.

[0058] The self-operating equipment first calculates the depth porosity of the abnormal area based on the depth information of the abnormal area, which can be expressed as: ; in, Indicates depth porosity. This indicates the number of pixels with holes within the abnormal area. This represents the total number of pixels within the abnormal region.

[0059] As shown above, after calculating the depth porosity of the abnormal area, the self-operating equipment further determines whether the depth porosity reaches the first porosity threshold. If it does, the corresponding dirt verification result of the actual dirt of the abnormal area with natural dirt category is obtained.

[0060] It should be noted that the embodiments of the present invention do not impose specific restrictions on the value of the first porosity threshold mentioned above. It can be configured by those skilled in the art according to actual needs. For example, the first porosity threshold can be configured to be 60%. That is, after the abnormal area with natural dirt in the binocular image is identified by the image anomaly detection model and the natural dirt category of the abnormal area is obtained, if the depth porosity of the abnormal area reaches 60%, the self-operating device will determine that there is actual dirt of the natural dirt category in the abnormal area. That is, the lens area of ​​the binocular camera corresponding to the abnormal area is attached with actual dirt of the natural dirt category.

[0061] Alternatively, in one embodiment, please refer to Figure 5 If the depth porosity reaches the first porosity threshold, then the actual dirt verification result indicating the presence of natural dirt categories in the abnormal area is obtained, including: S13021. If the depth porosity reaches the first porosity threshold, then obtain the depth change information of the edge of the abnormal area based on the depth information. S13022. If the depth change information matches the preset depth change information corresponding to the natural dirt category, then the dirt verification result indicating that the abnormal area has actual dirt of the natural dirt category is obtained.

[0062] To further reduce the false positive rate of dirt, in this embodiment of the invention, when the depth porosity of the self-operating equipment in the abnormal area reaches the first porosity threshold, it does not directly determine that there is actual dirt of the natural dirt category in the abnormal area. Instead, it first obtains the depth change information of the edge of the abnormal area, and further identifies whether the obtained depth change information matches the preset depth change information corresponding to the natural dirt category. If the obtained depth change information matches the preset depth change information corresponding to the natural dirt category, a dirt verification result indicating that there is actual dirt of the natural dirt category in the abnormal area is obtained; otherwise, a dirt verification result indicating that there is no actual dirt of the natural dirt category in the abnormal area is obtained. For example, if the natural dirt category is mud, because the mud is attached to the binocular camera, it will affect the readings of the left and right images. During depth calculation, due to the mismatch between the left and right eye images, large areas of invalid depth and a small number of abrupt depth values ​​will be formed, with relatively blurred edges and patchy discontinuities in the depth values. If the depth change information matches the patchy discontinuity change, it is considered that the abnormal area contains actual dirt of the mud type. If the natural dirt type is water droplets, and the water droplets are attached to the binocular camera, during depth calculation of the left and right eye images, there will be local continuous abnormal values, but they are not zero. The circular or elliptical edges of the water droplets are partially translucent, affecting the imaging module. The depth change is reflected in the local depth value being significantly closer to the depth value of the surrounding area. If the depth change information of the abnormal area matches the type of water droplets in the natural dirt category, it is considered that the abnormal area contains actual dirt of the water droplet category.

[0063] For example, taking a lawnmower robot as an example, in the working environment of a lawnmower robot, the natural dirt that can adhere to the robot's binocular cameras is usually water droplets and mud spots. If the natural dirt is water droplets, the local depth value of the corresponding abnormal area is significantly closer to the depth value of the surrounding area, and the depth information at its edge shows continuous and smooth abrupt changes. In contrast, the depth information at the edge of mud spots shows patchy discontinuous changes. Based on this, the depth change information of the abnormal area can be used for secondary verification of dirt. For example, assuming the depth change information of the edge of the abnormal area describes that its edge depth information presents patchy discontinuous changes, and the natural dirt type of the abnormal area is mud spots, then the self-operating equipment can determine that there is actual dirt of the natural dirt type mud spots in the abnormal area, and accordingly obtain the dirt verification result that there is actual dirt of the natural dirt type mud spots in the abnormal area; assuming the depth change information of the edge of the abnormal area describes that its edge presents continuous, smooth abrupt changes, and the natural dirt type of the abnormal area is water droplets, then the self-operating equipment can determine that there is actual dirt of the natural dirt type water droplets in the abnormal area, and accordingly obtain the dirt verification result that there is actual dirt of the natural dirt type water droplets in the abnormal area.

[0064] Optionally, in one embodiment, after calculating the depth porosity of the abnormal region based on the depth information, the method further includes: If the depth porosity is less than the second porosity threshold, a dirt verification result is obtained indicating that there is actual non-natural dirt in the abnormal area.

[0065] Based on the descriptions in the above embodiments, it can be understood that unnatural dirt is not directly attached to the binocular camera. Therefore, matching of the same scene point can be performed in the left and right eye images, and the disparity value can be calculated, thus obtaining effective depth information. Based on this, this embodiment of the invention uses hole pixels to verify whether the abnormal areas identified by the image anomaly detection model contain actual unnatural dirt.

[0066] In this process, after the depth porosity of the abnormal area is obtained by statistical analysis based on the depth information, if the obtained depth porosity is less than the second porosity threshold, the self-operating equipment determines that the image anomaly detection model is incorrect. That is, the abnormal area does not have actual natural dirt, but has actual non-natural dirt. Accordingly, a dirt verification result indicating that the abnormal area does not have actual natural dirt, but has actual non-natural dirt is obtained.

[0067] It should be noted that, with the second porosity threshold being less than or equal to the first porosity threshold as a constraint, the threshold of the second porosity threshold can be configured by those skilled in the art according to actual needs. No specific restrictions are imposed in the embodiments of the present invention. For example, the second porosity threshold can be configured to 50%.

[0068] Optionally, in one embodiment, after determining whether there is actual dirt of a certain type in the abnormal area based on the depth information and obtaining the dirt verification result, the method further includes: Obtain the dirt verification result set of all binocular images corresponding to the work area within the first time window of the binocular image; If the percentage of actual dirt in the dirt verification results indicating abnormal areas that are of the natural dirt category reaches the first percentage threshold, then a dirt cleaning operation will be performed.

[0069] In this embodiment of the invention, the dirt verification results of the binocular images acquired by the binocular camera are statistically analyzed using a preset time window, and a dirt cleaning operation for the binocular camera is determined based on the statistical results. Specifically, for the aforementioned binocular images, the time window containing the binocular image is designated as the first time window. A dirt verification result set is formed by the dirt verification results of all binocular images acquired by the binocular camera within this first time window, obtained from the operating equipment. Then, the proportion of actual dirt in the dirt verification result set indicating abnormal areas containing natural dirt is calculated. It is then determined whether this proportion reaches a first proportion threshold. If it does, a dirt cleaning operation for the binocular camera is required, and the operation is performed accordingly. Otherwise, no dirt cleaning operation is required. The value of the first proportion threshold is not specifically limited and can be configured by those skilled in the art according to actual needs. For example, in this embodiment of the invention, the first proportion threshold is configured as 60%.

[0070] It should be noted that the embodiments of the present invention do not impose specific limitations on the value of the above-mentioned preset duration, which can be configured by those skilled in the art according to actual needs. For example, in the embodiments of the present invention, the preset duration is configured to be 5 seconds. Assuming that the image acquisition array of the binocular camera is 5fps, the binocular camera will acquire 25 binocular images within a time window. The working device will obtain a dirt verification result set composed of the dirt verification results of these 25 binocular images, and use the dirt verification result set to determine the necessity of the dirt cleaning operation.

[0071] Optionally, in one embodiment, performing a dirt cleaning operation includes: Identify the target dirt cleaning operation corresponding to the natural dirt category and execute the target dirt cleaning operation.

[0072] It should be noted that, in this embodiment of the invention, to ensure the effectiveness of the dirt cleaning operation, that is, to ensure as much as possible that the dirt cleaning operation can remove the actual dirt of the natural dirt category attached to the binocular camera, the self-operating device, when performing the dirt cleaning operation, first determines the dirt cleaning operation corresponding to the above-mentioned natural dirt category as the target dirt cleaning operation according to the preset correspondence between natural dirt category and dirt cleaning operation, and then executes the determined target dirt cleaning operation. The configuration of the correspondence between the natural dirt category and dirt cleaning operation is not specifically limited here, and can be configured by those skilled in the art according to actual needs.

[0073] For example, taking a self-operating device as a lawnmower robot equipped with a windshield wiper for cleaning the binocular camera, the cleaning operation for natural dirt type - mud spots is: controlling the windshield wiper to spray cleaning fluid while performing the wiper operation; for natural dirt type - water droplets, the cleaning operation is: directly performing the wiper operation.

[0074] Furthermore, it should be noted that when there is no corresponding cleaning operation for the abnormal area in the mapping between natural dirt categories and cleaning operations, the self-operating equipment will also output a dirt cleaning prompt message to remind the user to manually remove the dirt. There are no specific restrictions on the output method of the dirt cleaning prompt message; for example, the self-operating equipment can output a voice prompt message such as "The camera is dirty, please clean it manually."

[0075] Optionally, in one embodiment, the dirt detection method provided by the present invention further includes: During the cleaning process, abnormal areas are marked as non-obstacle avoidance detection areas.

[0076] It should be noted that self-operating equipment is usually equipped with obstacle avoidance detection function. Obstacle avoidance detection is the core safety function for self-operating equipment to achieve autonomous operation. Its essence is to identify obstacles that may cause collisions (such as rocks, tree roots, animals) and trigger avoidance actions (such as slowing down or detouring) to ensure the safety of itself and the working environment.

[0077] In this embodiment of the invention, the obstacle avoidance detection function implemented by the self-operating device using binocular images captured by a binocular camera is used as an example. It is understood that when there are abnormal areas in the binocular image that affect image quality due to dirt, the presence of these abnormal areas may cause the self-operating device to falsely detect "virtual obstacles" and perform obstacle avoidance operations without warning. Therefore, in this embodiment of the invention, during the dirt cleaning operation, the self-operating device marks abnormal areas as non-obstacle avoidance detection areas, that is, it does not perform obstacle detection on these abnormal areas, thereby avoiding false obstacle detection.

[0078] Optionally, in one embodiment, after inputting the binocular images into a classification sub-model to identify abnormal regions in the binocular images that contain natural or unnatural dirt, the method further includes: If an abnormal area with unnatural dirt is identified in the binocular image, the recognition result set of all binocular images corresponding to the work area within the second time window of the binocular image is obtained. If the proportion of non-natural dirt in the identification result set reaches the second proportion threshold, then the non-natural dirt is determined to be a man-made obstruction.

[0079] In this embodiment of the invention, the identification results of whether there is natural or unnatural dirt in the binocular images captured by the binocular camera are statistically analyzed using a preset time window, and the specific type of unnatural dirt in the abnormal area is determined based on the statistical results.

[0080] Specifically, for the binocular images that identify abnormal areas with unnatural dirt, the time window in which the binocular image is located is designated as the second time window. The identification result set, comprised of the identification results of all binocular images acquired by the binocular camera within this second time window, is obtained from the operating equipment. Then, the proportion of abnormal areas indicating unnatural dirt in this identification result set is calculated, and it is determined whether this proportion reaches a second proportion threshold. If it does, the unnatural dirt in the abnormal area is determined to be a man-made obstruction, and no dirt cleaning operation is required. The value of the second proportion threshold is not specifically limited here and can be configured by those skilled in the art according to actual needs. For example, in this embodiment of the invention, the second proportion threshold is configured as 50%.

[0081] Alternatively, in one embodiment, please refer to Figure 6 Before inputting the binocular images into the image anomaly detection model, the following steps are also included: S140. Obtain the brightness histogram of the binocular image; S150, Statistical brightness distribution information of brightness histogram, and determine the target scene type of the scene where the work area is located based on the brightness distribution information; S160. Preprocess the stereo image according to the preprocessing strategy corresponding to the target scene type.

[0082] In this embodiment of the invention, corresponding preprocessing strategies are configured for different types of work scenarios to enhance the image quality of the binocular images captured by the binocular camera, thereby improving the accuracy of subsequent dirt detection.

[0083] The self-operating equipment first acquires the brightness histogram of the binocular images, then calculates the brightness distribution information of the brightness histogram, and determines the scene type of the scene where the operating area is located (i.e., the operating scene) based on the brightness distribution information, which is denoted as the target scene type. Specifically, if the peak value of the brightness histogram described by the brightness distribution information is in the low value range (e.g., the proportion of pixels with brightness <50 is >60%), the scene where the operating area is located is determined to be a dark light scene. If the brightness histogram described by the brightness distribution information shows a bimodal distribution (i.e., the dark and bright areas are densely populated with pixels, with a gap in the middle), the scene where the operating area is located is determined to be a backlight scene. If the brightness histogram described by the brightness distribution information is uniformly distributed and relatively bright (e.g., the proportion of pixels with brightness >200 is >40%), the scene where the operating area is located is determined to be a hazy scene.

[0084] As shown above, after determining the target scene type of the work area, the self-operating device further preprocesses the binocular images according to the preprocessing strategy corresponding to the target scene type.

[0085] It should be noted that the embodiments of the present invention do not impose specific restrictions on the prediction processing strategies corresponding to different scene types, and can be configured by those skilled in the art according to actual needs.

[0086] For example, for low-light scenes, the corresponding preprocessing strategy is to first use gamma correction to brighten the dark details of the binocular image, and then perform non-local mean denoising on the binocular image; for backlight scenes, the corresponding preprocessing strategy is to first compress the dynamic range of the binocular image, and then perform bilateral filtering on the binocular image; for hazy scenes, first perform dehazing enhancement on the binocular image, and then perform wavelet threshold denoising on the binocular image.

[0087] The following example uses a lawnmower robot. Please refer to... Figure 7 The process for a lawnmower robot to detect dirt can be as follows: The lawnmower robot operates in the lawnmower area, acquiring binocular images of the area using a binocular camera. After predicting the acquired images, the data is input into an image anomaly detection model. The model performs closed-loop judgment using its classification and detection sub-models. If an abnormal area with natural dirt is identified in the binocular image, and the type of natural dirt in the abnormal area is obtained, the depth information of the abnormal area is used to verify the identification result. The verification result is then obtained. Finally, based on multiple verification results within the same time window, a decision is made on whether to perform a dirt cleaning operation. If so, the dirt cleaning operation is performed.

[0088] As described above, the dirt detection scheme for cameras provided by this invention acquires binocular images of the work area captured by a binocular camera; inputs these binocular images into an image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and obtains the dirt category of the abnormal regions; acquires the depth information of the abnormal regions, and determines whether there is actual dirt of the specified category in the abnormal regions based on the depth information, thus obtaining a dirt verification result. Therefore, by employing an image anomaly detection model, preliminary detection of abnormal regions affecting image quality due to dirt is achieved, as well as classification of the dirt category in the abnormal regions. This eliminates the need for manual visual inspection, realizing automated detection of dirt in cameras. Furthermore, the detection and classification results of the image anomaly detection model are verified using depth information, ensuring the accuracy of automated detection.

[0089] In one embodiment, a self-operating device is provided, the internal structure of which can be as follows: Figure 8As shown. The self-operating device includes a main body and a memory 210, processor 220, power supply 230, sensor 240, operating mechanism 250, communication module 260, positioning module 270, drive wheel 280, and bus 290 disposed on the main body. The processor 220 is coupled to the memory 210, power supply 230, sensor 240, operating mechanism 250, communication module 260, positioning module 270, and drive wheel 280 respectively via the bus 290.

[0090] Memory 210 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The RAM can be directly read and written by the processor 220 and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDRAM). (memory, DDR SDRAM, etc.)

[0091] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 220. Non-volatile memory can include disk storage devices and flash memory.

[0092] The memory 210 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 220. The one or more computer programs include multiple instructions that, when executed by the processor 103, enable a dirt detection method for a camera to be executed on a self-operating device.

[0093] In other embodiments, the self-operating device also includes an external memory interface for connecting to an external memory to expand the storage capacity of the self-operating device.

[0094] Processor 220 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processor. Processing units (NPUs), etc. Different processing units can be independent devices or integrated into one or more processors.

[0095] Processor 220 provides computing and control capabilities; for example, processor 220 is used to execute computer programs stored in memory 210 to implement the path planning method described above.

[0096] The power supply 230 is used to supply power to the self-operating equipment. In one embodiment of this application, the power supply 230 may include any one or more power supply devices of the type of battery, fuel generator, solar power generation module, wind power generation module, etc.

[0097] Sensor 240 is used to acquire information for the self-operating device, such as environmental information and movement information of the self-operating device. In one embodiment of this application, sensor 240 may include a binocular camera, and may also include one or more sensors of the type such as lidar, infrared sensor, encoder, etc.

[0098] The working mechanism 250 is used to perform corresponding work tasks, such as mowing, de-icing, patrolling, sweeping, and spraying pesticides. In some embodiments of this application, the working mechanism 250 may include a motor, a transmission mechanism, and a blade disc. When the self-operating device is a lawnmower, the motor can drive the blade disc to rotate through the transmission mechanism to achieve the mowing function. The motor can also control the movement of the blades to adjust the mowing height and the mowing area.

[0099] The communication module 260 is used to enable communication between the self-operating device and other devices. In one embodiment of this application, the communication module 260 can interact with other devices via wired and / or wireless communication. The aforementioned wireless communication may include one or more combinations of communication methods such as Bluetooth communication, Wi-Fi communication, and Near Field Communication (NFC).

[0100] The positioning module 270 is used to determine the location of the self-operating device. In some embodiments of this application, the positioning module 270 may include one or more positioning modules of the type such as Global Positioning System (GPS), inertial navigation system, and real-time kinematic (RTK) carrier phase differential system.

[0101] The drive wheel 280 is used to enable the self-operating device to move. In some embodiments of this application, the drive wheel 280 can realize the movement function of the self-operating device according to the control of the processor 220. In some embodiments of this application, the drive wheel 280 may include a left drive wheel and a right drive wheel.

[0102] Bus 290 is used at least to provide a channel for communication between the memory 210, processor 220, power supply 230, sensor 240, working mechanism 250, communication module 260, positioning module 270, and drive wheel 280 in the self-operating device.

[0103] In other embodiments of this application, the self-operating device may further include a collision avoidance section and a steering assembly. The collision avoidance section can be used to prevent the drive wheels 280 from colliding with obstacles or the like in front of the self-operating device. The steering assembly can be used to adjust the driving direction of the drive wheels 280.

[0104] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the self-operating device. In other embodiments of this application, the self-operating device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] In one embodiment, a self-operating device is provided, including a binocular camera, a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the dirt detection method of the camera described in the above embodiment. For example, please refer to... Figure 9 The actual product form of the self-operating equipment can be a lawn mowing robot. The lawn mowing robot also includes an operating mechanism and drive wheels. The operating mechanism includes a rotatable blade, which can perform lawn mowing operations while the drive wheels drive the lawn mowing robot to move in the lawn mowing area.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the dirt detection method of the camera in the above embodiment.

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0109] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0110] It should be noted that when the above embodiments of the present invention are applied to specific products or technologies, and user-related data is involved, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for detecting dirt in a camera, characterized in that, include: Acquire binocular images of the work area captured by a binocular camera; The binocular image is input into an image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and to obtain the type of dirt in the abnormal regions. Obtain the depth information of the abnormal area, and determine whether there is actual dirt of the dirt category in the abnormal area based on the depth information, so as to obtain the dirt verification result.

2. The method for detecting dirt in a camera according to claim 1, characterized in that, The image anomaly detection model includes a classification sub-model and a detection sub-model. The binocular image is input into the image anomaly detection model to identify whether there are abnormal regions in the image that affect image quality, and to obtain the dirt category of the abnormal regions, including: The binocular image is input into the classification sub-model to identify whether there are abnormal areas of natural or unnatural dirt in the binocular image; If an abnormal region with natural dirt is detected in the binocular image, the binocular image is input into the detection sub-model to obtain the natural dirt category of the abnormal region.

3. The method for detecting dirt in a camera according to claim 2, characterized in that, Based on the depth information, determine whether the abnormal area contains actual dirt of the specified dirt type, and obtain a dirt verification result, including: The depth porosity of the abnormal region is calculated based on the depth information. If the depth porosity reaches the first porosity threshold, a dirt verification result indicating the presence of the natural dirt category in the abnormal area is obtained.

4. The method for detecting dirt in a camera according to claim 3, characterized in that, If the depth porosity reaches a first porosity threshold, a dirt verification result indicating the presence of the natural dirt category in the abnormal area is obtained, including: If the depth porosity reaches the first porosity threshold, then the depth change information of the edge of the abnormal region is obtained based on the depth information; If the depth change information matches the preset depth change information corresponding to the natural dirt category, a dirt verification result indicating that the abnormal area contains actual dirt of the natural dirt category is obtained.

5. The method for detecting dirt in a camera according to claim 3, characterized in that, After calculating the depth porosity of the abnormal region based on the depth information, the method further includes: If the porosity is less than the second porosity threshold, a dirt verification result indicating that there is actual non-natural dirt in the abnormal area is obtained.

6. The method for detecting dirt in a camera according to claim 2, characterized in that, After determining whether the abnormal area contains actual dirt of the specified dirt type based on the depth information and obtaining the dirt verification result, the method further includes: Obtain the dirt verification result set of all binocular images corresponding to the work area within the first time window of the binocular image; If the dirt verification results indicate that the proportion of actual dirt of the natural dirt category in the abnormal area reaches a first proportion threshold, then a dirt cleaning operation is performed.

7. The method for detecting dirt in a camera according to claim 6, characterized in that, The dirty cleaning operation includes: Determine the target dirt cleaning operation corresponding to the natural dirt category, and execute the target dirt cleaning operation.

8. The method for detecting dirt in a camera according to claim 6, characterized in that, Also includes: During the execution of the dirt cleaning operation, the abnormal area is marked as a non-obstacle avoidance detection area.

9. The method for detecting dirt in a camera according to claim 2, characterized in that, After inputting the binocular image into the classification sub-model to identify whether there are abnormal regions of natural or unnatural dirt in the binocular image, the method further includes: If an abnormal area with unnatural dirt is identified in the binocular image, then the recognition result set of all binocular images corresponding to the work area within the second time window of the binocular image is obtained; If the proportion of non-natural dirt in the identification result set reaches the second proportion threshold, then the non-natural dirt is determined to be a man-made obstruction.

10. The method for detecting dirt in a camera according to any one of claims 1-9, characterized in that, Before inputting the binocular image into the image anomaly detection model, the method further includes: Obtain the brightness histogram of the binocular image; The brightness distribution information of the brightness histogram is statistically analyzed, and the target scene type of the scene in which the work area is located is determined based on the brightness distribution information; The binocular image is preprocessed according to the preprocessing strategy corresponding to the target scene type.

11. A self-operating device, comprising a binocular camera, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the dirt detection method for the camera according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the dirt detection method for the camera according to any one of claims 1 to 10.

Images