A camera contamination detection confirmation method and system for a self-moving device

Abstract

The application discloses a camera pollution detection confirmation method and system applied to a self-moving device. A main control controller calls a camera to detect a travelable area and returns credibility. When the credibility is lower than a preset threshold, the main control controller calls the camera to detect foreign matters. Images of the camera at different moments are introduced into a pollution detection model respectively to obtain respective recognition results. If the two images have the same foreign matter, it is considered that the camera is polluted. The main control controller reduces the credibility of the camera to 0, starts a camera-free mode, and then starts the camera every interval. The application uses the credibility of the travelable area to determine whether to start the pollution detection model, uses the motion control of the self-moving device to detect and confirm the lens pollution, removes the interference of obstacles, and improves the accuracy of the pollution detection.

Description

TECHNICAL FIELD

[0001] The present application relates to a lens pollution detection method, in particular to a camera pollution detection and confirmation method and system applied to a self-moving device. BACKGROUND

[0002] Most self-moving devices on the market mainly rely on image data obtained by the camera to realize self-moving, and once the camera is polluted, many devices cannot recognize the surrounding environment and directly shut down. During the operation of outdoor devices, it is inevitable to encounter rainy, foggy and other bad weather, and rainwater will block the camera, causing the device to fail to operate autonomously. Generally, whether the camera is polluted is judged by analyzing the image obtained by the camera, and a pollution detection module using deep learning is used to judge whether the camera is polluted. However, the judgment method of pure algorithm is not 100% correct, and obstacles and pollutants may be confused, causing the device to stop incorrectly and affecting work efficiency. And the current treatment method for the pollutants on the camera is to stop, which is not conducive to the autonomous operation of the device. SUMMARY

[0003] In order to solve the above problems, the present application provides a camera pollution detection and confirmation method and system applied to a self-moving device, which can use the motion control of the self-moving device to detect and confirm the lens pollutants and reduce the possibility of false detection.

[0004] Therefore, the technical scheme of the present application is: a camera pollution detection and confirmation method applied to a self-moving device, which is provided with a main control controller, a camera, a positioning module, an ultrasonic sensor and a collision detection sensor on the self-moving device, and comprises the following steps:

[0005] 1) The device starts operation and the camera is enabled;

[0006] 2) The main control controller calls the camera to detect the travelable area;

[0007] 3) The camera imports the image data obtained by the camera into the travelable area detection model to detect the travelable area and returns the credibility;

[0008] 4) The credibility of the camera is compared with a preset threshold value, when the credibility is greater than the preset threshold value, the device continues to move, and when the credibility is lower than the preset threshold value, the main control controller considers that the camera has recognized a foreign object;

[0009] 5) The main control controller calls the camera to detect the foreign object, and the pollution detection model in the camera further identifies and analyzes; the pollution detection steps are as follows:

[0010] i) The camera obtains an image S1 at T1 moment and an image S2 at T1+△t moment, wherein the main control controller changes the travel strategy in the△t moment;

[0011] ii) the image S1 and the image S2 are respectively introduced into the contaminant detection model to obtain respective recognition results, if foreign matters are detected on the image S1 and the image S2, and the foreign matters are located at the same position of the images, it is considered that the camera is contaminated, and step 7) is executed; if foreign matters are detected on the image S1 and the image S2, but the positions of the foreign matters are different, or the foreign matter disappears on the image S2, it is considered that the foreign matter is an obstacle, and step 6) is executed;

[0012] 6) the master controller controls the device to avoid the obstacle, continues to work, and returns to execute step 2);

[0013] 7) the master controller reduces the credibility of the camera to 0, and starts the camera-free mode work.

[0014] Preferably, in the camera-free mode work, every interval of time, the camera is started to work, and the contaminant detection of step 5) is executed, if the contaminant still exists, the camera is closed, and the next call is waited; if the contaminant disappears, the master controller fuses the data obtained by the camera into the motion control again.

[0015] Preferably, the travel strategy in the step i) includes but is not limited to turning around, rotating in place.

[0016] Preferably, in the step 4), when the credibility is greater than a preset threshold, the device continues to move, if the ultrasonic sensor and / or the collision detection sensor detect that the device collides, step 5) is executed.

[0017] Preferably, in the step 7), in the camera-free mode work, the master controller controls the device to continue to work in combination with the positioning information sent by the positioning module, the ultrasonic sensor and the collision detection sensor.

[0018] Preferably, the construction step of the passable area detection model is as follows:

[0019] S1) image preprocessing: the original image is cropped to retain only the to-be-detected area, the size is adjusted to a set size, and then standardized processing is performed to obtain a standard image;

[0020] S2) a passable area detection model is constructed, the model is composed of a backbone network, a feature fusion layer and a classification head, the backbone network is a resnet network, a vggnet network or a googlenet network, the feature fusion layer splices and fuses features of different scale feature layers, the classification head performs operation on the features output by the upper layer, and finally outputs a 3x3 matrix;

[0021] S3) input the standard image into the passable area detection model to obtain an image block-based recognition result; that is, the image is divided into a 3x3 grid, and the result output by the model is a 3x3 matrix, the value of each grid being 0 or 1, 0 being an unpassable area and 1 being a passable area.

[0022] Preferably, the construction step of the pollutant detection model is as follows:

[0023] a1) image preprocessing: the original image is cropped to retain only the region to be detected, resized to a set size, and then normalized to obtain a standard image;

[0024] a2) constructing a pollutant detection model, the model comprising a backbone network, a feature fusion layer and a classification head, the backbone network being a resnet network, a vggnet network or a googlenet network, the feature fusion layer splicing and fusing features of different scale feature layers, and the classification head performing operation on the features output by the upper layer and finally outputting a 3x3 matrix;

[0025] a3) inputting the standard image into the pollutant detection model to obtain an image block-based recognition result; that is, the image is divided into a 3x3 grid, and the result output by the model is a 3x3 matrix, the value of each grid being 0 or 1, 0 being a non-pollutant and 1 being a pollutant.

[0026] Another technical solution of the present application is a camera pollution detection and confirmation system for a self-moving device, comprising a main control controller, a camera, a camera processing unit, a positioning module, an ultrasonic sensor and a collision detection sensor,

[0027] The camera is used to acquire image data in real time and send the image data to the camera processing unit;

[0028] The camera processing unit comprises a passable area detection model and a pollutant detection model;

[0029] The passable area detection model is used to identify the range in the area that meets the device movement and send the passable area information to the main control controller;

[0030] The pollutant detection model is used to detect the pollutant information on the camera and send the pollutant information to the main control controller;

[0031] The main control controller calls the pollutant detection model according to the received passable area information;

[0032] The main control controller controls the device movement according to the received pollutant information.

[0033] Preferably, the positioning module is configured to detect positioning information and send the positioning information to the master controller; the ultrasonic sensor and the collision detection sensor are configured to assist the device in traveling.

[0034] Compared with the prior art, the present application has the following advantages:

[0035] 1. The credibility of the travelable area is used to determine whether to start the pollutant detection model, and the motion control of the self-moving device is used to determine the lens pollutant detection and confirm the lens pollutant detection, so as to eliminate the interference of obstacles and improve the accuracy of the pollutant detection.

[0036] 2. The camera pollutant detection control is intermittently enabled, and after the camera is free from the pollutant, the camera data is quickly fused again to restore the normal operation mode of the device and ensure the working efficiency of the device. BRIEF DESCRIPTION OF DRAWINGS

[0037] The present application will be further described in detail below with reference to the accompanying drawings and embodiments of the present application

[0038] Figure 1 The present application is a workflow diagram.

[0039] Figure 2 The present application is a workflow diagram of the pollutant detection model.

[0040] Figure 3 The present application is a structural diagram of the pollutant detection model. DETAILED DESCRIPTION

[0041] In the description of the present application, it should be noted that for orientation words such as the terms "center", "lateral (X)", "longitudinal (Y)", "vertical (Z)", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. indicate the orientation and positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and cannot be understood as limiting the specific protection scope of the present application.

[0042] In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features. Therefore, the "first" and "second" features can explicitly or implicitly include one or more features, and in the description of the present application, the meaning of "several" is two or more, unless otherwise explicitly specified.

[0043] Referring to the drawings. The camera pollution detection and confirmation system described in this embodiment is applied to a self-moving device, which includes a smart robot, a smart mower, etc., and can move autonomously.

[0044] The camera pollution detection and confirmation system includes a master controller, a camera, a camera processing unit, a positioning module, an ultrasonic sensor, and a collision detection sensor,

[0045] The camera is used to acquire image data in real time and send the image data to the camera processing unit;

[0046] The camera processing unit includes a passable area detection model and a pollution detection model;

[0047] The passable area detection model is used to identify the range of the area that meets the device's travel and send the passable area information to the master controller;

[0048] The pollution detection model is used to detect pollution information on the camera and send the pollution information to the master controller;

[0049] The master controller calls the pollution detection model according to the received passable area information;

[0050] The master controller controls the device to travel according to the received pollution information; the master controller can make the device move in the expected manner; the master controller sends control instructions to the camera through the communication module, and the camera also returns the recognition result;

[0051] The positioning module is used to detect positioning information and send the positioning information to the master controller; the ultrasonic sensor and the collision detection sensor are used to assist the device to travel.

[0052] The construction steps of the passable area detection model are as follows:

[0053] S1) Image preprocessing: crop the original image to only keep the detection area, because part of the camera's field of view is useless and the recognition algorithm will not recognize those areas, so it is necessary to crop them in advance, adjust the size to the set size, and then perform standardization processing to obtain a standard image.

[0054] S2) Construct a passable area detection model, which consists of a backbone network, a feature fusion layer, and a classification head. The backbone network is a resnet network, a vggnet network, or a googlenet network. The feature fusion layer is used to splice and fuse features of different scale feature layers to improve the detection effect of targets of different sizes and expand the feature dimension. The classification head performs operations on the features output by the upper layer, and finally outputs a 3x3 matrix; the construction of this model is a mature technology, and different recognition abilities can be obtained by training with different data.

[0055] S3) input the standard image into the passable area detection model to obtain an image block-based recognition result; that is, the image is divided into a 3x3 grid, and the model outputs a 3x3 matrix, with each cell having a value of 0 or 1, 0 representing an unpassable area and 1 representing a passable area. Thus, the passable area detection model can finally return a credibility.

[0056] The construction steps of the pollutant detection model are as follows:

[0057] a1) image preprocessing: the original image is cropped to retain only the detection area, resized to a set size, and then standardized to obtain a standard image;

[0058] a2) constructing a pollutant detection model, which is composed of a backbone network, a feature fusion layer, and a classification head, the backbone network being a resnet network, a vggnet network, or a googlenet network, the feature fusion layer splicing and fusing features of different scale feature layers, and the classification head performing operations on the features output by the upper layer and finally outputting a 3x3 matrix;

[0059] a3) inputting the standard image into the pollutant detection model to obtain an image block-based recognition result; that is, the image is divided into a 3x3 grid, and the model outputs a 3x3 matrix, with each cell having a value of 0 or 1, 0 representing a non-pollutant and 1 representing a pollutant.

[0060] The pollutant detection model processing flow is as follows:

[0061] b1) during the operation of the self-moving device, two images are pre-stored. The pre-storing method is: every 10 seconds, the current image captured by the camera is stored, and if the current image storage has already stored two images, the earliest stored image is deleted. In this way, the device always stores two images.

[0062] B2) when the pollutant detection model is called, the two images are input into the model to obtain respective recognition results, and if the pollutant is detected twice and is located at the same position in the image, it is considered that the camera is contaminated.

[0063] The camera contamination detection confirmation method comprises the following steps:

[0064] 1) the device starts operation and enables the camera;

[0065] 2) the main control controller calls the camera to perform passable area detection;

[0066] 3) The camera imports the image data taken into the passable area detection model to detect the passable area and returns the confidence. The confidence refers to the percentage of the passable area in the current image. For example, the confidence of 70 means that 70% of the image is the passable area, and the remaining 30% of the image contains foreign matter, which may be an obstacle or a contaminant on the lens.

[0067] 4) The confidence of the camera is compared with the preset threshold. When the confidence is greater than the preset threshold, the device continues to move. When the confidence is lower than the preset threshold, the main controller considers that the camera has recognized foreign matter.

[0068] For example, when the confidence returned by the camera is lower than 70, the main controller calls the camera to detect foreign matter.

[0069] When the confidence returned by the camera is greater than 70, the device continues to move. However, if the positioning module, ultrasonic sensor, and collision detection sensor return information about foreign matter during the movement of the device, the contaminant detection is immediately started.

[0070] 5) The main controller calls the camera to detect foreign matter, and the contaminant detection model in the camera further identifies and analyzes. The contaminant detection steps are as follows:

[0071] i) The camera obtains the image S1 at T1 and the image S2 at T1+△t, wherein the main controller changes the movement strategy during △t. The movement strategy includes but is not limited to turning direction, rotating in place, such as moving forward, moving backward, rotating by a certain angle, etc.

[0072] ii) The images S1 and S2 are imported into the contaminant detection model to obtain the respective identification results. If the same foreign matter is detected on the images S1 and S2, and the foreign matter is located at the same position of the images, it is considered that the camera is contaminated, and step 7) is performed. If the foreign matter is detected on the images S1 and S2, but the positions of the foreign matter are different, or the foreign matter disappears on the image S2, it is considered that the foreign matter is an obstacle, and step 6) is performed.

[0073] 6) The main controller controls the device to avoid the obstacle and continue to work, and returns to step 2).

[0074] 7) The main controller reduces the confidence of the camera to 0 and starts the camera-free mode.

[0075] 8) In the no camera mode, every interval, for example, every 5 minutes, 10 minutes, etc., the camera is started and the pollutant detection of step 5) is performed. If the pollutant still exists, the camera is closed and the next call is waited. If the pollutant disappears, the data obtained by the camera is fused into the motion control by the master controller again. In the no camera mode, the master controller combines the positioning information sent by the positioning module, the ultrasonic sensor and the collision detection sensor to control the device to continue working.

[0076] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A method for detecting and confirming camera contamination in a self-moving device, wherein the self-moving device is equipped with a main controller, a camera, a positioning module, an ultrasonic sensor, and a collision detection sensor, characterized in that: Includes the following steps: 1) The equipment starts operating and the camera is activated; 2) The main controller calls the camera to detect the feasible area; 3) The camera imports the captured image data into the feasible region detection model to perform feasible region detection, and at the same time sends back the confidence level; 4) Compare the camera's reliability with a preset threshold. If the reliability is greater than the preset threshold, the device continues to move. When the confidence level is lower than a preset threshold, the main controller assumes that the camera has detected a foreign object. 5) The main controller calls the camera to perform foreign object detection, and the contaminant detection model in the camera further identifies and analyzes the contaminants; the contaminant detection steps are as follows: i) The camera acquires image S1 at time T1 and image S2 at time T1+Δt, where the main controller changes the travel strategy during time Δt; ii) Import images S1 and S2 into the pollutant detection model respectively to obtain their respective recognition results. If foreign objects are detected in both images S1 and S2 and the foreign objects are located in the same position in the images, the camera is considered to be contaminated, and step 7) is executed. If foreign objects are detected in both images S1 and S2 but the positions of the foreign objects are different, or the foreign object disappears in image S2, the foreign object is considered to be an obstacle, and step 6) is executed. 6) The main controller controls the equipment to avoid obstacles, continues working, and returns to step 2); 7) The main controller reduces the camera's confidence level to 0 and starts working in camera-free mode; In camera-free mode, the camera is activated at regular intervals to perform pollutant detection in step 5). If pollutants are still present, the camera is turned off and the system waits for the next call. If the pollutants disappear, the main controller will again integrate the data acquired by the camera into the motion control. The confidence level is the percentage of passable area on the current image.

2. The method for detecting and confirming camera contamination in self-moving devices as described in claim 1, characterized in that: The travel strategy in step i) includes, but is not limited to, changing direction and rotating in place.

3. The method for detecting and confirming camera contamination in self-moving devices as described in claim 1, characterized in that: In step 4), if the confidence level is greater than the preset threshold, the device continues to move. If the ultrasonic sensor and / or collision detection sensor detects a collision, step 5 is executed.

4. The method for detecting and confirming camera contamination in self-moving devices as described in claim 1, characterized in that: In step 7), in camera-free mode, the main controller combines the positioning information sent by the positioning module, the ultrasonic sensor, and the collision detection sensor to control the device to continue working.

5. The method for detecting and confirming camera contamination in self-moving devices as described in claim 1, characterized in that: The construction steps of the feasible region detection model are as follows: S1) Image preprocessing: The original image is cropped, keeping only the area to be detected, and the size is adjusted to the set size. Then, the image is standardized to obtain a standard image. S2) Construct a feasible region detection model. The model consists of a backbone network, a feature fusion layer, and a classification head. The backbone network is a ResNet network, a VGGNet network, or a GoogLeNet network. The feature fusion layer splices and fuses features from feature layers of different scales. The classification head performs calculations on the features output by the upper layer and finally outputs a 3x3 matrix. S3) Input the standard image into the feasible region detection model to obtain the recognition result based on image patches; that is, the image is divided into a 3x3 grid, and the model output is a 3x3 matrix, where each cell has a value of 0 or 1, where 0 is an infeasible region and 1 is a feasible region.

6. The method for detecting and confirming camera contamination in self-moving devices as described in claim 1, characterized in that: The steps for constructing the pollutant detection model are as follows: a1) Image preprocessing: The original image is cropped, keeping only the area to be detected, and the size is adjusted to the set size. Then, the image is standardized to obtain a standard image. a2) Construct a pollutant detection model. The model consists of a backbone network, a feature fusion layer, and a classification head. The backbone network is a ResNet network, a VGGNet network, or a GoogleNet network. The feature fusion layer splices and fuses features from feature layers of different scales. The classification head performs calculations on the features output by the upper layer and finally outputs a 3x3 matrix. a3) Input the standard image into the pollutant detection model to obtain the recognition result based on image patches; that is, the image is divided into a 3x3 grid, and the model output is a 3x3 matrix, where the value of each cell is 0 or 1, 0 for non-pollutants and 1 for pollutants.

7. A camera contamination detection and verification system for self-moving devices, used to implement the camera contamination detection and verification method for self-moving devices as described in any one of claims 1 to 6, characterized in that: It includes a main controller, camera, camera processing unit, positioning module, ultrasonic sensor, and collision detection sensor. The camera is used to acquire image data in real time and send the image data to the camera processing unit; The camera processing unit includes a drivable area detection model and a contaminant detection model; The drivable area detection model is used to identify the range within the area that is suitable for the equipment to travel, and sends the drivable area information to the main controller; The pollutant detection model is used to detect pollutant information on the camera and send the pollutant information to the main controller; The main controller invokes the pollutant detection model based on the received feasible area information; The main controller controls the movement of the equipment based on the received pollutant information.

8. The camera contamination detection and verification system for self-moving devices as described in claim 7, characterized in that: The positioning module is used to detect positioning information and send the positioning information to the main controller; the ultrasonic sensor and collision detection sensor are used to assist the movement of the equipment.

Images