Camera dirt detection method, device and electronic equipment

Abstract

The present invention provides a method, device and electronic device for detecting camera dirtiness. The method implements dirtiness level segmentation and scene classification of an image to be processed through a multi-task neural network model, and optimizes the dirtiness level segmentation result through a target dirtiness classification threshold value corresponding to the scene classification result, so that the dirty area in the optimized dirtiness level segmentation map is more accurate, and then the result of whether the camera to be detected is dirty, determined based on the above accurate dirty area, is also more accurate. The multi-task neural network model is used in the above process. The multi-task neural network model has good robustness when dealing with dirtiness level segmentation of images to be processed taken in various scenes. In addition, it is considered to further optimize the dirtiness level segmentation result by using the target dirtiness classification threshold value corresponding to the scene classification result, so that the final detection result is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device and electronic equipment for detecting camera dirtiness. Background Art

[0002] A large number of advanced sensors are used in the field of autonomous driving, which enables autonomous vehicles to have intelligent environmental perception capabilities and automatically analyze the safety and danger status of vehicle driving. However, most of these sensors are exposed to the outside of the vehicle. When the vehicle is driving in extreme scenarios, these sensors will be contaminated to varying degrees, especially the camera. For example, mud on the road splashes onto the camera surface, causing camera contamination, which reduces the clarity of the image captured by the camera, and then causes the image recognition perception results to decline, which ultimately has a great impact on the safety of autonomous driving. Therefore, it is extremely important to accurately detect camera contamination and handle it in a timely manner after detecting camera contamination.

[0003] At present, the process of detecting camera dirtiness is generally as follows: the image quality of the image captured by the camera is calculated by information entropy, and then it is determined whether the image quality is greater than a preset fixed threshold. If not, it is determined that the camera is dirty. In the above detection process, if the camera is shooting images at night or in backlighting, the image quality captured in such scenes is inherently poor, and is not caused by camera dirtiness. When the above method is used to detect camera dirtiness, it will be determined that the camera is dirty, resulting in false detection.

[0004] In summary, the existing camera dirtiness detection methods have technical problems of inaccurate detection and poor robustness. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a method, device and electronic device for detecting camera dirtiness, so as to alleviate the technical problems of inaccurate detection and poor robustness of existing camera dirtiness detection methods.

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting camera dirtiness, comprising:

[0007] Obtain the image to be processed taken by the camera to be detected;

[0008] Performing dirtiness level segmentation processing and scene classification processing on the image to be processed by a multi-task neural network model to obtain a dirtiness level segmentation result and a scene classification result, wherein the dirtiness level segmentation result includes: the dirtiness level category of each pixel point in the image to be processed and the probability corresponding to the dirtiness level category;

[0009] Determining a target dirt classification threshold corresponding to the scene classification result in a preset dirt classification threshold table, and optimizing the dirt degree segmentation result based on the target dirt classification threshold to obtain an optimized dirt degree segmentation map;

[0010] Performing binary morphological segmentation on the optimized dirtiness level segmentation map to obtain a dirtiness region map contained in the optimized dirtiness level segmentation map;

[0011] The number of dirty pixels contained in each of the dirty area maps is counted, and whether the camera to be detected is dirty is determined based on the number of dirty pixels.

[0012] Furthermore, the dirt level segmentation result is optimized based on the target dirt classification threshold, including:

[0013] Determine whether the probability corresponding to the dirtiness category of each pixel is greater than the target dirtiness classification threshold;

[0014] If there is a probability corresponding to the dirtiness level category of the first target pixel point that is not greater than the target dirtiness classification threshold, updating the dirtiness level category of the first target pixel point to a non-dirty background category;

[0015] If the probability corresponding to the dirtiness level category of the second target pixel point exists and is greater than the target dirtiness classification threshold, the dirtiness level category of the second target pixel point is not updated, thereby obtaining an optimized dirtiness level segmentation result, and the optimized dirtiness level segmentation result is visualized to obtain the optimized dirtiness level segmentation map.

[0016] Furthermore, counting the number of dirty pixels contained in each of the dirty area maps includes:

[0017] Counting the total number of pixels corresponding to each dirtiness level category in each of the dirtiness area maps;

[0018] The total number of pixels corresponding to each dirt level category and the preset weight corresponding to each dirt level category are weightedly calculated to obtain the number of dirty pixels contained in each of the dirt area maps.

[0019] Further, determining whether the camera to be detected is dirty based on the number of dirty pixels includes:

[0020] Determine whether there is at least one dirty pixel number among the dirty pixel numbers that is greater than a preset dirty pixel number threshold;

[0021] If there is at least one dirty pixel number greater than the preset dirty pixel number threshold, it is determined that the camera to be detected is dirty, and a dirt alarm is issued;

[0022] If the number of non-dirty pixels is greater than the preset dirty pixel number threshold, it is determined that the camera to be detected is not dirty.

[0023] Furthermore, the scene classification results include any one of the following: daytime scene, night scene, underground garage scene, and backlight scene, wherein, in the dirt classification threshold table, the dirt classification threshold corresponding to the night scene, the dirt classification threshold corresponding to the underground garage scene, and the dirt classification threshold corresponding to the backlight scene are all greater than the dirt classification threshold corresponding to the daytime scene.

[0024] Furthermore, the method further comprises:

[0025] Counting the total number of pixels contained in all the dirty area maps, and calculating the ratio of the total number of pixels to the total number of pixels of the image to be processed;

[0026] If the ratio is greater than a preset threshold, it is determined that the camera to be detected is blocked and an occlusion alarm is issued.

[0027] Furthermore, the multi-task neural network model is pre-based on the loss function Loss=loss seg +0.3*loss cls The multi-task neural network model includes a feature extraction module, a dirtiness segmentation module and a scene classification module respectively connected to the feature extraction module, wherein the loss seg Indicates the loss corresponding to the dirtiness segmentation, loss cls It represents the loss corresponding to the scene classification, the feature extraction module, the dirtiness segmentation module and the scene classification module all include a residual network layer, and the residual network layer includes channel shuffle fusion.

[0028] In a second aspect, an embodiment of the present invention further provides a camera dirt detection device, comprising:

[0029] An acquisition unit, used for acquiring an image to be processed taken by a camera to be detected;

[0030] A processing unit, configured to perform dirtiness level segmentation processing and scene classification processing on the image to be processed through a multi-task neural network model to obtain a dirtiness level segmentation result and a scene classification result, wherein the dirtiness level segmentation result includes: a dirtiness level category of each pixel point in the image to be processed and a probability corresponding to the dirtiness level category;

[0031] an optimization unit, configured to determine a target dirt classification threshold corresponding to the scene classification result in a preset dirt classification threshold table, and optimize the dirt degree segmentation result based on the target dirt classification threshold to obtain an optimized dirt degree segmentation map;

[0032] A binary morphological segmentation unit, used for performing binary morphological segmentation on the optimized dirtiness level segmentation map to obtain a dirtiness region map contained in the optimized dirtiness level segmentation map;

[0033] The determination unit is used to count the number of dirty pixels contained in each of the dirty area maps, and determine whether the camera to be detected is dirty based on the number of dirty pixels.

[0034] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute any method described in the first aspect above.

[0036] In an embodiment of the present invention, a method for detecting camera dirtiness is provided, comprising: obtaining an image to be processed taken by a camera to be detected; performing dirtiness degree segmentation processing and scene classification processing on the image to be processed through a multi-task neural network model to obtain a dirtiness degree segmentation result and a scene classification result, wherein the dirtiness degree segmentation result comprises: a dirtiness degree category of each pixel point in the image to be processed and a probability corresponding to the dirtiness degree category; determining a target dirtiness classification threshold corresponding to the scene classification result in a preset dirtiness classification threshold table, and optimizing the dirtiness degree segmentation result based on the target dirtiness classification threshold to obtain an optimized dirtiness degree segmentation map; performing binary morphological segmentation on the optimized dirtiness degree segmentation map to obtain a dirtiness region map contained in the optimized dirtiness degree segmentation map; counting the number of dirty pixels contained in each dirty region map, and determining whether the camera to be detected is dirty based on the number of dirty pixels. From the above description, it can be seen that the camera dirtiness detection method of the present invention implements the dirtiness degree segmentation and scene classification of the image to be processed through a multi-task neural network model, and also optimizes the dirtiness degree segmentation result through the target dirtiness classification threshold value corresponding to the scene classification result, so that the dirty area in the optimized dirtiness degree segmentation map is more accurate, and then the result of whether the camera to be detected is dirty based on the above accurate dirty area is also more accurate. The multi-task neural network model is used in the above process. The multi-task neural network model has good robustness when dealing with the dirtiness degree segmentation of the image to be processed taken in various scenes. In addition, it is also considered to use the target dirtiness classification threshold value corresponding to the scene classification result to further optimize the dirtiness degree segmentation result, so that the final detection result is more accurate, alleviating the technical problems of inaccurate detection and poor robustness of the existing camera dirtiness detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A flowchart of a method for detecting camera dirtiness provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of a multi-task neural network model provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of the residual network layer of the scene classification module provided by an embodiment of the present invention;

[0041] FIG4( a) is a schematic diagram of an image to be processed captured by a camera with sludge in a daytime scene according to an embodiment of the present invention;

[0042] FIG4( b ) is a schematic diagram of the optimized dirtiness level segmentation map of FIG4( a ) provided by an embodiment of the present invention;

[0043] FIG4( c ) is a schematic diagram of the dirty area map of FIG4( a ) provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of a camera dirt detection device provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the prior art, when detecting camera dirtiness, the image quality of the image captured by the camera is generally calculated by information entropy, and then it is determined whether the image quality is greater than a preset fixed threshold. If not, it is determined that the camera is dirty. In the above detection process, if the camera is shooting images at night or in backlighting, the image quality captured in such scenes is inherently poor, and is not caused by camera dirtiness. When the above method is used to detect camera dirtiness, it will be determined that the camera is dirty, resulting in false detection.

[0048] Based on this, the camera dirtiness detection method of the present invention realizes the dirtiness degree segmentation and scene classification of the image to be processed through a multi-task neural network model, and also optimizes the dirtiness degree segmentation result through the target dirtiness classification threshold value corresponding to the scene classification result, so that the dirty area in the optimized dirtiness degree segmentation map is more accurate, and then the result of whether the camera to be detected is dirty determined based on the above accurate dirty area is also more accurate. The multi-task neural network model is adopted in the above process. The multi-task neural network model has good robustness when dealing with the dirtiness degree segmentation of the image to be processed taken in various scenes. In addition, it is also considered to use the target dirtiness classification threshold value corresponding to the scene classification result to further optimize the dirtiness degree segmentation result, so that the final detection result is more accurate.

[0049] To facilitate understanding of this embodiment, a camera dirt detection method disclosed in an embodiment of the present invention is first introduced in detail.

[0050] Embodiment 1:

[0051] According to an embodiment of the present invention, an embodiment of a method for detecting camera dirtiness is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] Figure 1 FIG. 1 is a flow chart of a method for detecting camera dirtiness according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0053] Step S102, obtaining an image to be processed taken by the camera to be detected;

[0054] Specifically, the image to be processed may be an image captured by a camera to be detected.

[0055] Step S104, performing dirtiness level segmentation processing and scene classification processing on the image to be processed by a multi-task neural network model to obtain dirtiness level segmentation results and scene classification results, wherein the dirtiness level segmentation results include: dirtiness level categories of each pixel point in the image to be processed and the probability corresponding to the dirtiness level categories;

[0056] In the embodiment of the present invention, the multi-task neural network model is pre-based on the loss function Loss=loss seg +0.3*loss cls The trained neural network model includes a feature extraction module, a dirtiness segmentation module and a scene classification module respectively connected to the feature extraction module, wherein the loss seg Indicates the loss corresponding to the dirtiness segmentation, loss cls It represents the loss corresponding to scene classification. The feature extraction module, dirtiness segmentation module and scene classification module all contain residual network layers, and the residual network layer contains channel shuffle fusion.

[0057] Specifically, the feature extraction module is the backbone, and the classic residual idea of ​​Resnet is adopted. The dirtiness segmentation module (a segmentation head) detects the dirtiness category of each pixel in the image and the probability corresponding to the dirtiness category (i.e., the dirtiness segmentation result) through the image segmentation idea. The scene classification module (another classification head) detects the shooting scene of the image (i.e., the scene classification result) through the image classification idea. The multi-task neural network model is trained by various dirtiness data (e.g., training samples of water drop dirtiness, training samples of fog dirtiness, and training samples of sludge dirtiness, etc.). Due to the diversity of dirtiness data samples, the trained multi-task neural network model can identify various types of dirtiness (i.e., it can identify dirtiness caused by water droplets, dirtiness caused by fog, dirtiness caused by sludge, etc.), and has good robustness to dirtiness caused by various reasons, and obtains the dirtiness category and the probability corresponding to the dirtiness category of each pixel in the to-be-processed image taken by the camera with various reasons of dirtiness.

[0058] The dirtiness level categories may include: heavy dirtiness level category, medium dirtiness level category, light dirtiness level category and non-dirty background category (the present invention does not impose any specific limitation on this). When training the model, the dirtiness level category is manually defined according to the transparency of each pixel of the training sample image. The training samples include: a first training sample image taken when the camera is dirty with water droplets, a second training sample image taken when the camera is dirty with fog, and a third training sample image taken when the camera is dirty with sludge (it may also include training sample images taken under other types of dirt, and the embodiment of the present invention does not specifically limit the above-mentioned dirt types). After obtaining the above-mentioned three types of training sample images, the dirt level category of each pixel point is manually labeled according to the transparency of each pixel point in the training sample image (that is, the dirt level category label of each pixel point is obtained), and the scene category is labeled for each training sample image at the same time. The above-mentioned labeled training samples are input into the multi-task neural network model for training. The dirt level segmentation module of the model can obtain the probability that each pixel point in the training sample image belongs to the above-mentioned four dirt level categories, and then the dirt level category corresponding to the maximum probability is used as the dirt level category of the corresponding pixel point. In this way, the prediction result of the dirt level category of each pixel point and the prediction result of the probability corresponding to the dirt level category are obtained; the scene classification module of the model can obtain the prediction result of the scene category corresponding to the training sample image.

[0059] The loss function used in the training process is Loss = loss seg +0.3*loss cls , where loss seg Indicates the loss corresponding to the dirtiness segmentation, losscls Represents the loss corresponding to scene classification. Specifically, Among them, loss seg (y) represents the loss corresponding to the dirtiness level segmentation, y i represents the true value of the segmentation (i.e. the true value of the annotation - the true value of the dirtiness category), y′ i Represents the predicted value of the segmentation (i.e., the prediction result - the prediction result of the dirtiness category); Among them, loss cls (y) represents the loss corresponding to scene classification, y i Represents the true value of the classification, y′ i Represents the predicted value of the classification.

[0060] Figure 2 The schematic diagram of the structure of the multi-task neural network model of the present invention is shown in FIG. From the figure, it can be seen that the feature extraction module, the dirtiness segmentation module and the scene classification module all include a residual network layer. The residual network layer in the present invention includes channel shuffle fusion. Figure 3 The schematic diagram of the structure of the residual network layer of the scene classification module shown in the figure shows that, compared with the traditional residual network layer, the residual network layer of the present invention adds channel shuffle fusion, thereby improving the feature extraction capability of the network.

[0061] In the embodiment of the present invention, the above scene classification results include any of the following: daytime scene, night scene, underground garage scene, backlit scene. Of course, the embodiment of the present invention does not specifically limit the above scene classification results.

[0062] Step S106, determining a target dirt classification threshold corresponding to the scene classification result in a preset dirt classification threshold table, and optimizing the dirt level segmentation result based on the target dirt classification threshold to obtain an optimized dirt level segmentation map;

[0063] In an embodiment of the present invention, the dirt classification threshold table is a correspondence between scene categories and dirt classification thresholds, wherein the dirt classification threshold corresponding to night scenes, the dirt classification threshold corresponding to underground garage scenes, and the dirt classification threshold corresponding to backlight scenes are all greater than the dirt classification threshold corresponding to daytime scenes. The purpose is to prevent false dirt alarms in night scenes, underground garage scenes, and backlight scenes. For example, in a daytime scene, the dirt classification threshold can be 0.3. At this time, if the probability corresponding to the dirtiness category of a pixel point of the image to be processed in the daytime scene is 0.4 (greater than the dirtiness classification threshold of 0.3), it means that this pixel point is a dirty pixel point, and the corresponding dirtiness category output by the model is retained. In night scenes, underground garage scenes and backlight scenes, due to the influence of light on the image quality of the image to be processed, if the probability corresponding to the dirtiness category of a pixel point of the image to be processed in the night scene is also 0.4, although it is greater than the above-mentioned dirtiness classification threshold of 0.3, this pixel point is not a dirty pixel point (not because of the poor transparency of the pixel point caused by dirt). If it is judged as a dirty pixel point again, a false alarm will occur. Therefore, for night scenes, underground garage scenes and backlight scenes, the dirtiness classification threshold is larger than the dirtiness classification threshold corresponding to the daytime scene.

[0064] Specifically, the dirt classification threshold corresponding to the daytime scene can be 0.3, the dirt classification threshold corresponding to the night scene can be 0.75, the dirt classification threshold corresponding to the underground garage scene can be 0.7, and the dirt classification threshold corresponding to the backlight scene can be 0.8. The above dirt classification thresholds are obtained through statistics of a large number of experiments.

[0065] Step S108, performing binary morphological segmentation on the optimized dirt level segmentation map to obtain a dirt region map contained in the optimized dirt level segmentation map;

[0066] The above-mentioned optimized dirtiness level segmentation map has distinguished various dirtiness level categories, and then, binary morphological segmentation (segmentation) is performed on it to segment the dirtiness area map, where the dirty area map is a connected area composed of various dirtiness level categories except the non-dirty background category.

[0067] Step S110, counting the number of dirty pixels contained in each dirty area map, and determining whether the camera to be detected is dirty based on the number of dirty pixels.

[0068] The process is described in detail below and will not be repeated here.

[0069] In an embodiment of the present invention, a method for detecting camera dirtiness is provided, comprising: obtaining an image to be processed taken by a camera to be detected; performing dirtiness degree segmentation processing and scene classification processing on the image to be processed through a multi-task neural network model to obtain a dirtiness degree segmentation result and a scene classification result, wherein the dirtiness degree segmentation result comprises: a dirtiness degree category of each pixel point in the image to be processed and a probability corresponding to the dirtiness degree category; determining a target dirtiness classification threshold corresponding to the scene classification result in a preset dirtiness classification threshold table, and optimizing the dirtiness degree segmentation result based on the target dirtiness classification threshold to obtain an optimized dirtiness degree segmentation map; performing binary morphological segmentation on the optimized dirtiness degree segmentation map to obtain a dirtiness region map contained in the optimized dirtiness degree segmentation map; counting the number of dirty pixels contained in each dirty region map, and determining whether the camera to be detected is dirty based on the number of dirty pixels. From the above description, it can be seen that the camera dirtiness detection method of the present invention implements the dirtiness degree segmentation and scene classification of the image to be processed through a multi-task neural network model, and also optimizes the dirtiness degree segmentation result through the target dirtiness classification threshold value corresponding to the scene classification result, so that the dirty area in the optimized dirtiness degree segmentation map is more accurate, and then the result of whether the camera to be detected is dirty based on the above accurate dirty area is also more accurate. The multi-task neural network model is used in the above process. The multi-task neural network model has good robustness when dealing with the dirtiness degree segmentation of the image to be processed taken in various scenes. In addition, it is also considered to use the target dirtiness classification threshold value corresponding to the scene classification result to further optimize the dirtiness degree segmentation result, so that the final detection result is more accurate, alleviating the technical problems of inaccurate detection and poor robustness of the existing camera dirtiness detection method.

[0070] The above content briefly introduces the data annotation method of the present invention, and the specific contents involved therein are described in detail below.

[0071] In an optional embodiment of the present invention, the above step S106, optimizing the dirt level segmentation result based on the target dirt classification threshold, specifically includes the following steps:

[0072] (1) Determine whether the probability corresponding to the dirtiness category of each pixel is greater than the target dirtiness classification threshold;

[0073] (2) if the probability corresponding to the dirtiness level category of the first target pixel point exists is not greater than the target dirtiness classification threshold, then the dirtiness level category of the first target pixel point is updated to the non-dirty background category;

[0074] The first target pixel point refers to a pixel point whose probability is not greater than the target dirt classification threshold among all the pixel points.

[0075] (3) If the probability corresponding to the dirtiness level category of the second target pixel point exists and is greater than the target dirtiness classification threshold, the dirtiness level category of the second target pixel point is not updated, thereby obtaining an optimized dirtiness level segmentation result, and the optimized dirtiness level segmentation result is visualized to obtain an optimized dirtiness level segmentation map.

[0076] The second target pixel point refers to a pixel point whose probability is greater than the target dirt classification threshold among all the pixels.

[0077] In an optional embodiment of the present invention, the above step S110, counting the number of dirty pixels contained in each dirty region map, specifically includes the following steps:

[0078] (1) Count the total number of pixels corresponding to each dirtiness level category in each dirtiness area map;

[0079] (2) Performing weighted calculation on the total number of pixels corresponding to each dirtiness category and the preset weight corresponding to each dirtiness category to obtain the number of dirty pixels contained in each dirtiness region map.

[0080] For example, if in a dirty area map, the total number of pixels corresponding to the heavy dirtiness category is 20, the total number of pixels corresponding to the medium dirtiness category is 10, and the total number of pixels corresponding to the light dirtiness category is 5, and the preset weight corresponding to the heavy dirtiness category is 1, the preset weight corresponding to the medium dirtiness category is 0.8, and the preset weight corresponding to the light dirtiness category is 0.5, then the number of dirty pixels contained in the dirty area map can be obtained as: 20*1+10*0.8+5*0.5=30.5.

[0081] The embodiment of the present invention does not impose any specific restrictions on the preset weights corresponding to each of the above-mentioned dirtiness categories, and may also be other values. The above-mentioned preset weights are obtained by the inventor through a large number of experimental statistics.

[0082] In an optional embodiment of the present invention, the above step S110, determining whether the camera to be detected is dirty based on the number of dirty pixels, specifically includes the following steps:

[0083] (1) determining whether there is at least one dirty pixel number among the dirty pixel numbers that is greater than a preset dirty pixel number threshold;

[0084] (2) If there is at least one dirty pixel whose number is greater than a preset dirty pixel number threshold, it is determined that the camera to be detected is dirty, that is, the perception result of the camera is affected, and a dirt alarm is issued;

[0085] Specifically, for a 128*128 image to be processed, the preset dirty pixel count threshold may be 50, which is also determined by the inventor through a large number of experiments and is not specifically limited in the embodiments of the present invention. It should be noted that for images to be processed of different sizes, the preset dirty pixel count threshold is also different.

[0086] (3) If the number of non-dirty pixels is greater than a preset dirty pixel number threshold, it is determined that the camera to be detected is not dirty.

[0087] The inventors have considered that in some special cases, such as in an image to be processed, the number of dirty pixels in each individual dirty area map is not greater than the preset dirty pixel number threshold, that is, it will eventually be determined that the camera is not dirty, but the image to be processed contains many small dirty area maps, and many small dirty area maps will eventually affect the perception result, but they cannot be identified by the camera being dirty. Therefore, in order to be able to identify this situation, the inventors have also designed a camera occlusion judgment strategy to determine multiple small dirty area maps. In an optional embodiment of the present invention, the method also includes:

[0088] (1) Counting the total number of pixels contained in all dirty area maps, and calculating the ratio of the total number of pixels to the total number of pixels of the image to be processed;

[0089] The above-mentioned all dirty area maps are all dirty area maps corresponding to an image to be processed, and the above-mentioned total number of pixels is the sum of the number of all pixel points in all dirty area maps. There is no need to distinguish the categories of dirtiness and no weighted calculation is performed.

[0090] (2) If the ratio is greater than a preset threshold, it is determined that the camera to be detected is blocked and an occlusion alarm is issued.

[0091] Specifically, the preset threshold may be 60%, and the embodiment of the present invention does not impose any specific limitation on the preset threshold.

[0092] In order to better understand the camera dirt detection method of the present invention, some specific embodiments are described below:

[0093] The experiment was conducted with an image to be processed (as shown in FIG4(a), with a size of 128*128) taken by a camera (i.e., the camera to be detected) with sludge in a daytime scene. The above-mentioned image to be processed was input into the multi-task neural network model, and the dirtiness category of each pixel in the image to be processed, the probability corresponding to the dirtiness category and the scene classification result - the daytime scene category were output. Then, the target dirtiness classification threshold corresponding to the daytime scene category was determined to be 0.3, and the dirtiness segmentation result was optimized based on the target dirtiness classification threshold of 0.3 (the dirtiness category of the pixel points with a probability greater than the target dirtiness classification threshold of 0.3 remained unchanged, and the dirtiness category of the pixel points with a probability not greater than the target dirtiness classification threshold of 0.3 was updated to the non-dirty background category), and the optimized dirtiness segmentation map was obtained, as shown in FIG4(b), where the black area is the non-dirty background category, and the rest of the area is the dirty area consisting of the heavy dirtiness category and / or the medium dirtiness category and / or the light dirtiness category. b) Perform binary morphological segmentation to obtain the two dirty area maps contained in Figure 4(b), as shown in Figure 4(c). Count the number of dirty pixels contained in each dirty area map in Figure 4(c) (specifically, the total number of pixels corresponding to each dirtiness category and the weighted sum of the preset weights corresponding to each dirtiness category) to obtain two dirty pixel numbers, and determine whether there is at least one dirty pixel number greater than the preset dirty pixel number threshold of 50. If so, it is determined that the camera is dirty and a dirt alarm is issued (in this experiment, at least one dirty pixel number is greater than the preset dirty pixel number threshold of 50, and a dirt alarm will be issued). If not, it is determined that the camera is not dirty, and the total number of pixels contained in the two dirty area maps is further counted, and the ratio of the total number of pixels to the total number of pixels of the image to be processed is calculated; if the ratio is greater than the preset threshold of 60%, it is determined that the camera to be detected is occluded, and an occlusion alarm is issued (the ratio obtained in this experiment is not greater than the preset threshold of 60%, and no occlusion alarm is issued).

[0094] After a dirt alarm or an occlusion alarm is issued, the dirt or occlusion of the camera can be removed manually or automatically, thereby ensuring that the camera captures accurate, clear and stable images.

[0095] The camera dirtiness detection method of the present invention implements dirtiness degree segmentation and scene classification of the image to be processed through a multi-task neural network model, and optimizes the dirtiness degree segmentation result through a target dirtiness classification threshold value corresponding to the scene classification result, so that the dirty area in the optimized dirtiness degree segmentation map is more accurate, and then the result of whether the camera to be detected is dirty based on the above accurate dirty area is also more accurate. The multi-task neural network model is used in the above process. The multi-task neural network model has good robustness when dealing with the dirtiness degree segmentation of the image to be processed taken in various scenes. In addition, it is considered to further optimize the dirtiness degree segmentation result by using the target dirtiness classification threshold value corresponding to the scene classification result, so that the final detection result is more accurate.

[0096] Embodiment 2:

[0097] An embodiment of the present invention further provides a camera dirt detection device, which is mainly used to execute the camera dirt detection method provided in the first embodiment of the present invention. The camera dirt detection device provided in the embodiment of the present invention is specifically introduced below.

[0098] Figure 5 is a schematic diagram of a camera dirt detection device according to an embodiment of the present invention, such as Figure 5 As shown, the device mainly includes: an acquisition unit 10, a processing unit 20, an optimization unit 30, a binary morphology segmentation unit 40 and a determination unit 50, wherein:

[0099] An acquisition unit, used for acquiring an image to be processed taken by a camera to be detected;

[0100] A processing unit is used to perform dirtiness level segmentation processing and scene classification processing on the image to be processed through a multi-task neural network model to obtain a dirtiness level segmentation result and a scene classification result, wherein the dirtiness level segmentation result includes: a dirtiness level category of each pixel point in the image to be processed and a probability corresponding to the dirtiness level category;

[0101] An optimization unit, used to determine a target dirt classification threshold corresponding to the scene classification result in a preset dirt classification threshold table, and optimize the dirt level segmentation result based on the target dirt classification threshold to obtain an optimized dirt level segmentation map;

[0102] A binary morphological segmentation unit is used to perform binary morphological segmentation on the optimized dirtiness degree segmentation map to obtain a dirtiness region map contained in the optimized dirtiness degree segmentation map;

[0103] The determination unit is used to count the number of dirty pixels contained in each dirty area map, and determine whether the camera to be detected is dirty based on the number of dirty pixels.

[0104] In an embodiment of the present invention, a camera dirtiness detection device is provided, comprising: obtaining an image to be processed taken by a camera to be detected; performing dirtiness degree segmentation processing and scene classification processing on the image to be processed through a multi-task neural network model to obtain a dirtiness degree segmentation result and a scene classification result, wherein the dirtiness degree segmentation result comprises: a dirtiness degree category of each pixel point in the image to be processed and a probability corresponding to the dirtiness degree category; determining a target dirtiness classification threshold value corresponding to the scene classification result in a preset dirtiness classification threshold value table, and optimizing the dirtiness degree segmentation result based on the target dirtiness classification threshold value to obtain an optimized dirtiness degree segmentation map; performing binary morphological segmentation on the optimized dirtiness degree segmentation map to obtain a dirtiness region map contained in the optimized dirtiness degree segmentation map; counting the number of dirty pixels contained in each dirty region map, and determining whether the camera to be detected is dirty based on the number of dirty pixels. From the above description, it can be seen that the camera dirtiness detection device of the present invention realizes the dirtiness level segmentation and scene classification of the image to be processed through a multi-task neural network model, and also optimizes the dirtiness level segmentation result through the target dirtiness classification threshold value corresponding to the scene classification result, so that the dirty area in the optimized dirtiness level segmentation map is more accurate, and then the result of whether the camera to be detected is dirty based on the above accurate dirty area is also more accurate. The multi-task neural network model is adopted in the above process. The multi-task neural network model has good robustness when dealing with the dirtiness level segmentation of the image to be processed taken in various scenes. In addition, it is also considered to use the target dirtiness classification threshold value corresponding to the scene classification result to further optimize the dirtiness level segmentation result, so that the final detection result is more accurate, which alleviates the technical problems of inaccurate detection and poor robustness of the existing camera dirtiness detection method.

[0105] Optionally, the optimization unit is also used to: determine whether the probability corresponding to the dirtiness category of each pixel point is greater than the target dirtiness classification threshold; if the probability corresponding to the dirtiness category of the first target pixel point is not greater than the target dirtiness classification threshold, then update the dirtiness category of the first target pixel point to a non-dirty background category; if the probability corresponding to the dirtiness category of the second target pixel point is greater than the target dirtiness classification threshold, then do not update the dirtiness category of the second target pixel point, thereby obtaining an optimized dirtiness segmentation result, and visualize the optimized dirtiness segmentation result to obtain an optimized dirtiness segmentation map.

[0106] Optionally, the determination unit is also used to: count the total number of pixels corresponding to each dirtiness category in each dirtiness area map; perform weighted calculation on the total number of pixels corresponding to each dirtiness category and the preset weight corresponding to each dirtiness category to obtain the number of dirty pixels contained in each dirtiness area map.

[0107] Optionally, the determination unit is also used to: determine whether there is at least one dirty pixel number among the dirty pixels that is greater than a preset dirty pixel number threshold; if there is at least one dirty pixel number that is greater than the preset dirty pixel number threshold, determine that the camera to be detected is dirty, and issue a dirt alarm; if there is no dirty pixel number that is greater than the preset dirty pixel number threshold, determine that the camera to be detected is not dirty.

[0108] Optionally, the scene classification results include any one of the following: daytime scene, night scene, underground garage scene, backlight scene, wherein, in the dirt classification threshold table, the dirt classification threshold corresponding to the night scene, the dirt classification threshold corresponding to the underground garage scene, and the dirt classification threshold corresponding to the backlight scene are all greater than the dirt classification threshold corresponding to the daytime scene.

[0109] Optionally, the device is also used to: count the total number of pixels contained in all dirty area maps, and calculate the ratio of the total number of pixels to the total number of pixels of the image to be processed; if the ratio is greater than a preset threshold, it is determined that there is occlusion in the camera to be detected, and an occlusion alarm is issued.

[0110] Optionally, the multi-task neural network model is pre-based on the loss function Loss = loss seg +0.3*loss cls The trained neural network model includes a feature extraction module, a dirtiness segmentation module and a scene classification module respectively connected to the feature extraction module, wherein the loss seg Indicates the loss corresponding to the dirtiness segmentation, loss cls It represents the loss corresponding to scene classification. The feature extraction module, dirtiness segmentation module and scene classification module all contain residual network layers, and the residual network layer contains channel shuffle fusion.

[0111] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0112] like Figure 6As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the above-mentioned camera dirt detection method.

[0113] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memories and processors, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, the above-mentioned camera dirt detection method can be executed.

[0114] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0115] Corresponding to the above-mentioned camera dirtiness detection method, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned camera dirtiness detection method.

[0116] The camera dirt detection device provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0117] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0118] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0121] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or partly contribute to the prior art or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the vehicle marking method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0122] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0123] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting camera dirtiness, characterized in that: include: Obtain the image to be processed taken by the camera to be detected; Performing dirtiness level segmentation processing and scene classification processing on the image to be processed by a multi-task neural network model to obtain a dirtiness level segmentation result and a scene classification result, wherein the dirtiness level segmentation result includes: the dirtiness level category of each pixel point in the image to be processed and the probability corresponding to the dirtiness level category; Determining a target dirt classification threshold corresponding to the scene classification result in a preset dirt classification threshold table, and optimizing the dirt degree segmentation result based on the target dirt classification threshold to obtain an optimized dirt degree segmentation map; Performing binary morphological segmentation on the optimized dirtiness level segmentation map to obtain a dirtiness region map contained in the optimized dirtiness level segmentation map; The number of dirty pixels contained in each of the dirty area maps is counted, and whether the camera to be detected is dirty is determined based on the number of dirty pixels.

2. The method according to claim 1, characterized in that Optimizing the dirtiness level segmentation result based on the target dirtiness classification threshold includes: Determine whether the probability corresponding to the dirtiness category of each pixel is greater than the target dirtiness classification threshold; If there is a probability corresponding to the dirtiness level category of the first target pixel point that is not greater than the target dirtiness classification threshold, updating the dirtiness level category of the first target pixel point to a non-dirty background category; If the probability corresponding to the dirtiness level category of the second target pixel point exists and is greater than the target dirtiness classification threshold, the dirtiness level category of the second target pixel point is not updated, thereby obtaining an optimized dirtiness level segmentation result, and the optimized dirtiness level segmentation result is visualized to obtain the optimized dirtiness level segmentation map.

3. The method according to claim 1, characterized in that: Counting the number of dirty pixels contained in each of the dirty area maps includes: Counting the total number of pixels corresponding to each dirtiness level category in each of the dirtiness area maps; The total number of pixels corresponding to each dirt level category and the preset weight corresponding to each dirt level category are weightedly calculated to obtain the number of dirty pixels contained in each of the dirt area maps.

4. The method according to claim 1, characterized in that: Determining whether the camera to be detected is dirty based on the number of dirty pixels includes: Determine whether there is at least one dirty pixel number among the dirty pixel numbers that is greater than a preset dirty pixel number threshold; If there is at least one dirty pixel number greater than the preset dirty pixel number threshold, it is determined that the camera to be detected is dirty, and a dirt alarm is issued; If the number of non-dirty pixels is greater than the preset dirty pixel number threshold, it is determined that the camera to be detected is not dirty.

5. The method according to claim 1, characterized in that: The scene classification results include any one of the following: daytime scene, nighttime scene, underground garage scene, and backlighting scene, wherein in the dirt classification threshold table, the dirt classification threshold corresponding to the nighttime scene, the dirt classification threshold corresponding to the underground garage scene, and the dirt classification threshold corresponding to the backlighting scene are all greater than the dirt classification threshold corresponding to the daytime scene.

6. The method according to claim 1, characterized in that The method further comprises: Counting the total number of pixels contained in all the dirty area maps, and calculating the ratio of the total number of pixels to the total number of pixels of the image to be processed; If the ratio is greater than a preset threshold, it is determined that the camera to be detected is blocked and an occlusion alarm is issued.

7. The method according to claim 1, characterized in that The multi-task neural network model is pre-based on the loss function Loss=loss seg +0.3*loss cls The multi-task neural network model includes a feature extraction module, a dirtiness segmentation module and a scene classification module respectively connected to the feature extraction module, wherein the loss seg Indicates the loss corresponding to the dirtiness segmentation, loss cls It represents the loss corresponding to the scene classification, the feature extraction module, the dirtiness segmentation module and the scene classification module all include a residual network layer, and the residual network layer includes channel shuffle fusion.

8. A camera dirt detection device, characterized in that: include: An acquisition unit, used for acquiring an image to be processed taken by a camera to be detected; A processing unit, configured to perform dirtiness level segmentation processing and scene classification processing on the image to be processed through a multi-task neural network model to obtain a dirtiness level segmentation result and a scene classification result, wherein the dirtiness level segmentation result includes: a dirtiness level category of each pixel point in the image to be processed and a probability corresponding to the dirtiness level category; an optimization unit, configured to determine a target dirt classification threshold corresponding to the scene classification result in a preset dirt classification threshold table, and optimize the dirt degree segmentation result based on the target dirt classification threshold to obtain an optimized dirt degree segmentation map; A binary morphological segmentation unit, used for performing binary morphological segmentation on the optimized dirtiness level segmentation map to obtain a dirtiness region map contained in the optimized dirtiness level segmentation map; The determination unit is used to count the number of dirty pixels contained in each of the dirty area maps, and determine whether the camera to be detected is dirty based on the number of dirty pixels.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.

Images