A method and device for detecting dirt on a vehicle-mounted camera

Abstract

The application provides a kind of detection method and device for dirty of vehicle-mounted camera, and continuous multiple frames of image to be processed are obtained by acquisition;Each image to be processed is converted to obtain multiple gray scale images, and the region of interest in the gray scale image is cropped to obtain multiple target images;The average information entropy of each target image is calculated;The gray scale evaluation value corresponding to each target image is evaluated;For each target image, the adaptive gray scale threshold is calculated based on the corresponding average information entropy and gray scale evaluation value, and the average gray scale value of the contour is calculated;When the average gray scale value of the contour is less than the adaptive gray scale threshold, and less than the average gray scale evaluation value obtained based on each gray scale evaluation value, mark the target image as a dirty image;When the number of target images continuously marked as dirty images exceeds the preset number threshold, it is determined that the vehicle-mounted camera is in a dirty state.In this scheme, the scene is not sensitive to the dirty detection, which is suitable for various scenes.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to a method and apparatus for detecting dirt in vehicle-mounted cameras. Background Technology

[0002] When intelligent vehicles drive on muddy roads, mud can easily splash onto the camera surface. Contaminated cameras capture images with reduced clarity, failing to accurately reflect real road conditions. This erroneous information, transmitted to the control unit, can lead to incorrect judgments and potentially cause traffic accidents. Therefore, detecting the cleanliness of camera surfaces is crucial for the safe driving of intelligent vehicles.

[0003] In existing technologies, dirt detection is performed using traditional algorithms, which are typically based on image processing techniques such as edge detection, frequency analysis, and information entropy to evaluate image quality. These methods can be further categorized into full-reference methods, half-reference methods, and no-reference methods. No-reference methods, which do not rely on a reference image, directly extract feature information (such as edge information, frequency domain energy, and information entropy) from the target image to determine the degree of blurriness. No-reference methods are suitable for vehicle-mounted cameras because suitable reference images are difficult to find in practical applications. In patent application number CN202110918666.5, the gradient method within the no-reference method is used, which is sensitive to the gradient of dirt and prone to false detections in scenes with unclear dirt boundaries or blurriness. In patent application number CN202111671878.4, the information entropy method within the no-reference method is mainly used, taking advantage of the fixed driving path and monotonous driving environment of mining trucks. However, it performs poorly in complex driving scenarios such as non-fixed routes and varying weather conditions. Furthermore, existing no-reference methods, when the image size is the same, cannot fully meet the real-time requirements of some algorithms, especially in highly dynamic driving scenarios.

[0004] In summary, existing methods for detecting dirt in vehicle cameras have limited applicability and cannot meet the needs of multiple scenarios. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for detecting dirt in vehicle-mounted cameras, so as to achieve dirt detection that is imperceptible to the scene and applicable to a variety of scenarios.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses a method for detecting dirt in a vehicle-mounted camera, the method comprising:

[0008] The vehicle-mounted camera under test acquires multiple consecutive frames of images to be processed.

[0009] Each of the images to be processed is converted to obtain multiple grayscale images;

[0010] For each grayscale image, the region of interest in the grayscale image is cropped to obtain multiple target images;

[0011] The average information entropy of each target image is calculated.

[0012] The grayscale evaluation value corresponding to each target image is obtained by using a fitting function;

[0013] For each target image, a corresponding adaptive grayscale threshold is calculated based on the corresponding average information entropy and the corresponding grayscale evaluation value, and the average grayscale value of the contour is calculated for the contour in the target image.

[0014] When the average gray value of the contour is less than the adaptive gray threshold and less than the average gray evaluation value obtained based on each of the gray evaluation values, the target image is marked as a dirty image.

[0015] When the number of target images consecutively marked as dirty images exceeds a preset threshold, the vehicle camera is determined to be in a dirty state.

[0016] Preferably, the step of converting each of the images to be processed to obtain multiple grayscale images includes:

[0017] If each of the images to be processed is a three-channel image, then the format of each of the images to be processed is converted into a single-channel image to obtain multiple grayscale images.

[0018] If the format of each of the images to be processed is NV12, then the Y component of each of the images to be processed is used as a grayscale image to obtain multiple grayscale images.

[0019] Preferably, the calculation of the average information entropy of each target image includes:

[0020] For each target image, it is divided into multiple small regions, the information entropy of each small region is calculated, and the average information entropy of each small region is calculated to obtain the average information entropy of the target image.

[0021] Preferably, the step of using a fitting function to evaluate the grayscale evaluation value corresponding to each target image includes:

[0022] Numerical features of each target image are obtained, and feature curves of numerical features versus time are plotted.

[0023] Based on the aforementioned feature curves, preliminary piecewise fitting is performed to obtain various piecewise point configurations;

[0024] The goodness of fit of each segment point configuration is calculated based on the Bayesian information criterion, and the segment point configuration corresponding to the smallest goodness of fit is determined as the optimal segment point configuration.

[0025] From the fitting models corresponding to each of the segmentation points, determine the optimal fitting model corresponding to the optimal segmentation point configuration;

[0026] Based on the optimal fitting model and the optimal segmentation point configuration, the grayscale evaluation value corresponding to each target image is obtained.

[0027] Preferably, calculating the average gray value of the contours in the target image includes:

[0028] Edge information is obtained by performing edge detection on the target image using the Sobel operator;

[0029] The target image after edge detection is binarized, and noise is removed by erosion to enhance the edge information and obtain an edge image;

[0030] The target edge image is obtained by intersecting the edge image and the historical edge image; the historical edge image includes edge images obtained from the target image in the past.

[0031] Extract the target contour with an area greater than the area threshold from the target edge image;

[0032] If the number of target contours is greater than the number threshold, then the target contours are sorted in descending order of area, and the target contours sorted after the number threshold are removed.

[0033] For each of the remaining target contours, a contour mask is drawn, and multiple sampling points are obtained based on the contour mask. The average gray value of each sampling point is calculated.

[0034] The smallest mean gray value is used as the average gray value of the contour.

[0035] A second aspect of this invention discloses a device for detecting dirt in a vehicle-mounted camera, the device comprising:

[0036] The acquisition unit is used to acquire multiple consecutive frames of images to be processed using the vehicle-mounted camera to be detected.

[0037] A conversion unit is used to convert each of the images to be processed to obtain multiple grayscale images;

[0038] The cropping unit is used to crop the region of interest in each grayscale image to obtain multiple target images.

[0039] The first calculation unit is used to calculate the average information entropy of each target image;

[0040] An evaluation unit is used to evaluate the grayscale evaluation value corresponding to each target image using a fitting function;

[0041] The second calculation unit is used to calculate the corresponding adaptive gray threshold for each target image based on the corresponding average information entropy and the corresponding gray level evaluation value, and to calculate the average gray level value of the contour for the contour in the target image.

[0042] A labeling unit is used to label the target image as a dirty image when the average gray value of the contour is less than the adaptive gray threshold and less than the average gray evaluation value obtained based on each of the gray evaluation values.

[0043] The determination unit is used to determine that the vehicle camera is dirty when the number of target images consecutively marked as dirty images exceeds a preset number threshold.

[0044] Preferably, the conversion unit is specifically used for:

[0045] If each of the images to be processed is a three-channel image, then the format of each of the images to be processed is converted into a single-channel image to obtain multiple grayscale images.

[0046] If the format of each of the images to be processed is NV12, then the Y component of each of the images to be processed is used as a grayscale image to obtain multiple grayscale images.

[0047] Preferably, the first computing unit is specifically used for:

[0048] For each target image, it is divided into multiple small regions, the information entropy of each small region is calculated, and the average information entropy of each small region is calculated to obtain the average information entropy of the target image.

[0049] Preferably, the evaluation unit is specifically used for:

[0050] Numerical features of each target image are obtained, and feature curves of numerical features versus time are plotted.

[0051] Based on the aforementioned feature curves, preliminary piecewise fitting is performed to obtain various piecewise point configurations;

[0052] The goodness of fit of each segment point configuration is calculated based on the Bayesian information criterion, and the segment point configuration corresponding to the smallest goodness of fit is determined as the optimal segment point configuration.

[0053] From the fitting models corresponding to each of the segmentation points, determine the optimal fitting model corresponding to the optimal segmentation point configuration;

[0054] Based on the optimal fitting model and the optimal segmentation point configuration, the grayscale evaluation value corresponding to each target image is obtained.

[0055] Preferably, the second calculation unit, used to calculate the average gray value of the contours in the target image, is specifically used for:

[0056] Edge information is obtained by performing edge detection on the target image using the Sobel operator;

[0057] The target image after edge detection is binarized, and noise is removed by erosion to enhance the edge information and obtain an edge image;

[0058] The target edge image is obtained by intersecting the edge image and the historical edge image; the historical edge image includes edge images obtained from the target image in the past.

[0059] Extract the target contour with an area greater than the area threshold from the target edge image;

[0060] If the number of target contours is greater than the number threshold, then the target contours are sorted in descending order of area, and the target contours sorted after the number threshold are removed.

[0061] For each of the remaining target contours, a contour mask is drawn, and multiple sampling points are obtained based on the contour mask. The average gray value of each sampling point is calculated.

[0062] The smallest mean gray value is used as the average gray value of the contour.

[0063] A method and apparatus for detecting dirt in a vehicle-mounted camera, based on the above embodiments of the present invention, includes: acquiring multiple consecutive frames of images to be processed using a vehicle-mounted camera to be detected; converting each image to be processed to obtain multiple grayscale images; cropping the region of interest in each grayscale image to obtain multiple target images; calculating the average information entropy of each target image; evaluating the grayscale evaluation value corresponding to each target image using a fitting function; for each target image, calculating a corresponding adaptive grayscale threshold based on the corresponding average information entropy and the corresponding grayscale evaluation value, and calculating the average grayscale value of the contour in the target image; when the average grayscale value of the contour is less than the adaptive grayscale threshold and less than the average grayscale evaluation value obtained based on each grayscale evaluation value, marking the target image as a dirty image; when the number of target images consecutively marked as dirty images exceeds a preset number threshold, determining that the vehicle-mounted camera is in a dirty state. This solution integrates information entropy and image grayscale, solving the problems that single information entropy is only applicable to fixed scenes, has poor robustness to different lighting conditions, and lacks real-time performance. This achieves scene-insensitive dirt detection, making it suitable for a variety of scenarios. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0065] Figure 1 This is a flowchart of a method for detecting dirt in a vehicle-mounted camera, as disclosed in an embodiment of the present invention.

[0066] Figure 2 This is a flowchart of a grayscale evaluation method disclosed in an embodiment of the present invention;

[0067] Figure 3 This is a flowchart for calculating the average gray value of a contour, as disclosed in an embodiment of the present invention;

[0068] Figure 4 This is a structural diagram of a vehicle-mounted camera dirt detection device disclosed in an embodiment of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0071] As can be seen from the background technology, existing methods for detecting dirt in vehicle cameras have limited applicability and cannot meet the needs of multiple scenarios.

[0072] Therefore, this invention discloses a method and apparatus for detecting dirt in a vehicle-mounted camera. In this solution, information entropy and image grayscale are integrated, which solves the problems that single information entropy is only applicable to fixed scenes, has poor robustness to different lighting conditions, and lacks real-time performance. This achieves the goal of dirt detection being scene-insensitive and applicable to a variety of scenarios.

[0073] like Figure 1 The diagram shown is a flowchart of a method for detecting dirt in a vehicle-mounted camera according to an embodiment of the present invention, which mainly includes the following steps:

[0074] Step S101: Use the vehicle-mounted camera to be tested to acquire multiple consecutive frames of images to be processed.

[0075] Step S102: Convert each image to be processed to obtain multiple grayscale images.

[0076] In step S102, since the image to be processed is generally not single-channel, it needs to be converted into a single-channel grayscale image. The specific conversion process depends on the format of the image to be processed. The image to be processed usually uses three-channel image data with an RGB color space; there are also images to be processed represented in YUV format.

[0077] It should be noted that the YUV format is a color encoding method, which includes luminance (Y), blue color difference (U), and red color difference (V).

[0078] In the specific implementation of step S102, if the format of each image to be processed is a three-channel image, then the format of each image to be processed is converted into a single-channel image to obtain multiple grayscale images; if the format of each image to be processed is NV12 format, then the Y component of each image to be processed is used as a grayscale image to obtain multiple grayscale images.

[0079] Step S103: For each grayscale image, crop the region of interest in the grayscale image to obtain multiple target images.

[0080] In step S103, cropping refers to copying an image of a certain region to another memory location, with the aim of not altering the original data. Subsequent processing then applies only to the copied region of interest, which reduces computational load and minimizes interference from the vehicle's image viewpoint, as these vehicle areas are fixed relative to the camera.

[0081] The "vehicle region" refers to the parts of the vehicle itself that are identified and processed from images captured by the vehicle's onboard cameras. These include, for example, the front, sides, or any other fixed parts of the vehicle whose position relative to the camera remains constant. During image analysis, these parts can cause interference because they are constant across consecutive image frames and may obscure other dynamic objects or environmental features that need to be analyzed.

[0082] In the specific implementation of step S103, for each grayscale image, the region of interest (ROI) is cropped to obtain different ROIs, and each ROI is used as a target image.

[0083] Step S104: Calculate the average information entropy of each target image.

[0084] In the specific implementation of step S104, each target image is divided into multiple small regions, the information entropy of each small region is calculated, and the average information entropy of each small region is calculated to obtain the average information entropy of the target image.

[0085] Assuming the target image is ultimately divided into N×M small regions, the information entropy of each small region is calculated. The information entropy of the entire target image can be obtained by averaging the information entropies of these small regions. The following is the mathematical formula notation for this process:

[0086] Calculate the information entropy of each small region:

[0087]

[0088] Where H is the information entropy of the image, k is the pixel value, and p(k) is the frequency of occurrence of the pixel with pixel value k.

[0089] Calculate the average information entropy of the entire target image:

[0090]

[0091] Step S105: Use the fitting function to evaluate the grayscale evaluation value corresponding to each target image.

[0092] like Figure 2 The diagram shown is a flowchart of a grayscale evaluation method disclosed in an embodiment of the present invention, including the following steps:

[0093] Step S201: Obtain the numerical features of each target image and plot the feature curves of numerical features versus time.

[0094] In step S201, the numerical features of the target image can be average gray level, variance, etc.

[0095] It should be noted that each target image is composed of multi-frame data, which are continuous in time. Data features of different frames can be obtained from these continuous frames, and the feature curves of numerical features versus time can reflect the distribution of features.

[0096] Step S202: Perform preliminary piecewise fitting based on the characteristic curve to obtain various piecewise point configurations.

[0097] In step S202, the preliminary piecewise fitting involves segmenting the data based on the curve of numerical characteristics versus time. The purpose is to identify the trend of data change and preliminarily determine the segmentation points. These preliminary segmentation points can have multiple configurations, each representing a possible segmentation scheme.

[0098] Step S203: Calculate the goodness of fit for each segment point configuration based on the Bayesian information criterion, determine the segment point configuration corresponding to the minimum goodness of fit as the optimal segment point configuration, and determine the optimal fitting model corresponding to the optimal segment point configuration from the fitting models corresponding to each segment point configuration.

[0099] In step S203, the Bayesian Information Criterion (BIC) is used to evaluate the goodness of fit of these different segmentation point configurations, and the best model is selected. A smaller BIC value indicates lower model complexity and prediction error; therefore, among multiple candidate models, the model with the smallest BIC should be selected. In this way, the optimal segmentation is determined, i.e., the best segmentation point configuration is selected, to ensure the accuracy and simplicity of the final model.

[0100] Here, "model" refers to the mathematical model used to describe the data, such as a linear regression model. In piecewise linear regression, each piecewise point configuration can correspond to an independent linear model. BIC (Best in Segmented Regression) can help us evaluate the goodness of fit and complexity of different piecewise models, thereby selecting the optimal piecewise model.

[0101] Step S204: Based on the optimal fitting model and the optimal segmentation point configuration, evaluate the grayscale evaluation value corresponding to each target image.

[0102] Step S106: For each target image, the corresponding adaptive gray threshold is calculated by weighting based on the corresponding average information entropy and the corresponding gray level evaluation value, and the average gray level value of the contour is calculated for the contour in the target image.

[0103] In step S106, a weight can be assigned to each grayscale evaluation value, the weight being proportional to the average information entropy. Then, an adaptive grayscale threshold is calculated based on these weighted grayscale evaluation values. For example, a weighted average method can be used to calculate the adaptive grayscale threshold.

[0104] In this embodiment of the invention, the average information entropy and image data brightness (i.e., grayscale evaluation value) are integrated, which solves the problem that single information entropy is only applicable to fixed scenes.

[0105] like Figure 3 The diagram shows a flowchart for calculating the average gray value of a contour, as disclosed in an embodiment of the present invention, including the following steps:

[0106] Step S301: Use the Sobel operator to perform edge detection on the target image to obtain edge information.

[0107] In step S301, the Sobel operator is a first-order differential operator that can detect edge information in an image. The edge intensity and direction are obtained by calculating the gradients of the image in the x and y directions.

[0108] Step S302: Binarize the target image after edge detection and remove noise through erosion to enhance edge information and obtain an edge image.

[0109] In step S302, the erosion operation is a type of morphological operation.

[0110] Step S303: Perform intersection processing on the edge image and the historical edge image to obtain the target edge image.

[0111] Historical edge images include edge images obtained from target images in the past.

[0112] In step S303, the edge image detected in the current frame is accumulated with the edge images of all previous historical frames. This means that the edge information of each frame is saved and added to the existing set of historical edge images when a new edge detection result appears.

[0113] Intersection calculation: Calculate the intersection of these accumulated edge images. The intersection refers to edge regions that appear in all historical frames. These regions remain stable across multiple frames and are therefore more likely to be important, persistent edge features in the image.

[0114] In this way, the consistency and stability of edge information can be enhanced, because only those edges that appear in multiple consecutive frames will be retained, which can gradually narrow the filtering range and reduce the amount of computation.

[0115] Step S304: Extract the target contour with an area greater than the area threshold from the target edge image.

[0116] Step S305: If the number of target contours is greater than the number threshold, sort the target contours in descending order of area and remove the target contours that are sorted after the number threshold.

[0117] In step S305, the target contours are sorted in descending order of area. When the number of target contours is greater than the number threshold, the target contours that are ranked first and equal to the number threshold are selected for further processing. If the number of target contours is not greater than the number threshold, no processing is performed.

[0118] Step S306: For each of the remaining target contours, draw a contour mask, sample multiple sampling points based on the contour mask, and calculate the grayscale mean of each sampling point.

[0119] In step S306, a contour mask is drawn: First, a mask is drawn based on the target contour. This mask is a binary image of the same size as the original image, where the pixel value inside the contour is 1 (or any non-zero value), and the pixel value outside the contour is 0.

[0120] Image sampling: The original image is sampled using this mask. This means that only pixels with a value of 1 in the mask are considered, and pixels with a value of 0 are ignored.

[0121] Calculate the average grayscale value of each sampling point: Statistically analyze the grayscale values ​​of the sampled pixels and calculate the average of these grayscale values. This average is the average grayscale value of the target contour.

[0122] Step S307: Use the smallest grayscale mean as the average grayscale value of the contour.

[0123] In step S307, the mean gray values ​​corresponding to all target contours are compared. The minimum mean gray value is selected as the average gray value of the contour because this minimum mean gray value may indicate a dirty area, since dirty areas are usually darker than the surrounding areas.

[0124] In this embodiment of the invention, the comparison between grayscale averages is adopted, which effectively solves the problem that the centroid of the C-shaped contour is not inside the contour in the original scheme of comparing centroid grayscale values.

[0125] Step S107: When the average gray value of the contour is less than the adaptive gray value threshold and less than the average gray value obtained based on each gray value evaluation value, the target image is marked as a dirty image.

[0126] It should be noted that since a smaller gray value represents a lower brightness, when the average gray value of the contour is less than the adaptive gray value threshold and less than the average gray value obtained based on each gray value evaluation value, it indicates that there is a dark area in the target image with an area larger than the area threshold. In this case, it is marked as a dirty image.

[0127] Step S108: When the number of target images consecutively marked as dirty images exceeds a preset threshold, the vehicle camera is determined to be in a dirty state.

[0128] In step S108, to avoid misjudgment, the current vehicle camera is considered to be dirty only when the number of images marked as dirty in the target image obtained from multiple consecutive frames of images to be processed over a period of time meets a certain threshold; otherwise, it is not dirty.

[0129] A method for detecting dirt in a vehicle-mounted camera, based on the above embodiments of the present invention, involves acquiring multiple consecutive frames of images to be processed using the vehicle-mounted camera to be detected; converting each image to obtain multiple grayscale images; cropping the region of interest in each grayscale image to obtain multiple target images; calculating the average information entropy of each target image; evaluating the grayscale evaluation value corresponding to each target image using a fitting function; for each target image, calculating a corresponding adaptive grayscale threshold based on the corresponding average information entropy and the corresponding grayscale evaluation value, and calculating the average grayscale value of the contour in the target image; when the average grayscale value of the contour is less than the adaptive grayscale threshold and less than the average grayscale evaluation value obtained based on each grayscale evaluation value, the target image is marked as a dirty image; when the number of consecutively marked target images as dirty images exceeds a preset number threshold, the vehicle-mounted camera is determined to be in a dirty state. This solution integrates information entropy and image grayscale, solving the problems that single information entropy is only applicable to fixed scenes, has poor robustness to different lighting conditions, and lacks real-time performance. This enables dirt detection to be scene-insensitive and is applicable to scenarios including but not limited to urban roads, closed parks, parking lots, and highways.

[0130] Corresponding to the method for detecting dirt in a vehicle-mounted camera disclosed in the above embodiments of the present invention, such as Figure 4 The diagram shown is a structural diagram of a vehicle camera dirt detection device disclosed in an embodiment of the present invention. The device includes: a data acquisition unit 401, a conversion unit 402, a cropping unit 403, a first calculation unit 404, an evaluation unit 405, a second calculation unit 406, a marking unit 407, and a judgment unit 408.

[0131] The acquisition unit 401 is used to acquire multiple consecutive frames of images to be processed using the vehicle-mounted camera to be detected.

[0132] The conversion unit 402 is used to convert each image to be processed to obtain multiple grayscale images.

[0133] The conversion unit 402 is specifically used for:

[0134] If each image to be processed is a three-channel image, then the format of each image to be processed is converted to a single-channel image to obtain multiple grayscale images.

[0135] If the format of each image to be processed is NV12, then the Y component of each image to be processed is used as a grayscale image to obtain multiple grayscale images.

[0136] The cropping unit 403 is used to crop the region of interest in each grayscale image to obtain multiple target images.

[0137] The first calculation unit 404 is used to calculate the average information entropy of each target image.

[0138] The first calculation unit 404 is specifically used for:

[0139] For each target image, it is divided into multiple small regions. The information entropy of each small region is calculated, and the average information entropy of each small region is calculated to obtain the average information entropy of the target image.

[0140] Evaluation unit 405 is used to evaluate the grayscale evaluation value corresponding to each target image using a fitting function.

[0141] Evaluation unit 405 is specifically used for:

[0142] Obtain the numerical features of each target image and plot the feature curves of the numerical features versus time;

[0143] Preliminary piecewise fitting is performed based on the characteristic curves to obtain various piecewise point configurations;

[0144] The goodness of fit of each segment point configuration is calculated based on the Bayesian information criterion, and the segment point configuration corresponding to the minimum goodness of fit is determined as the optimal segment point configuration.

[0145] From the fitting models configured for each segment point, determine the optimal fitting model corresponding to the optimal segment point configuration;

[0146] Based on the optimal fitting model and the optimal segmentation point configuration, the grayscale evaluation value corresponding to each target image is obtained.

[0147] The second calculation unit 406 is used to calculate the corresponding adaptive grayscale threshold for each target image based on the corresponding average information entropy and the corresponding grayscale evaluation value, and to calculate the average grayscale value of the contour for the contour in the target image.

[0148] The second calculation unit 406, used to calculate the average gray value of contours in a target image, is specifically used for:

[0149] Edge information is obtained by performing edge detection on the target image using the Sobel operator;

[0150] The target image after edge detection is binarized, and noise is removed by erosion to enhance edge information and obtain an edge image;

[0151] The target edge image is obtained by intersecting the edge image and the historical edge images; the historical edge images include edge images obtained from the target image in the past.

[0152] Extract the target contour with an area greater than a threshold from the target edge image;

[0153] If the number of target contours is greater than the number threshold, then sort the target contours in descending order of area and remove the target contours that are sorted after the number threshold.

[0154] For each remaining target contour, draw a contour mask, sample multiple sampling points based on the contour mask, and calculate the gray-scale mean of each sampling point.

[0155] The minimum mean gray level is used as the average gray level of the contour.

[0156] The labeling unit 407 is used to label the target image as a dirty image when the average gray value of the contour is less than the adaptive gray value threshold and less than the average gray value obtained based on each gray value evaluation value.

[0157] The determination unit 408 is used to determine that the vehicle camera is dirty when the number of consecutive target images marked as dirty images exceeds a preset number threshold.

[0158] According to the above-described embodiments of the present invention, a device for detecting dirt in a vehicle-mounted camera acquires multiple consecutive frames of images to be processed using the vehicle-mounted camera to be detected; each image to be processed is converted to obtain multiple grayscale images; for each grayscale image, the region of interest in the grayscale image is cropped to obtain multiple target images; the average information entropy of each target image is calculated; a grayscale evaluation value corresponding to each target image is evaluated using a fitting function; for each target image, a corresponding adaptive grayscale threshold is calculated based on the corresponding average information entropy and the corresponding grayscale evaluation value, and the average grayscale value of the contour in the target image is calculated; when the average grayscale value of the contour is less than the adaptive grayscale threshold and less than the average grayscale evaluation value obtained based on each grayscale evaluation value, the target image is marked as a dirty image; when the number of consecutively marked target images as dirty images exceeds a preset number threshold, the vehicle-mounted camera is determined to be in a dirty state. This solution integrates information entropy and image grayscale, solving the problems that single information entropy is only applicable to fixed scenes, has poor robustness to different lighting conditions, and lacks real-time performance. This enables dirt detection to be scene-insensitive and is applicable to scenarios including but not limited to urban roads, closed parks, parking lots, and highways.

[0159] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0160] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0161] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting dirt in a vehicle-mounted camera, characterized in that, The method includes: The vehicle-mounted camera under test acquires multiple consecutive frames of images to be processed. Each of the images to be processed is converted to obtain multiple grayscale images; For each grayscale image, the region of interest in the grayscale image is cropped to obtain multiple target images; The average information entropy of each target image is calculated. The grayscale evaluation value corresponding to each target image is obtained by using a fitting function; For each target image, a corresponding adaptive grayscale threshold is calculated based on the corresponding average information entropy and the corresponding grayscale evaluation value, and the average grayscale value of the contour is calculated for the contour in the target image. When the average gray value of the contour is less than the adaptive gray threshold and less than the average gray evaluation value obtained based on each of the gray evaluation values, the target image is marked as a dirty image. When the number of target images consecutively marked as dirty images exceeds a preset threshold, the vehicle camera is determined to be in a dirty state.

2. The method according to claim 1, characterized in that, The process of converting each of the images to be processed to obtain multiple grayscale images includes: If each of the images to be processed is a three-channel image, then the format of each of the images to be processed is converted into a single-channel image to obtain multiple grayscale images. If the format of each of the images to be processed is NV12, then the Y component of each of the images to be processed is used as a grayscale image to obtain multiple grayscale images.

3. The method according to claim 1, characterized in that, The calculation of the average information entropy of each target image includes: For each target image, it is divided into multiple small regions, the information entropy of each small region is calculated, and the average information entropy of each small region is calculated to obtain the average information entropy of the target image.

4. The method according to claim 1, characterized in that, The step of using a fitting function to evaluate the grayscale evaluation value corresponding to each target image includes: Numerical features of each target image are obtained, and feature curves of numerical features versus time are plotted. Based on the aforementioned feature curves, preliminary piecewise fitting is performed to obtain various piecewise point configurations; The goodness of fit of each segment point configuration is calculated based on the Bayesian information criterion, and the segment point configuration corresponding to the smallest goodness of fit is determined as the optimal segment point configuration. From the fitting models corresponding to each of the segmentation points, determine the optimal fitting model corresponding to the optimal segmentation point configuration; Based on the optimal fitting model and the optimal segmentation point configuration, the grayscale evaluation value corresponding to each target image is obtained.

5. The method according to claim 1, characterized in that, The step of calculating the average gray value of the contour in the target image includes: Edge information is obtained by performing edge detection on the target image using the Sobel operator; The target image after edge detection is binarized, and noise is removed by erosion to enhance the edge information and obtain an edge image; The target edge image is obtained by intersecting the edge image and the historical edge image; the historical edge image includes edge images obtained from the target image in the past. Extract the target contour with an area greater than the area threshold from the target edge image; If the number of target contours is greater than the number threshold, then the target contours are sorted in descending order of area, and the target contours sorted after the number threshold are removed. For each of the remaining target contours, a contour mask is drawn, and multiple sampling points are obtained based on the contour mask. The average gray value of each sampling point is calculated. The smallest mean gray value is used as the average gray value of the contour.

6. A device for detecting dirt in a vehicle-mounted camera, characterized in that, The device includes: The acquisition unit is used to acquire multiple consecutive frames of images to be processed using the vehicle-mounted camera to be detected. A conversion unit is used to convert each of the images to be processed to obtain multiple grayscale images; The cropping unit is used to crop the region of interest in each grayscale image to obtain multiple target images. The first calculation unit is used to calculate the average information entropy of each target image; An evaluation unit is used to evaluate the grayscale evaluation value corresponding to each target image using a fitting function; The second calculation unit is used to calculate the corresponding adaptive gray threshold for each target image based on the corresponding average information entropy and the corresponding gray level evaluation value, and to calculate the average gray level value of the contour for the contour in the target image. A labeling unit is used to label the target image as a dirty image when the average gray value of the contour is less than the adaptive gray threshold and less than the average gray evaluation value obtained based on each of the gray evaluation values. The determination unit is used to determine that the vehicle camera is dirty when the number of target images consecutively marked as dirty images exceeds a preset number threshold.

7. The apparatus according to claim 6, characterized in that, The conversion unit is specifically used for: If each of the images to be processed is a three-channel image, then the format of each of the images to be processed is converted into a single-channel image to obtain multiple grayscale images. If the format of each of the images to be processed is NV12, then the Y component of each of the images to be processed is used as a grayscale image to obtain multiple grayscale images.

8. The apparatus according to claim 6, characterized in that, The first computing unit is specifically used for: For each target image, it is divided into multiple small regions, the information entropy of each small region is calculated, and the average information entropy of each small region is calculated to obtain the average information entropy of the target image.

9. The apparatus according to claim 6, characterized in that, The evaluation unit is specifically used for: Numerical features of each target image are obtained, and feature curves of numerical features versus time are plotted. Based on the aforementioned feature curves, preliminary piecewise fitting is performed to obtain various piecewise point configurations; The goodness of fit of each segment point configuration is calculated based on the Bayesian information criterion, and the segment point configuration corresponding to the smallest goodness of fit is determined as the optimal segment point configuration. From the fitting models corresponding to each of the segmentation points, determine the optimal fitting model corresponding to the optimal segmentation point configuration; Based on the optimal fitting model and the optimal segmentation point configuration, the grayscale evaluation value corresponding to each target image is obtained.

10. The apparatus according to claim 6, characterized in that, The second calculation unit, used to calculate the average gray value of the contours in the target image, is specifically used for: Edge information is obtained by performing edge detection on the target image using the Sobel operator; The target image after edge detection is binarized, and noise is removed by erosion to enhance the edge information and obtain an edge image; The target edge image is obtained by intersecting the edge image and the historical edge image; the historical edge image includes edge images obtained from the target image in the past. Extract the target contour with an area greater than the area threshold from the target edge image; If the number of target contours is greater than the number threshold, then the target contours are sorted in descending order of area, and the target contours sorted after the number threshold are removed. For each of the remaining target contours, a contour mask is drawn, and multiple sampling points are obtained based on the contour mask. The average gray value of each sampling point is calculated. The smallest mean gray value is used as the average gray value of the contour.

Images