Intelligent automobile vehicle camera fouling detection method, operating system and vehicle-mounted device

Abstract

The specification discloses a kind of intelligent automobile vehicle camera contamination detection method, operating system and vehicle-mounted equipment, the method includes: through the vehicle camera acquisition image frame to be detected;Utilize the contamination detection model of prior training to the contamination classification detection of image frame to be detected, obtain the first contamination area result of image frame to be detected;Based on local image information entropy, the first contamination area result is adjusted, and second contamination area result is obtained;The second contamination area result and overall contamination area result are integrated, and third contamination area result is obtained;Wherein, the overall contamination area result is obtained by integrating the contamination area result of image frame adjusted before image frame to be detected;Using the scoring mechanism based on distance measurement to the contamination score of third contamination area result, and according to contamination score determines the contamination condition of vehicle camera.

Description

Technical Field

[0001] This manual relates to the field of intelligent vehicle visual perception technology, and in particular to methods for detecting dirt on intelligent vehicle cameras, operating systems, and in-vehicle equipment. Background Technology

[0002] In-vehicle cameras play a crucial role in the operation of intelligent vehicles. Imaging systems and automatic parking assistance systems rely heavily on camera images. However, due to the significant uncertainties in real-world driving scenarios, such as dirt sources, weather conditions, and outdoor lighting, and because in-vehicle cameras are generally exposed to the elements and susceptible to environmental factors like mud, rain, snow, and malicious obstruction, images from these cameras may be unclear or contain obstructed important information. This results in lower confidence levels for surround-view functions and poses driving risks.

[0003] To avoid the aforementioned issues, existing technologies typically utilize image imaging to identify dirt in cameras. For example, they may employ frame difference analysis to locate dirty areas or divide the image into multiple blocks for individual dirt scoring. However, the frame difference analysis method essentially identifies dirty areas based on brightness contrast, making it unsuitable for detecting opaque dirt where brightness changes are subtle. Furthermore, cropping and dividing the image into blocks loses the contextual relationships between the blocks, resulting in poor detection of dirt, especially point-like dirt.

[0004] Both of these detection methods increase the risk of false positives due to dirt and cannot effectively detect dirt in vehicle cameras. Therefore, reducing the false positive rate of dirt in vehicle cameras is an urgent problem to be solved. Summary of the Invention

[0005] This manual provides a method, operating system, and in-vehicle equipment for detecting dirt in intelligent automotive cameras, in order to solve or partially solve the technical problem of high false detection rates for dirty automotive cameras.

[0006] To address the aforementioned technical problems, this specification discloses a method for detecting dirt in an intelligent vehicle-mounted camera, the method comprising:

[0007] The vehicle-mounted camera captures image frames to be detected.

[0008] The image frame to be detected is classified and detected using a pre-trained dirt detection model to obtain the first dirt region result of the image frame to be detected;

[0009] The results of the first contaminated area are adjusted based on the local image information entropy to obtain the results of the second contaminated area.

[0010] The second contaminated area result and the overall contaminated area result are integrated to obtain the third contaminated area result; wherein, the overall contaminated area result is obtained by integrating the contaminated area results of the image frames adjusted before the image frame to be detected;

[0011] A distance-based scoring mechanism is used to score the third contaminated area, and the contamination status of the vehicle camera is determined based on the contamination score.

[0012] Preferably, after acquiring the image frame to be detected through the vehicle-mounted camera, the method further includes:

[0013] Determine whether the image frame to be detected is the first frame;

[0014] If the image frame to be detected is the first frame, the image frame to be detected is used as the reference frame, and the step of "using the pre-trained dirt detection model to classify and detect dirt in the image frame to be detected" is directly executed.

[0015] If the image frame to be detected is not the first frame, calculate the inter-frame similarity between the image frame to be detected and the reference frame, and compare the reference similarity with the inter-frame similarity.

[0016] If the inter-frame similarity is greater than the reference similarity, the image frame to be detected is filtered out, and the next image frame to be detected is acquired through the vehicle-mounted camera.

[0017] If the inter-frame similarity is less than or equal to the reference similarity, the reference similarity is replaced by the inter-frame similarity, and the step of "acquiring the image frame to be detected through the vehicle-mounted camera" is executed; and the reference frame is replaced by the image frame to be detected, and the step of "using a pre-trained dirt detection model to classify and detect dirt on the image frame to be detected" is executed.

[0018] Preferably, the step of using a pre-trained contamination detection model to perform contamination classification and detection on the image frame to be detected, and obtaining the first contamination region result of the image frame to be detected, specifically includes:

[0019] A pre-trained dirt detection model is used to classify and detect dirt in each pixel of the image frame to be detected, so as to obtain the dirt category of each pixel in the image frame to be detected. Different dirt categories are represented by different dirt values.

[0020] The first soiled region result is obtained based on the soiling category of each pixel in the image frame to be detected; the first soiled region result contains different soiling categories.

[0021] Preferably, adjusting the first contaminated area result based on local image information entropy to obtain the second contaminated area result specifically includes:

[0022] Determine the local defined region corresponding to each pixel;

[0023] Based on the local defined region corresponding to each pixel, determine the local image information entropy of each local defined region;

[0024] Based on the local image information entropy of each local defined region, the dirt category of the corresponding pixel is adjusted;

[0025] The second stained area result is obtained based on the stained category adjusted for all pixels.

[0026] Preferably, the step of using a distance-based scoring mechanism to score the contamination of the third contaminated area specifically includes:

[0027] A set distance threshold is obtained based on the distance from any vertex to the center point of the image captured by the vehicle-mounted camera;

[0028] Traverse the target distance and its contamination value from each pixel in the third contaminated region result to the center point;

[0029] The target distance from each pixel to the center point, the dirt value, and the set distance threshold are used to calculate the dirt score for each pixel.

[0030] The dirt score of the third dirty area is obtained based on the dirt score of each pixel.

[0031] This specification discloses an operating system for an intelligent vehicle, including:

[0032] The camera acquisition module is used to acquire the image frames to be detected;

[0033] The classification and detection module is used to perform dirt classification and detection on the image frame to be detected using a pre-trained dirt detection model, and obtain the first dirt region result of the image frame to be detected.

[0034] The adjustment module is used to adjust the result of the first contaminated area based on the local image information entropy to obtain the result of the second contaminated area.

[0035] An integration module is used to integrate the second contaminated area result and the overall contaminated area result to obtain a third contaminated area result; wherein, the overall contaminated area result is obtained by integrating the contaminated area results of the image frames adjusted before the image frame to be detected;

[0036] The scoring module is used to score the third contaminated area using a distance-based scoring mechanism, and to determine the contamination status of the vehicle camera based on the contamination score.

[0037] Preferably, the system further includes:

[0038] The judgment module is used to determine whether the image frame to be detected is the first frame;

[0039] The first replacement module is used to, if the image frame to be detected is the first frame, take the image frame to be detected as the reference frame and directly execute the step of "using the pre-trained dirt detection model to classify and detect dirt in the image frame to be detected".

[0040] The calculation module is used to calculate the inter-frame similarity between the image frame to be detected and the reference frame if the image frame to be detected is not the first frame, and compare the reference similarity with the inter-frame similarity.

[0041] The filtering module is used to filter out the image frame to be detected if the inter-frame similarity is greater than the reference similarity, and to acquire the next image frame to be detected through the vehicle-mounted camera.

[0042] The second replacement module is used to replace the reference similarity with the inter-frame similarity if the inter-frame similarity is less than the reference similarity, and to perform the step of "acquiring the image frame to be detected through the vehicle-mounted camera"; and to replace the reference frame with the image frame to be detected, and to perform the step of "using a pre-trained dirt detection model to classify and detect dirt on the image frame to be detected".

[0043] Preferably, the classification and detection module is specifically used for:

[0044] A pre-trained dirt detection model is used to classify and detect dirt in each pixel of the image frame to be detected, so as to obtain the dirt category of each pixel in the image frame to be detected. Different dirt categories are represented by different dirt values.

[0045] The first soiled region result is obtained based on the soiling category of each pixel in the image frame to be detected; the first soiled region result contains different soiling categories.

[0046] Preferably, the adjustment module is specifically used for:

[0047] Determine the local defined region corresponding to each pixel;

[0048] Based on the local defined region corresponding to each pixel, determine the local image information entropy of each local defined region;

[0049] Based on the local image information entropy of each local defined region, the dirt category of the corresponding pixel is adjusted;

[0050] The second stained area result is obtained based on the stained category adjusted for all pixels.

[0051] Preferably, the scoring module is specifically used for:

[0052] A set distance threshold is obtained based on the distance from any vertex to the center point of the image captured by the vehicle-mounted camera;

[0053] Traverse the target distance and its contamination value from each pixel in the third contaminated region result to the center point;

[0054] The target distance from each pixel to the center point, the dirt value, and the set distance threshold are used to calculate the dirt score for each pixel.

[0055] The dirt score of the third dirty area is obtained based on the dirt score of each pixel.

[0056] This specification discloses an in-vehicle device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0057] Through one or more embodiments of this specification, this specification has the following beneficial effects or advantages:

[0058] The solution described in this specification employs a contamination segmentation network model to classify contamination regions in incoming vehicle camera image frames, thus avoiding missed or false detections of contamination types. Subsequently, the first contamination region results obtained from the classification detection are further screened based on the image's local information entropy. Since this secondary screening considers the influence of surrounding local contamination regions on individual pixels, the resulting contamination detection is more accurate. Finally, the overall contamination region results are integrated with the screened contamination detection results, and a scoring mechanism based on distance metrics is used to score the final integrated contamination detection result, thereby determining whether the camera is contaminated. This solution reduces the false detection rate and exhibits strong robustness, performing well in various real-world scenarios and contamination sources.

[0059] The above description is merely an overview of the technical solution in this specification. In order to better understand the technical means in this specification and to implement it in accordance with the contents of this specification, and to make the above and other objects, features and advantages of this specification more apparent and understandable, specific embodiments of this specification are given below. Attached Figure Description

[0060] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0061] Figure 1 A flowchart of a method for detecting dirt in an intelligent vehicle camera according to an embodiment of this specification is shown.

[0062] Figure 2 A flowchart illustrating an implementation of similarity comparison of image frames to be detected according to an embodiment of this specification is shown.

[0063] Figure 3 A schematic diagram of the operating system of an intelligent vehicle according to one embodiment of this specification is shown. Detailed Implementation

[0064] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0065] This specification provides an embodiment of a method for detecting dirt in an intelligent automotive in-vehicle camera, which can be applied to automotive camera visual perception systems. See also... Figure 1 The method includes the following steps:

[0066] Step 101: Acquire the image frame to be detected using the vehicle-mounted camera.

[0067] Typically, an in-vehicle camera can continuously capture 30 image frames per second. Since the image frames captured in such a short time have high similarity, and the computing power of in-vehicle equipment is limited, frame-by-frame detection would undoubtedly increase the computational burden on the equipment. Therefore, to avoid the overhead of frame-by-frame detection, this embodiment filters out highly similar image frames within the video segment by comparing their similarity with a reference frame after capturing the image frame to be detected. Only image frames with low similarity are subjected to subsequent contamination detection. This improves the real-time performance of the detection and reduces the CPU usage of the in-vehicle system, ensuring high-precision detection with low CPU usage and low processing time.

[0068] Specifically, see Figure 2 This is a flowchart illustrating the implementation of similarity comparison for the image frames to be detected in this embodiment.

[0069] Step 201: Determine whether the image frame to be detected is the first frame. If it is the first frame, proceed to step 202; otherwise, proceed to step 203.

[0070] Step 202: Use the image frame to be detected as the reference frame and proceed directly to step 102 for execution.

[0071] Step 203: Calculate the inter-frame similarity between the image frame to be detected and the reference frame.

[0072] Because vehicle-mounted cameras may experience sudden events (such as splashes of mud from puddles or snow), causing changes in their soiling parameters (e.g., changes in the location and area of ​​soiling), and there may also be situations where the soiling location remains constant while the external scene changes continuously, the inter-frame similarity can be calculated using the entire regions of the image frame to be detected and the reference frame. Alternatively, when the soiling location remains constant, the inter-frame similarity can be calculated based on the scene region in the image frame excluding the soiled area and the scene region in the reference frame. The specific algorithm for inter-frame similarity can employ conventional algorithms, which will not be elaborated upon in this embodiment.

[0073] Step 204: Compare reference similarity and inter-frame similarity.

[0074] If the inter-frame similarity is greater than the reference similarity, it means that the image frame to be detected and the reference frame are highly similar, and there is no need to consume the computing power of the vehicle-mounted equipment for detection. In this case, the image frame to be detected is filtered out, and the next image frame to be detected is acquired by the vehicle-mounted camera. Specifically, after filtering out the image frame to be detected, the process can directly proceed to step 101.

[0075] If the inter-frame similarity is less than or equal to the reference similarity, it indicates that the image frame to be detected has changed. For example, the dirt may remain basically unchanged, but the driving scene of the smart car may change; or the smart car may have encountered a sudden event, such as the onboard camera being splashed with new mud. In either case, further detection of the image frame to be detected is required.

[0076] Since the image frame to be detected has changed, the reference frame and the reference similarity will also change accordingly to improve the detection accuracy of subsequent image frames. Specifically, the reference similarity is replaced by the inter-frame similarity, and step 101 is executed; and the reference frame is replaced by the image frame to be detected, and step 102 is executed.

[0077] Specifically, since the image frames are detected continuously, if the inter-frame similarity is less than or equal to the reference similarity, step 205 is executed to detect whether the number of frames of the image frame to be detected has reached the set frame number threshold, such as 5 frames.

[0078] If the set frame count threshold is not reached, proceed to step 206, replacing the reference similarity with inter-frame similarity to reduce the reference similarity value and improve the accuracy of the comparison. Then, proceed to step 101. If the set frame count threshold is reached, proceed to step 202. Here, the step of replacing the reference frame with the image frame to be detected is performed to improve the detection accuracy of subsequent image frames.

[0079] Step 102: Use the pre-trained dirt detection model to perform dirt classification and detection on the image frame to be detected, and obtain the first dirt region result of the image frame to be detected.

[0080] In this embodiment, the contamination detection model is a convolutional neural network, which includes an encoder and a decoder, each containing multiple layers of convolutional kernels. The convolutional kernels perform convolution operations and pass the results to the next layer. The encoder extracts high-dimensional features from the output image using the convolutional kernels, generating a feature vector. The decoder reduces the dimensionality of the feature vector. During training, the model takes training data as input and outputs a contamination probability distribution map. This map is then compared with labeled real contamination images to calculate a loss function, which is the cross-entropy loss function. To ensure the accuracy of the contamination detection model meets requirements, this loss function value needs to be low.

[0081] The training data was manually collected and labeled. Data was collected from various scenarios when the vehicle-mounted camera encountered different sources of contamination, and standard values ​​were obtained by manual labeling according to the type of contamination. In this embodiment, the degree and type of contamination are divided into four categories: no contamination, slight contamination, transparent contamination, and opaque contamination, each represented by a different numerical value, for example, 0, 1, 2, and 3, representing no contamination, slight contamination, transparent contamination, and opaque contamination, respectively. Transparent contamination refers to contamination that does not affect the transmission of external light sources, such as water droplets and non-dark films; while opaque contamination refers to contamination that affects the transmission of external light sources, such as chewing gum and dark tape. During the training of the contamination detection model, to ensure the robustness of the model, data augmentation can be performed on the original data, including but not limited to: center cropping, scaling transformation, random cropping, random left-right flipping, and random color and contrast transformation of the input image.

[0082] During model training, the input images are normalized to ensure a consistent distribution among the training images. For example, using the Adam optimizer with an initial learning rate of 0.01, a poly learning rate decay strategy, and 200 training epochs.

[0083] After training, the image to be detected is input into the pre-trained smudge detection model. The model then performs smudge classification detection on each pixel in the image frame, obtaining the smudge category for each pixel. Different smudge categories are represented by different smudge values. Continuing the example above, 0, 1, 2, and 3 represent no smudge, light smudge, transparent smudge, and opaque smudge, respectively. Based on the smudge category of each pixel in the image frame, the first smudge region result is obtained. The first smudge region result contains different smudge categories, which can be represented by corresponding smudge values.

[0084] In this embodiment, because the types of contamination are clearly categorized, all contamination in the image frame to be detected can be accurately classified, thus avoiding false detections caused by minor contamination and solid color areas. Furthermore, opaque contamination and dot-like contamination can also be accurately classified. Moreover, this embodiment focuses on contamination category detection in the image; scene switching does not affect the contamination classification results, and accurate classification is possible even if the contamination changes. Therefore, this solution demonstrates good detection performance in multiple real-world scenarios and contamination sources.

[0085] Step 103: Adjust the result of the first contaminated area based on the local image information entropy to obtain the result of the second contaminated area.

[0086] In this embodiment, entropy is used to measure the expected value of a random variable, and in an image, it represents the amount of information contained in the clustered features of gray-level distribution; local image information entropy represents the distribution of gray-level values ​​in a local region of the image.

[0087] This embodiment adjusts each pixel individually. Specifically, a local defined region corresponding to each pixel is first determined. Based on the local defined region corresponding to each pixel, the local image information entropy of each local defined region is determined. Based on the local image information entropy of each local defined region, the smudge category of the corresponding pixel is adjusted. During adjustment, the smudge value of the corresponding pixel is adjusted according to the comparison result between the local information entropy and the set information entropy threshold. Based on the adjusted smudge categories of all pixels, the second smudge region result is obtained.

[0088] Specifically, the formula for calculating the local image information entropy is as follows:

[0089]

[0090] Where H is the image information entropy, i is the gray value, and p i This represents the proportion of a pixel with grayscale value i within a defined local area (e.g., a 5x5 area or a 10x10 area).

[0091] The image information entropy corresponding to the contaminated area is smaller than that of the normal area. Based on this characteristic, this embodiment calculates the image information entropy of each pixel in the first contaminated area result.

[0092] After obtaining the image information entropy of the local defined region where the pixel is located, it is compared with a set information entropy threshold. If the image information entropy of the local defined region does not exceed the set information entropy threshold, it indicates a high concentration of dirt, and the dirt value detected by the corresponding pixel in the dirt detection model is not changed. If the image information entropy of the local defined region exceeds the set information entropy threshold, it indicates that the dirt is relatively dispersed, and the dirt value detected by the corresponding pixel in the dirt detection model is lowered. For example, the dirt value of the corresponding pixel is reduced by 1. If the original dirt value of the pixel was 3, it is changed to 2; if the original dirt value of the pixel was 2, it is changed to 1. It is worth noting that the adjusted dirt value is within the range of [0, 4] and cannot exceed the dirt category detected by the dirt detection model.

[0093] In this embodiment, by using local image information entropy to constrain the result of the soiling distribution area of ​​the image frame to be detected, the influence of the surrounding local area is considered for each pixel, thereby improving the detection accuracy of the soiling category of each pixel and effectively reducing the false detection rate of soiling. The detection effect is good in multiple real-world scenarios and soiling sources.

[0094] Step 104: Integrate the results of the second contaminated area and the overall contaminated area to obtain the results of the third contaminated area.

[0095] The overall contaminated area result is obtained by integrating the contaminated area results of the image frames adjusted before the image frame to be detected. For example, the contaminated area result of the first frame is used as the overall contaminated area result. If the similarity between the second frame and the first frame is low, the contaminated area results of the second frame and the first frame are integrated and used as the overall contaminated area result for the next frame integration.

[0096] During the integration process, the first smudge value of the same pixel in the second smudged area and the second smudge value of the overall smudged area are extracted. The first and second smudge values ​​are then compared. If the first smudge value is greater than the second smudge value, the second smudge value is increased. For example, if the first smudge value is greater than the second smudge value, the second smudge value is incremented by 1 (or any other value). If the first smudge value is less than the second smudge value, the second smudge value is decreased. For example, if the first smudge value is less than the second smudge value, the second smudge value is decremented by 1 (or any other value). If the first smudge value is equal to the second smudge value, no adjustment is made.

[0097] In this embodiment, by integrating with the overall contaminated area results, the contamination status of each pixel in different scenarios can be integrated for overall scoring. Even if the contamination status changes, it can be well compatible, thus improving the detection accuracy of the contamination category of each pixel and reducing the false detection rate of contamination. Therefore, the detection effect is good in multiple real-world scenarios and contamination sources.

[0098] Step 105: Use a distance-based scoring mechanism to score the third contaminated area and determine the contamination status of the vehicle camera based on the contamination score.

[0099] Typically, contamination scoring is derived through contamination scoring models. However, since these models optimize for contamination scores, manual labeling of the dataset is required. Human bias in contamination scoring significantly impacts the accuracy and robustness of the final model. Therefore, to avoid this issue, this solution employs a distance-based scoring mechanism for the third contamination region. The objective distance between each pixel and the center point is used as a scoring factor, and the final score is calculated by combining this distance with the contamination value of each pixel. This approach improves both the accuracy and robustness of the scoring.

[0100] In the specific implementation process, a set distance threshold is obtained based on the distance from any vertex to the center point of the image captured by the vehicle-mounted camera. The target distance to the center point and the dirt value of each pixel in the third dirty area result are traversed. For each pixel, the target distance to the center point, the dirt value, and the set distance threshold are used to calculate a dirt score for each pixel. Based on the dirt score of each pixel, the dirt score of the third dirty area result is obtained.

[0101] Specifically, the image captured by the vehicle-mounted camera has four vertices, and the distance from any vertex to the center point is a set distance threshold. This set distance threshold is equivalent to half the diagonal length of the image captured by the vehicle-mounted camera. When calculating the smudge score, for a single pixel, the smudge score is obtained by multiplying the target distance from the pixel to the center point by the set distance threshold, using the proportional relationship between the two distances. The formula for calculating the smudge score is: Where P is the dirt score of a pixel, k is the dirt value of a pixel, d is the target distance, and D is the set distance threshold.

[0102] Once the scores of each pixel in the third contaminated area result are calculated, the average contamination score can be obtained as the contamination score of the third contaminated area result by using the ratio of the sum of the scores of each pixel in the third contaminated area result to the total number of pixels in the third contaminated area result.

[0103] Furthermore, if the dirt score of the third dirt area exceeds a set scoring threshold, the vehicle camera is determined to be dirty. In this embodiment, the set scoring threshold is 2.0, but this is not a limitation. If dirt is present, a reminder is displayed on the vehicle's screen, and the camera cleaning device is controlled to automatically clean the corresponding vehicle camera. If the dirt score of the third dirt area does not exceed the set scoring threshold, the vehicle camera is determined not to be dirty, and no action is taken.

[0104] The above describes the intelligent vehicle-mounted camera contamination detection solution in this specification. It employs a contamination segmentation network model to segment the incoming vehicle-mounted camera image for contamination, obtaining preliminary detection results to avoid missed or false detections of contamination types. Subsequently, the preliminary detection results are further screened based on the image's local information entropy, considering the impact of surrounding local contamination areas on pixels, making the screened contamination detection results more accurate. Finally, the overall contamination area results are used to integrate the screened contamination detection results, and a distance-based scoring mechanism is used to score the final integrated contamination detection result, thereby determining whether the camera is contaminated. This solution reduces the false detection rate and exhibits good robustness, performing well in various real-world scenarios and contamination sources.

[0105] Based on the same inventive concept as in the foregoing embodiments, this specification also provides an operating system for an intelligent vehicle, see below. Figure 3 ,include:

[0106] Camera acquisition module 301 is used to acquire image frames to be detected;

[0107] The classification and detection module 302 is used to perform dirt classification and detection on the image frame to be detected using a pre-trained dirt detection model, and obtain the first dirt region result of the image frame to be detected.

[0108] The adjustment module 303 is used to adjust the result of the first contaminated area based on the local image information entropy to obtain the result of the second contaminated area.

[0109] Integration module 304 is used to integrate the second contaminated area result and the overall contaminated area result to obtain a third contaminated area result; wherein, the overall contaminated area result is obtained by integrating the contaminated area results of the image frames adjusted before the image frame to be detected;

[0110] The scoring module 305 is used to score the third soiled area result using a distance-based scoring mechanism, and to determine the soiling status of the vehicle camera based on the soiling score.

[0111] In some alternative implementations, the system further includes:

[0112] The judgment module is used to determine whether the image frame to be detected is the first frame;

[0113] The first replacement module is used to, if the image frame to be detected is the first frame, take the image frame to be detected as the reference frame and directly execute the step of "using the pre-trained dirt detection model to classify and detect dirt in the image frame to be detected".

[0114] The calculation module is used to calculate the inter-frame similarity between the image frame to be detected and the reference frame if the image frame to be detected is not the first frame, and compare the reference similarity with the inter-frame similarity.

[0115] The filtering module is used to filter out the image frame to be detected if the inter-frame similarity is greater than the reference similarity, and to acquire the next image frame to be detected through the vehicle-mounted camera.

[0116] The second replacement module is used to replace the reference similarity and the reference frame with the inter-frame similarity and the image frame to be detected, respectively, if the inter-frame similarity is less than the reference similarity, and to perform the step of "using a pre-trained dirt detection model to classify and detect dirt on the image frame to be detected".

[0117] In some alternative implementations, the second replacement module is specifically used for:

[0118] Detect whether the frame number of the image frame to be detected has reached a set frame number threshold;

[0119] If the set frame count threshold is not reached, the reference similarity is replaced by the inter-frame similarity.

[0120] If the set frame count threshold is reached, the reference frame is replaced with the image frame to be detected.

[0121] In some optional implementations, the computing module is specifically used for:

[0122] The inter-frame similarity is calculated based on the scene region of the image frame to be detected and the scene region in the reference frame; the scene region is the area outside the smudged area.

[0123] In some optional implementations, the classification detection module 302 is specifically used for:

[0124] A pre-trained dirt detection model is used to classify and detect dirt in each pixel of the image frame to be detected, so as to obtain the dirt category of each pixel in the image frame to be detected. Different dirt categories are represented by different dirt values.

[0125] The first soiled region result is obtained based on the soiling category of each pixel in the image frame to be detected; the first soiled region result contains different soiling categories.

[0126] In some optional implementations, the adjustment module 303 is specifically used for:

[0127] Determine the local defined region corresponding to each pixel;

[0128] Based on the local defined region corresponding to each pixel, determine the local image information entropy of each local defined region;

[0129] Based on the local image information entropy of each local defined region, the dirt category of the corresponding pixel is adjusted;

[0130] The second stained area result is obtained based on the stained category adjusted for all pixels.

[0131] In some optional implementations, the adjustment module 303 is specifically used for:

[0132] Based on the comparison result between the local information entropy and the set information entropy threshold, the smudge value of the corresponding pixel is adjusted.

[0133] In some optional implementations, the integration module 304 is specifically used for:

[0134] Extract the first dirt value of the same pixel in the second dirt area and the second dirt value of the overall dirt area;

[0135] Compare the magnitudes of the first and second soiling values;

[0136] If the first soiling value is greater than the second soiling value, increase the second soiling value;

[0137] If the first soiling value is less than the second soiling value, the second soiling value will be reduced.

[0138] If the first soiling value is equal to the second soiling value, no adjustment is made.

[0139] In some optional implementations, the scoring module 305 is specifically used for:

[0140] A set distance threshold is obtained based on the distance from any vertex to the center point of the image captured by the vehicle-mounted camera;

[0141] Traverse the target distance and its contamination value from each pixel in the third contaminated region result to the center point;

[0142] The target distance from each pixel to the center point, the dirt value, and the set distance threshold are used to calculate the dirt score for each pixel.

[0143] The dirt score of the third dirty area is obtained based on the dirt score of each pixel.

[0144] In some optional implementations, the scoring module 305 is specifically used for:

[0145] If the dirt score of the third dirty area exceeds the set score threshold, the vehicle camera is determined to be dirty.

[0146] If the dirt score of the third dirty area does not exceed the set score threshold, it is determined that the vehicle camera is not dirty.

[0147] Based on the same inventive concept as in the foregoing embodiments, embodiments of this specification also provide an in-vehicle device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0148] The solution presented in this specification performs well in multiple real-world scenarios and with various sources of contamination. By combining a contamination detection model, a re-screening algorithm based on local image entropy, and a scoring mechanism based on distance metrics, the false detection rate of contamination can be effectively reduced. This solution is robust and performs well in multiple real-world scenarios and with various sources of contamination.

[0149] The solution presented in this manual avoids frame-by-frame detection. By detecting frames with low similarity in a video segment, it effectively reduces CPU usage while meeting real-time requirements, thus ensuring high-precision detection with low CPU usage and low time consumption.

[0150] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is readily apparent from the above description. Furthermore, this specification is not directed to any particular programming language. It should be understood that the contents of this specification can be implemented using various programming languages, and the above descriptions of specific languages ​​are for the purpose of disclosing the best implementation of this specification.

[0151] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this specification may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0152] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of this specification, various features of this specification are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed specification requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this specification.

[0153] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0154] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this specification and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0155] The various component embodiments of this specification can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components of the gateway, proxy server, or system according to embodiments of this specification. This specification can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such implementations of this specification can be stored on a computer-readable medium or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0156] It should be noted that the above embodiments are illustrative of this specification and not limiting of it, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This specification can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. An intelligent vehicle camera fouling detection method, the method comprising: collecting a to-be-detected image frame by the vehicle camera; determining whether the to-be-detected image frame is a first frame; if the to-be-detected image frame is a first frame, taking the to-be-detected image frame as a reference frame, using a pre-trained fouling detection model to perform fouling classification detection on the to-be-detected image frame to obtain a first fouling area result of the to-be-detected image frame; if the to-be-detected image frame is not a first frame, calculating an inter-frame similarity between the to-be-detected image frame and the reference frame, and comparing the inter-frame similarity with a reference similarity; if the inter-frame similarity is greater than the reference similarity, filtering out the to-be-detected image frame, and collecting a next to-be-detected image frame by the vehicle camera; if the inter-frame similarity is less than or equal to the reference similarity, replacing the reference similarity with the inter-frame similarity, collecting a to-be-detected image frame by the vehicle camera, replacing the reference frame with the to-be-detected image frame, and using a pre-trained fouling detection model to perform fouling classification detection on the to-be-detected image frame to obtain a first fouling area result of the to-be-detected image frame; adjusting the first fouling area result based on local image information entropy to obtain a second fouling area result; integrating the second fouling area result and a whole fouling area result to obtain a third fouling area result, wherein the whole fouling area result is obtained by integrating fouling area results of image frames adjusted before the to-be-detected image frame; using a scoring mechanism based on distance measurement to perform fouling scoring on the third fouling area result, and determining a fouling condition of the vehicle camera according to the fouling score. 2.The method of claim 1, wherein the using a pre-trained fouling detection model to perform fouling classification detection on the to-be-detected image frame to obtain a first fouling area result of the to-be-detected image frame specifically comprises: using a pre-trained fouling detection model to perform fouling classification detection on each pixel point in the to-be-detected image frame to obtain a fouling category of each pixel point in the to-be-detected image frame, wherein different fouling categories are represented by different fouling values; obtaining the first fouling area result based on the fouling category of each pixel point in the to-be-detected image frame, wherein the first fouling area result contains different fouling categories. 3.The method of claim 2, wherein the adjusting the first fouling area result based on local image information entropy to obtain a second fouling area result specifically comprises: determining a local setting area corresponding to each pixel point; determining a local image information entropy of each local setting area according to the local setting area corresponding to each pixel point; adjusting the fouling category of the corresponding pixel point according to the local image information entropy of each local setting area; obtaining the second fouling area result based on the adjusted fouling category of all pixel points. 4.The method of claim 1, wherein the using a scoring mechanism based on distance measurement to perform fouling scoring on the third fouling area result specifically comprises: Obtaining a set distance threshold based on a distance from any vertex of the image captured by the vehicle-mounted camera to the center point; Traversing a target distance of each pixel point in the third smear area result to the center point and a smear value thereof; Calculating a smear score of each pixel point based on the target distance of each pixel point to the center point, the smear value, and the set distance threshold; Obtaining a smear score of the third smear area result based on the smear score of each pixel point. 5.An operating system of an intelligent vehicle, comprising: a camera acquisition module configured to acquire a to-be-detected image frame; a judgment module configured to judge whether the to-be-detected image frame is a first frame; a first replacement module configured to, if the to-be-detected image frame is the first frame, take the to-be-detected image frame as a reference frame and directly enter a classification detection module for execution; a calculation module configured to, if the to-be-detected image frame is not the first frame, calculate an inter-frame similarity between the to-be-detected image frame and the reference frame, and compare a reference similarity with the inter-frame similarity; a filtering module configured to, if the inter-frame similarity is greater than the reference similarity, filter out the to-be-detected image frame, and acquire a next to-be-detected image frame through a vehicle-mounted camera; a second replacement module configured to, if the inter-frame similarity is less than the reference similarity, replace the reference similarity with the inter-frame similarity, and enter the camera acquisition module to acquire a to-be-detected image frame; and replace the reference frame with the to-be-detected image frame and enter the classification detection module for execution; the classification detection module is configured to perform smear classification detection on the to-be-detected image frame by using a smear detection model trained in advance, to obtain a first smear area result of the to-be-detected image frame; an adjustment module configured to adjust the first smear area result based on a local image information entropy, to obtain a second smear area result; an integration module configured to integrate the second smear area result and an overall smear area result, to obtain a third smear area result; wherein the overall smear area result is obtained by integrating smear area results of image frames adjusted before the to-be-detected image frame; a scoring module configured to perform smear scoring on the third smear area result by using a scoring mechanism based on distance measurement, and determine a smear condition of the vehicle-mounted camera according to a smear score. 6.The system of claim 5, wherein the classification detection module is specifically configured to: perform smear classification detection on each pixel point in the to-be-detected image frame by using a smear detection model trained in advance, to obtain a smear category of each pixel point in the to-be-detected image frame, and different smear categories are represented by different smear values; obtain the first smear area result based on the smear category of each pixel point in the to-be-detected image frame; and the first smear area result contains different smear categories. 7.The system of claim 6, wherein the adjustment module is specifically configured to: determine a local set area corresponding to each pixel point; determine a local image information entropy of each local set area according to the local set area corresponding to each pixel point. According to the local image information entropy of each local setting area, adjust the dirtiness category of the corresponding pixel point; Based on the adjusted dirtiness category of all pixel points, obtain the second dirtiness area result.

8. The system of claim 5, wherein the scoring module is specifically configured to: Based on the distance from any vertex of the image collected by the vehicle-mounted camera to the center point, obtain a set distance threshold; Traverse the target distance from each pixel point in the third dirtiness area result to the center point and the dirtiness value thereof; The target distance from each pixel point to the center point, the dirtiness value, and the set distance threshold are used to calculate the dirtiness score of each pixel point; Based on the dirtiness score of each pixel point, obtain the dirtiness score of the third dirtiness area result.

9. A vehicle-mounted device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of any one of claims 1-4.

Images