A camera dirty detection method and device, electronic equipment and storage medium

Abstract

The application provides a camera dirt detection method and device, electronic equipment and storage medium, wherein the camera dirt detection method comprises the following steps: in response to the target camera collecting a first image, determining the dirt probability of the target camera according to the first image, and obtaining the target weather state when the target camera collects the first image; wherein the dirt probability is used to represent the probability of the target camera being dirty; determining the dirt probability threshold corresponding to the target weather state; and determining the dirt detection result of the target camera according to the comparison result between the dirt probability and the dirt probability threshold. The application compares the dirt probability of the target camera determined based on the first image with the dirt probability threshold determined based on the target weather state when the target camera collects the first image to determine the dirt detection result of the target camera, quantifies the standard of camera dirt detection, and can improve the accuracy of camera dirt detection.

Description

Technical Field

[0001] This application relates to the field of image technology, specifically to a method, apparatus, electronic device, and storage medium for detecting dirt in a camera. Background Technology

[0002] In fields such as autonomous driving, security monitoring, and industrial inspection, cameras serve as core sensors, and their image quality directly impacts the performance of vision-based intelligent systems. However, camera lenses are susceptible to contamination from dust, water droplets, oil stains, and other contaminants. When a camera is dirty, its image quality deteriorates, potentially threatening the safety and reliability of the intelligent system's decision-making.

[0003] Currently, the detection of dirt on cameras mainly relies on manual inspection. The results are greatly affected by human subjective judgment, and each person's standard for judging dirt varies significantly, resulting in poor accuracy in camera dirt detection. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for detecting dirt in a camera, aiming to solve the problem of poor accuracy in camera dirt detection in the prior art.

[0005] This application provides a method for detecting dirt in a camera, comprising: responding to a target camera acquiring a first image, determining a dirt probability of the target camera based on the first image, and acquiring a target weather state when the target camera acquired the first image; wherein the dirt probability is used to characterize the probability that the target camera is dirty; determining a dirt probability threshold corresponding to the target weather state; and determining a dirt detection result of the target camera based on a comparison result between the dirt probability and the dirt probability threshold.

[0006] In one embodiment, determining the probability of dirtiness of the target camera based on the first image and obtaining the target weather state when the target camera captured the first image includes: inputting the first image into a trained target neural network model; the target neural network model includes a first prediction network and a second prediction network; outputting the probability of dirtiness of the target camera based on the first image through the first prediction network; and outputting the target weather state based on the first image through the second prediction network.

[0007] In one embodiment, before inputting the first image into the trained target neural network model, the dirt detection method further includes: inputting training samples into the neural network model to be trained to obtain a dirt prediction probability and a weather state prediction result corresponding to the sample image, wherein the training samples include the sample image and a dirt label probability and a weather state label result corresponding to the sample image; determining a first loss function value of the neural network model to be trained based on the weather state prediction result and the weather state label; and determining a second loss function value of the neural network model to be trained based on the dirt prediction probability and the dirt label probability; determining a target loss function value based on the first loss function value and the second loss function value; and updating the network parameters of the neural network model to be trained based on the target loss function value to train the target neural network model; wherein the network parameters include the parameters of the first prediction network and the parameters of the second prediction network.

[0008] In one embodiment, the training samples further include dirt label box information corresponding to the sample images; determining the target loss function value based on the first loss function value and the second loss function value includes: determining a third loss function value for the neural network model to be trained based on the dirt prediction box information corresponding to the sample images and the dirt label box information; the dirt prediction box information is obtained by inputting the training samples into the neural network model to be trained; and determining the target loss function value based on the first loss function value, the second loss function value, and the third loss function.

[0009] In one embodiment, the target camera is mounted on a vehicle; before determining the probability of dirt on the target camera based on the first image and obtaining the target weather conditions when the camera captured the first image, the dirt detection method further includes: determining the image acquisition frequency of the target camera based on a preset range in which the vehicle's driving parameters are located; and acquiring the first image using the target camera according to the image acquisition frequency.

[0010] In one embodiment, determining the dirt probability threshold corresponding to the target weather state includes: obtaining a dirt probability threshold that corresponds to the target weather state based on the correspondence between weather states and dirt probability thresholds; wherein the correspondence is determined based on the false detection rate of dirt corresponding to different weather states.

[0011] In one embodiment, determining the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold includes: if the dirt probability is greater than or equal to the dirt probability threshold, acquiring the dirt detection result corresponding to a target number of second images; the target number of second images are consecutive frame images captured by the target camera preceding the first image; if the dirt detection result corresponding to the target number of second images indicates that the target camera is dirty, determining that the dirt detection result of the target camera is that the target camera is dirty.

[0012] This application embodiment also provides a camera dirt detection device, the dirt detection device including an acquisition module, a dirt probability threshold module, and a dirt detection module; the acquisition module is used to determine the dirt probability of the target camera based on the first image in response to the target camera acquiring a first image, and to acquire the target weather state when the target camera acquired the first image; wherein, the dirt probability is used to characterize the probability of the target camera being dirty; the dirt probability threshold module is used to determine a dirt probability threshold corresponding to the target weather state; the dirt detection module is used to determine the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold.

[0013] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for detecting dirt in a camera.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting dirt in a camera.

[0015] This application provides a method, apparatus, electronic device, and storage medium for detecting dirt in a camera. In response to a target camera acquiring a first image, the method determines the probability of dirt in the target camera based on the first image, and acquires the target weather state at the time the first image was acquired. The dirt probability is used to characterize the probability of dirt appearing in the target camera. A dirt probability threshold corresponding to the target weather state is determined. Based on a comparison between the dirt probability and the dirt probability threshold, a dirt detection result for the target camera is determined. This quantifies the standard for camera dirt detection and improves the accuracy of camera dirt detection. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of the dirt detection method for a camera provided in the embodiments of this application;

[0017] Figure 2 This is a schematic diagram illustrating the training of the target neural network model provided in an embodiment of this application;

[0018] Figure 3 This is a schematic flowchart of the camera dirt detection method provided in the embodiments of this application;

[0019] Figure 4 This is a schematic diagram of the camera dirt detection device provided in the embodiments of this application;

[0020] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the digit " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] The camera dirt detection method provided in this application can be applied to electronic devices or their software. The electronic device can be a vehicle, a terminal, or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers. The software can be an application that implements the camera dirt detection method, but is not limited to the above forms.

[0024] The method for detecting dirt in a camera provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Please see Figure 1 This application provides a method for detecting dirt in a camera, comprising:

[0026] Step S101: In response to the target camera acquiring a first image, determine the dirt probability of the target camera based on the first image, and obtain the target weather state when the target camera acquired the first image; wherein, the dirt probability is used to characterize the probability that the target camera is dirty.

[0027] In practical implementation, upon receiving the first image captured by the target camera, physical reflection and optical characteristics analysis can be performed on the first image to obtain target image information. Then, based on the correspondence between the target image information and the probability of dirt, the target dirt probability corresponding to the target image information of the first image can be obtained, and the target dirt probability can be determined as the dirt probability of the target camera. Optionally, the target image information may include at least one of reflectivity, gloss, and spectral characteristics.

[0028] In actual implementation, upon receiving the first image captured by the target camera, the first image can also be input into the trained target neural network model. Based on the first image, the first prediction network of the target neural network model outputs the dirt probability of the target camera. For specific implementation details, please refer to the relevant description below, which will not be described here.

[0029] In actual implementation, upon receiving the first image captured by the target camera, the first image can be input into the trained target neural network model. The second prediction network of the target neural network model outputs the target weather state at the time the target camera captured the first image based on the first image. For specific implementation details, please refer to the relevant description below, which will not be described here.

[0030] In practical implementation, upon receiving the first image captured by the target camera, the weather forecast content matching the capture timestamp of the first image can be read to obtain the target weather state when the target camera captured the first image; alternatively, temperature and / or humidity data matching the capture timestamp of the first image can be extracted from the temperature and / or humidity sensors, and the target weather state when the target camera captured the first image can be obtained by analyzing the temperature and / or humidity data matching the capture timestamp. This can save the computational resources required to determine the target weather state using the aforementioned target neural network model.

[0031] Step S102: Determine the dirt probability threshold corresponding to the target weather state;

[0032] In practice, the target weather state when the target camera captures the first image can be used to query a table that stores the correspondence between weather state and dirt probability threshold to obtain the dirt probability threshold corresponding to the target weather. Alternatively, the target weather state when the target camera captures the first image can be input into a model that has learned the correspondence between weather state and dirt probability threshold to obtain the dirt probability threshold corresponding to the target weather.

[0033] Step S103: Determine the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold.

[0034] Optionally, if the dirt probability is less than the dirt probability threshold, the above dirt detection result can be determined as the target camera being undirty; if the dirt probability is greater than or equal to the dirt probability threshold, the above dirt detection result can be determined as the target camera being dirty.

[0035] This application embodiment, in response to the target camera acquiring a first image, determines the dirt probability of the target camera based on the first image, obtains the target weather state when the target camera acquired the first image, determines a dirt probability threshold corresponding to the target weather state, and determines the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold. This quantifies the standard for camera dirt detection and can improve the accuracy of camera dirt detection.

[0036] In one embodiment, step S101, which involves determining the probability of dirt on the target camera based on the first image and obtaining the target weather condition when the target camera captured the first image, includes:

[0037] The first image is input into the trained target neural network model; the target neural network model includes a first prediction network and a second prediction network.

[0038] Based on the first image, the first prediction network outputs the probability of dirtiness of the target camera; and based on the first image, the second prediction network outputs the target weather status.

[0039] In practical implementation, a neural network model to be trained can be pre-built. This model can have a first prediction network for predicting the probability of dirt and a second prediction network for predicting weather conditions. Optionally, MobileNetV3, ConvNeXt, Transformer base networks, or YOLO series networks can be selected as the backbone network. Based on the backbone network, a dirt detection head for predicting the probability of dirt (equivalent to obtaining the first prediction network) and a weather detection head for predicting weather conditions (equivalent to obtaining the second prediction network) can be designed to obtain the neural network model to be trained. Then, the neural network model to be trained can be trained to obtain the target neural network model. For specific implementation details, please refer to the relevant descriptions below, which will not be described here.

[0040] Upon receiving the first image captured by the target camera, the first image can be input into the target neural network model. The target neural network model can use its first prediction network to detect the probability of dirt corresponding to the first image and obtain the probability of dirt of the target camera. The target neural network model can also use its second prediction network to detect the weather state corresponding to the first image and obtain the target weather state when the target camera captured the first image.

[0041] It's worth noting that once the target neural network model is trained, it can be stored. When the trained target neural network model is needed in practice, it can be directly called, eliminating the need to repeat the steps of building and training the neural network model.

[0042] This application embodiment improves the efficiency and accuracy of determining the dirt probability of the target camera and the efficiency and accuracy of obtaining the target weather state when the target camera captures the first image by inputting the first image into a trained target neural network model including a first prediction network and a second prediction network. In this way, the first prediction network outputs the dirt probability of the target camera based on the first image, and the second prediction network outputs the target weather state based on the first image. This improves the efficiency and accuracy of dirt detection of the target camera based on the dirt probability of the target camera and the target weather state when the target camera captures the first image.

[0043] In one embodiment, before inputting the first image into the trained target neural network model, the dirt detection method provided in this application further includes:

[0044] The training samples are input into the neural network model to be trained to obtain the dirt prediction probability and weather state prediction results corresponding to the sample images. The training samples include the sample images and the dirt label probability and weather state label results corresponding to the sample images.

[0045] Based on the weather state prediction results and weather state labels, determine the first loss function value of the neural network model to be trained; and based on the dirt prediction probability and dirt label probability, determine the second loss function value of the neural network model to be trained.

[0046] The target loss function value is determined based on the first loss function value and the second loss function value.

[0047] Based on the target loss function value, the network parameters of the neural network model to be trained are updated to train the target neural network model; the network parameters include the parameters of the first prediction network and the parameters of the second prediction network.

[0048] In practice, after inputting the training samples into the neural network model to be trained, the network parameters of the model can be updated for each individual sample image in the training samples: For the nth sample image, the first loss function value can be determined based on its corresponding weather state prediction result and weather state label result; the second loss function value can be determined based on its corresponding dirt prediction probability and dirt label probability; and the target loss function value can be determined based on the sum of the first and second loss function values. If the target loss function value of the nth sample image does not meet the requirements, the network parameters of the neural network model to be trained can be adjusted based on the target loss function value of the nth sample image using backpropagation and loss gradient descent. After the parameter adjustment is completed, n can be updated to n+1, and the above steps are repeated until the target loss function value meets the requirements (which can be understood as the neural network model to be trained converging), thus obtaining the trained target neural network model.

[0049] In practice, after inputting the training samples into the neural network model to be trained, the network parameters of the neural network model can be updated for a single batch of sample images in the training samples: For the nth batch of sample images, the first loss function value corresponding to the nth batch of sample images can be determined based on its corresponding weather state prediction result and weather state label result; the second loss function value corresponding to the nth batch of sample images can be determined based on its corresponding dirt prediction probability and dirt label probability; and the target loss function value corresponding to the nth batch of sample images can be determined based on the sum of the first loss function value and the second loss function value corresponding to the nth batch of sample images. If the target loss function value corresponding to the nth batch of sample images does not meet the requirements, the network parameters of the neural network model to be trained can be adjusted based on the target loss function value corresponding to the nth batch of sample images using backpropagation and loss gradient descent. After the parameter adjustment is completed, n can be updated to n+1, and the above steps can be repeated until the target loss function value meets the requirements, thus obtaining the trained target neural network model.

[0050] This application embodiment trains a neural network model using training samples, including sample images and corresponding dirt label probabilities and weather state label results. It then updates the network parameters of the neural network model by determining a first loss function value based on the weather state label and the weather state prediction results output by the model, and a second loss function value based on the dirt label probabilities and the dirt prediction probabilities output by the model. This ensures the training effect of the neural network model and improves the accuracy of dirt detection using the trained target neural network model.

[0051] In one embodiment, determining the first loss function value of the neural network model to be trained based on the weather state prediction results and the weather state label results includes:

[0052] For each sample image in a batch of sample images, the value of the first objective loss function corresponding to the individual sample image is determined based on the weather state label result corresponding to the individual sample image and the weather state prediction result corresponding to the individual sample image.

[0053] The first loss function value corresponding to each batch of sample images is determined based on the first target loss function value corresponding to each individual sample image in each batch of sample images.

[0054] In practice, for each sample image in a batch of sample images, the first target loss function value corresponding to a single sample image can be obtained by querying the relationship table between the weather state label results, weather state prediction results and loss function values ​​based on the weather state label results and weather state prediction results corresponding to the single sample image; alternatively, the weather state label results and weather state prediction results corresponding to a single sample image can be substituted into the relationship formula between the weather state label results, weather state prediction results and loss function values ​​to obtain the first target loss function value corresponding to the single sample image.

[0055] In practice, for each batch of sample images, the average of the first target loss function values ​​corresponding to all individual sample images in each batch can be used to determine the first loss function value corresponding to each batch of sample images; alternatively, a coefficient can be added to the average of the first target loss function values ​​corresponding to all individual sample images in each batch to obtain the first loss function value corresponding to each batch of sample images.

[0056] Optionally, the weather state label result corresponding to a single sample image can be used to characterize the real weather state (such as sunny, rainy, snowy, etc.) when the camera acquires the sample image. It can be obtained by acquiring the weather state label information of a single sample image. The camera and the target camera mentioned above can be the same camera or different cameras.

[0057] In practice, the weather status label results can be represented by probabilities. For example, (100%, 0, 0) can represent that the true probability of the weather being sunny, rainy, or snowy when the camera captured the sample image is 100%, 0, and 0 respectively, meaning the true weather condition when the camera captured the sample image is sunny. Alternatively, the weather status label results can also be represented by one-hot encoding. For example, (1, 0, 0) can represent that the true weather condition when the camera captured the sample image is sunny.

[0058] Optionally, the weather state prediction result corresponding to a single sample image can characterize the predicted weather state when the camera acquires the sample image, and can be represented by probability. For example, (70%, 20%, 10%) can represent the predicted probabilities of sunny, rainy, and snowy weather when the camera acquires the sample image as 70%, 20%, and 10%, respectively.

[0059] In practice, the multivariate cross-entropy loss function can be used, and the value of the first loss function corresponding to the nth batch of sample images can be determined using formula (1):

[0060]

[0061] Among them, L weather(n) This represents the value of the first loss function corresponding to the nth batch of sample images. w represents the value of the first objective loss function corresponding to the i-th sample image in the n-th batch of sample images. i,k , Let represent the one-hot encoded value or true probability and predicted probability of the i-th sample image and the k-th weather state in the n-th batch of sample images, respectively, and N represent the total number of sample images in the n-th batch of sample images.

[0062] In this embodiment, for a single sample image in each batch of sample images, the first target loss function value corresponding to the single sample image is determined based on the weather state label result and the weather state prediction result corresponding to the single sample image. Furthermore, based on the first target loss function values ​​corresponding to all single sample images in each batch of sample images, the first loss function value corresponding to each batch of sample images is determined. This improves the accuracy of determining the first loss function value corresponding to each batch of sample images, thereby improving the training effect of the neural network model to be trained based on the first loss function value corresponding to each batch of sample images. As a result, the accuracy of using the trained target neural network model to detect dirt in the camera is improved.

[0063] In one embodiment, determining the second loss function value of the neural network model to be trained based on the dirt prediction probability and the dirt label probability includes:

[0064] For each sample image in a batch of sample images, the value of the second objective loss function corresponding to the individual sample image is determined based on the dirt label probability and the dirt prediction probability corresponding to the individual sample image.

[0065] The second loss function value is determined based on the second target loss function value corresponding to each individual sample image in each batch of sample images.

[0066] In practice, for a single sample image in each batch of sample images, the second target loss function value for a single sample image can be obtained by looking up the relationship table between the dirt label probability, the dirt prediction probability and the loss function value based on the dirt label probability and the dirt prediction probability corresponding to the single sample image; alternatively, the dirt label probability and the dirt prediction probability corresponding to the single sample image can be substituted into the relationship formula between the dirt label probability, the dirt prediction probability and the loss function value to obtain the second target loss function value for the single sample image.

[0067] In practice, for each batch of sample images, the average of the second objective loss function values ​​corresponding to all individual sample images in each batch can be used to determine the second loss function value corresponding to each batch of sample images; alternatively, a coefficient can be added to the average of the second objective loss function values ​​corresponding to all individual sample images in each batch to obtain the second loss function value corresponding to each batch of sample images.

[0068] Optionally, the dirt label probability corresponding to a single sample image can be used to characterize the true probability that the camera acquiring the sample image is dirty, which can be obtained by acquiring the dirt label information of a single sample image. In actual implementation, the dirt label probability corresponding to a single sample image can be represented by a probability value, such as (100%, 0) which can represent that the true probability that the camera acquiring the sample image is dirty is 100%; (0, 100%) can represent that the true probability that the camera acquiring the sample image is dirty is 0. In actual implementation, the dirt label probability corresponding to a single sample image can be represented by one-hot encoding, such as (1, 0) which can represent that the true probability that the camera acquiring the sample image is dirty is 100%; (0, 1) can represent that the true probability that the camera acquiring the sample image is dirty is 0.

[0069] Optionally, the dirt prediction probability corresponding to a single sample image can characterize the prediction probability that the camera acquiring the sample image is dirty. For example, (60%, 40%) can represent the prediction probabilities that the camera acquiring the sample image is dirty and not dirty, respectively, which are 60% and 40%.

[0070] In practice, the binary cross-entropy loss function can be used, and the value of the second loss function corresponding to the nth batch of sample images can be determined using formula (2):

[0071]

[0072] Among them, L cls_dirt(n) This represents the value of the second loss function corresponding to the nth batch of sample images. Let y represent the value of the second objective loss function corresponding to the i-th sample image in the n-th batch of sample images. i , Let represent the true probability and the predicted probability that the camera capturing the i-th sample image in the n-th batch of sample images is dirty, respectively, and N is the total number of sample images in the n-th batch of sample images.

[0073] In this embodiment, for a single sample image in each batch of sample images, the second target loss function value corresponding to a single sample image is determined based on the dirt label probability and the dirt prediction probability corresponding to the single sample image. The second loss function value is determined based on the second target loss function values ​​corresponding to all single sample images in each batch of sample images. This improves the accuracy of determining the second loss function value corresponding to each batch of sample images, thereby improving the training effect of the neural network model to be trained based on the second loss function value corresponding to each batch of sample images. This improves the accuracy of dirt detection of the camera using the trained target neural network model.

[0074] Optionally, the training samples also include dirt label box information corresponding to the sample images. In one embodiment, determining the target loss function value based on the first loss function value and the second loss function value includes:

[0075] The third loss function value of the neural network model to be trained is determined based on the dirt prediction box information and dirt label box information corresponding to the sample images; the dirt prediction box information is obtained by inputting the training samples into the neural network model to be trained.

[0076] The target loss function value is determined based on the first loss function value, the second loss function value, and the third loss function.

[0077] In practice, after inputting the training samples into the neural network model to be trained, the network parameters of the neural network model can be updated for each individual sample image in the training samples: For the nth sample image, given the first and second loss function values ​​corresponding to the nth sample image, a third loss function can be determined based on its corresponding dirt prediction box and dirt label box information; then, the target loss function value corresponding to the nth sample image can be determined based on the sum of the first, second, and third loss function values; if the target loss function value corresponding to the nth sample image does not meet the requirements, the network parameters of the neural network model to be trained can be adjusted based on the target loss function value corresponding to the nth sample image using backpropagation and loss gradient descent; after the parameter adjustment is completed, n can be updated to n+1, and the above steps are repeated until the target loss function value meets the requirements (which can be understood as the neural network model to be trained converging), thus obtaining the trained target neural network model.

[0078] In practice, after inputting the training samples into the neural network model to be trained, the network parameters of the neural network model can be updated for a single batch of sample images in the training samples: For the nth batch of sample images, after determining the first and second loss function values ​​corresponding to the nth batch of sample images, a third loss function can be determined based on the corresponding dirt prediction box information and dirt label box information; then, the target loss function value corresponding to the nth batch of sample images can be determined based on the sum of the first, second, and third loss function values ​​corresponding to the nth batch of sample images; if the target loss function value corresponding to the nth batch of sample images does not meet the requirements, the network parameters of the neural network model to be trained can be adjusted based on the target loss function value corresponding to the nth batch of sample images using backpropagation and loss gradient descent; after the parameter adjustment is completed, n can be updated to n+1, and the above steps can be repeated until the target loss function value meets the requirements, thus obtaining the trained target neural network model.

[0079] This application embodiment determines the third loss function value of the neural network model to be trained based on the dirt prediction box information and dirt label box information corresponding to the sample image, and determines the target loss function value based on the first loss function value, the second loss function value and the third loss function. This can improve the effect of training the neural network model to be trained using the target loss function value, thereby improving the accuracy of dirt detection of the camera using the trained target neural network model.

[0080] In one embodiment, determining the third loss function value of the neural network model to be trained based on the dirt prediction box information and dirt label box information corresponding to the sample image includes:

[0081] For each sample image in a batch of sample images, the value of the third objective loss function corresponding to the individual sample image is determined based on the dirt prediction box information and dirt label box information corresponding to the individual sample image.

[0082] The third loss function value is determined based on the third objective loss function value corresponding to each individual sample image in each batch of sample images.

[0083] In practice, for a single sample image in each batch of sample images, the third target loss function value for a single sample image can be obtained by querying the relationship table between the dirt prediction box information, dirt label box information and loss function value based on the dirt prediction box information and dirt label box information corresponding to the single sample image; alternatively, the third target loss function value for a single sample image can be obtained by substituting the dirt prediction box information and dirt label box information corresponding to the single sample image into the relationship formula between the dirt prediction box information, dirt label box information and loss function value.

[0084] In practice, for each batch of sample images, the average of the third objective loss function values ​​corresponding to all individual sample images in each batch can be used to determine the third loss function value for each batch of sample images. Alternatively, a coefficient can be added to the average of the third objective loss function values ​​corresponding to all individual sample images in each batch to obtain the third loss function value for each batch of sample images. This application does not limit the specific method for determining the third loss function value for each batch of sample images based on the third objective loss function values ​​corresponding to all individual sample images in each batch; the specific method can be determined according to the actual situation.

[0085] Optionally, the dirt label bounding box information corresponding to a single sample image may include at least one of the following: the position of the center point of the dirt label bounding box in the sample image, the aspect ratio of the dirt label bounding box, and the selected area range of the dirt label bounding box in the sample image. This information can be obtained by acquiring the dirt annotation information of a single sample image. The dirt label bounding box can represent the true detection bounding box of the dirt region in the image. Correspondingly, the dirt prediction bounding box information corresponding to a single sample image may include at least one of the following: the position of the center point of the dirt prediction bounding box in the sample image, the aspect ratio of the dirt prediction bounding box, and the selected area range of the dirt prediction bounding box in the sample image. The dirt prediction bounding box can represent the predicted detection bounding box of the dirt region in the image.

[0086] In actual implementation, for a single sample image in each batch of sample images, the fourth loss function value corresponding to the sample image can be determined based on the overlapping area between the selected area of ​​the dirt prediction box and the selected area of ​​the dirt label box. The fifth loss function value can be determined based on the deviation between the center point of the dirt prediction box and the center point of the dirt label box in the sample image. The sixth loss function value can be determined based on the deviation between the aspect ratio of the dirt prediction box and the aspect ratio of the dirt label box. Then, the third target loss function value corresponding to the sample image can be determined based on the fourth, fifth, and sixth loss function values.

[0087] In practice, the Complete Intersection over Union Loss (CIOU) function can be used, and formulas (3) and (4) can be used to determine the value of the third objective loss function corresponding to a single sample image in each batch of sample images:

[0088] L box =1-CIoU (3)

[0089]

[0090] Among them, L box The value of the third objective loss function is represented by the value of a single sample image in each batch of sample images; IoU represents the value of the fourth loss function mentioned above, which can be in the range of [0, 1]. The specific value can be determined based on the overlapping area between the selected area of ​​the dirt prediction box and the selected area of ​​the dirt label box corresponding to a single sample image. This represents the value of the fifth loss function mentioned above, where ρ 2 (·) represents the square of the Euclidean distance between the center point of the dirt prediction box and the center point of the dirt label box corresponding to a single sample image, b, b gt represents the center point of the dirt prediction box and the center point of the dirt label box corresponding to a single sample image in the sample image, respectively; c represents the diagonal distance of the minimum closure region containing the dirt prediction box and the dirt label box corresponding to a single sample image; βv represents the value of the sixth loss function mentioned above, where v represents the deviation between the aspect ratio of the dirt prediction box and the aspect ratio of the dirt label box corresponding to a single sample image; β represents the weight coefficient of v.

[0091] In practice, the fourth, fifth, and sixth loss function values ​​corresponding to the sample image can also be determined by referring to the above method of determining the first or second target loss function value by querying the relationship table, which will not be elaborated here.

[0092] In this embodiment of the application, for a single sample image in each batch of sample images, the third target loss function value corresponding to a single sample image is determined based on the dirt prediction box information and dirt label box information corresponding to the single sample image. Furthermore, the third loss function value is determined based on the third target loss function values ​​corresponding to all single sample images in each batch of sample images. This can improve the accuracy of determining the third loss function value, thereby improving the training effect of the neural network model to be trained using the third loss function value, and thus improving the accuracy of dirt detection of the camera using the trained target neural network model.

[0093] Optionally, the target camera is mounted on the vehicle. In one embodiment, before determining the probability of dirt on the target camera based on the first image in step S101 above, and before acquiring the target weather conditions when the camera captured the first image, the dirt detection method provided in this application embodiment further includes:

[0094] The image acquisition frequency of the target camera is determined based on the preset range of the vehicle's driving parameters.

[0095] The first image is acquired using the target camera according to the image acquisition frequency.

[0096] When the target camera is mounted on a vehicle, considering that the impact of dirt on the target camera on vehicle safety may vary depending on the vehicle's driving parameters, in practical implementation, a pre-defined correspondence between the vehicle's driving parameter range and the camera's image acquisition frequency can be established. Then, in response to the vehicle being in motion, the vehicle's driving parameters are acquired, and these parameters are compared with the pre-defined driving parameter range to determine the range within which the vehicle's driving parameters fall. Based on this range, the pre-defined correspondence between the vehicle's driving parameter range and the camera's image acquisition frequency can be queried to determine the target camera's image acquisition frequency. The target camera can then be controlled to acquire a first image at this acquisition frequency. Thus, the dirt detection method for the camera provided in this application not only fully ensures vehicle driving safety but also precisely controls the consumption of computing resources.

[0097] Optionally, the vehicle's driving parameters may include at least one of the vehicle's speed and acceleration. Taking speed as an example, the correspondence between the aforementioned pre-set range of vehicle driving parameters and the camera image acquisition frequency can be expressed by formula (5):

[0098]

[0099] Among them, f detect The frequency of camera image acquisition is indicated; the vehicle speed range includes three intervals: V≥60km / h, 15km / h≤V<60km / h, and V<15km / h, where V represents the vehicle speed; f1, f2, and f3 represent the camera image acquisition frequencies corresponding to the vehicle speed intervals V≥60km / h, 15km / h≤V<60km / h, and V<15km / h, respectively; optionally, f1>f2>f3.

[0100] Thus, when the vehicle is traveling at medium to high speeds, the camera dirt detection method provided in this application allows the vehicle to continuously, stably, and accurately capture the external environment for the intelligent driving system through the target camera, avoiding missing key targets and allowing sufficient computing time for the intelligent driving system to react, thereby improving vehicle driving safety. When the vehicle is traveling at low speeds, the camera dirt detection method provided in this application can save the computing resources required to implement the camera dirt detection method provided in this application while ensuring vehicle driving safety.

[0101] In one embodiment, step S102 above, determining the dirt probability threshold corresponding to the target weather state, includes:

[0102] Based on the correspondence between weather conditions and dirt probability thresholds, a dirt probability threshold corresponding to the target weather condition is obtained.

[0103] The correspondence is determined based on the false detection rate of dirt and grime under different weather conditions.

[0104] Considering that the probability of a camera being mistakenly identified as dirty varies under different weather conditions, for example, in snowy weather, snowflakes adhering to the camera lens are more likely to be mistakenly identified as dirty; in rainy weather, although the fluidity of rainwater prevents it from adhering to the camera lens, it will still leave watermarks on the lens, and there is still a certain probability that the camera will be mistakenly identified as dirty; in sunny weather, the probability of the camera being mistakenly identified as dirty due to weather conditions is relatively small.

[0105] In practice, the correspondence between weather conditions and dirt probability thresholds can be pre-defined based on the false detection rate of dirt under different weather conditions. Then, the dirt probability threshold corresponding to the target weather condition can be selected from the pre-defined correspondence between weather conditions and dirt probability thresholds.

[0106] Optionally, the relationship between weather conditions and the dirt probability threshold can be expressed by formula (6):

[0107]

[0108] Where, θ threshold P1, P2, and P3 represent different weather conditions; θ1, θ2, and θ3 represent the dirt probability thresholds corresponding to different weather conditions. Optionally, the false detection rates of dirt corresponding to the weather conditions represented by P1, P2, and P3 are: false detection rate of dirt corresponding to P1 < false detection rate of dirt corresponding to P2 < false detection rate of dirt corresponding to P3, and correspondingly, θ1 < θ2 < θ3.

[0109] This application embodiment determines the correspondence between weather conditions and dirt probability thresholds based on the false detection rate of dirt corresponding to different weather conditions, and obtains a dirt probability threshold that corresponds to the target weather condition based on the correspondence between weather conditions and dirt probability thresholds, which can improve the accuracy of dirt detection of cameras based on dirt probability thresholds.

[0110] In one embodiment, step S103 above: determining the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold, includes:

[0111] If the probability of dirt is greater than or equal to the probability threshold of dirt, the dirt detection results corresponding to the target number of second images are obtained; the target number of second images are consecutive frame images captured by the target camera that are located before the first image.

[0112] If the dirt detection result corresponding to the second image of the target number indicates that the target camera is dirty, then the dirt detection result of the target camera is determined to be that the target camera is dirty.

[0113] Considering the risk of false positives in detecting dirt on a target camera based on a single frame image, an additional buffer for dirt detection results can be set up to store the most recent dirt detection results for the target camera. In practice, if the probability of dirt on the target camera determined based on the current frame image (i.e., the first image) is greater than or equal to a dirt probability threshold, the dirt detection results corresponding to the target number of second images can be read from the buffer. Then, if the dirt detection results corresponding to the target number of second images indicate that the target camera is dirty, the final determination of dirt on the target camera can be made, thereby improving the accuracy of dirt detection.

[0114] Optionally, the second image of the target number is a series of frame images captured by the target camera before the first image, which can characterize the second image of the target number as a series of frame images, and the latest captured image of the second image of the target number is a series of frame images with the first image;

[0115] Optionally, the number of targets in the second image can be determined according to the actual situation; the method for determining the dirt detection result corresponding to the second image can refer to the specific implementation process of steps S101 to S103 above, which will not be repeated here.

[0116] In this embodiment of the application, when the probability of dirt is greater than or equal to the probability threshold of dirt, the dirt detection result corresponding to a target number of second images is obtained, and when the dirt detection result corresponding to a target number of second images indicates that the target camera is dirty, the dirt detection result of the target camera is determined to be that the target camera is dirty, which can further improve the accuracy of dirt detection of the camera.

[0117] In one specific embodiment, the camera dirt detection method provided in this application can be applied to the camera dirt detection of autonomous vehicles. The autonomous vehicle can be equipped with multiple surround-view cameras, and the camera dirt detection method provided in this application can be used to detect dirt on each surround-view camera on the vehicle separately.

[0118] Please see Figure 2 The camera dirt detection method provided in this application embodiment may include the following steps:

[0119] Step S201: Construct the neural network model to be trained; a YOLOv4 convolutional network can be selected as the backbone network, and a dirt detection head for predicting dirt probability and a weather detection head for predicting weather conditions can be designed on the basis of the YOLOv4 convolutional backbone network to obtain the neural network model to be trained.

[0120] Step S202: Obtain training samples; the first surround view image (equivalent to the above sample image) captured by each surround view camera can be obtained by subscribing to nodes through the vehicle's Robot Operating System (ROS). The first surround view images obtained by the ROS subscription nodes can be packaged to obtain a ROS bag. Optionally, the first surround view image can be an RGB format image. Then, based on the node name corresponding to each surround view camera in the ROS bag, multiple sets of first surround view images with aligned timestamps can be extracted. Then, for each set of first surround view images, the labelme annotation tool can be used to annotate the first surround view images. The annotated information can include dirt label box information, weather status label results, and dirt label probability. Finally, data augmentation operations such as translation and flipping can be performed on the annotated first surround view images to expand the number of images in the training samples and ensure the training generalization of the neural network model to be trained.

[0121] Step S203: Train the neural network model to be trained using training samples to obtain the trained target neural network model. The specific implementation process of step S203 can be found in the description of the above embodiments, and will not be repeated here.

[0122] Please see Figure 3 Given the target neural network model that has been trained as described above, the camera dirt detection method provided in this application embodiment may further include the following steps:

[0123] Step S301: Determine the image acquisition frequency of each surround-view camera of the vehicle based on the preset speed range of the vehicle's speed; optionally, the vehicle's speed can be acquired in real time using a vehicle speed sensor.

[0124] Step S302: Using each surround-view camera, acquire a second surround-view image (equivalent to the first image mentioned above) according to the image acquisition frequency described above; optionally, the second surround-view image acquired by each surround-view camera can be subscribed to through a ROS node.

[0125] Step S303: Input the second surround view image captured by each surround view camera into the trained target neural network model to obtain the dirt probability of each surround view camera; Optionally, for each surround view camera, input its captured second surround view image into the target neural network model, and output the dirt probability of the surround view camera based on the second surround view image through the dirt detection head of the target neural network model.

[0126] Step S304: Input the second panoramic images captured by each panoramic camera into the trained target neural network model to obtain the target weather state when each panoramic camera captures the second panoramic images; Optionally, for each panoramic camera, input the second panoramic images it captures into the target neural network model, and output the target weather state when the panoramic camera captures the second panoramic images based on the weather detection head of the target neural network model;

[0127] Step S305: Based on the target weather conditions when each surround view camera collects the second surround view image, determine the dirt probability threshold for each surround view camera: For each surround view camera, determine the dirt probability threshold that corresponds to the target weather conditions when it collects the second surround view image, according to the correspondence between the weather conditions and the dirt probability threshold.

[0128] Step S306: Based on the comparison between the dirt probability of each surround view camera and the dirt probability threshold, determine the dirt detection result of each surround view camera.

[0129] The specific implementation process of this embodiment can be referred to the description in the above embodiments, and will not be repeated here.

[0130] It is worth mentioning that if the dirt detection result of any surround view camera indicates that the camera is dirty, the dirt detection result of that surround view camera can be reported to the intelligent driving system that relies on that surround view camera, and the vehicle's driving safety can be ensured through functions such as function downgrading or function suspension.

[0131] The camera dirt detection method provided in this application constructs and trains a target neural network model that simultaneously possesses weather condition detection and dirt detection capabilities by adding a weather detection head and a dirt detection head to a classic detection network. This allows the model to quickly and accurately determine the weather condition at the time the first image was captured by the target camera by inputting the first image into the trained target neural network model. Similarly, the dirt detection head can quickly and accurately determine the dirt probability of the target camera based on the first image. Furthermore, by establishing a correspondence between weather conditions and a dirt probability threshold, a dirt probability threshold corresponding to the target weather condition is obtained, enabling adaptive adjustment of the dirt probability threshold. The dirt detection result of the target camera is determined by comparing the dirt probability with the adaptively adjusted dirt probability threshold, which can significantly reduce the false detection rate of dirt on the target camera. In addition, by determining the image acquisition frequency of the target camera based on the preset range of the vehicle's driving parameters, and using the target camera to acquire the first image according to the image acquisition frequency, the dirt detection method of the camera provided in this application embodiment can adapt to different driving scenarios, balance user experience and driving safety risks in different driving scenarios, and more accurately control the consumption of computing resources. Moreover, the adjustment of the dirt probability threshold and the image acquisition frequency does not rely on the Internet, realizing dynamic adjustment in offline state, which can ensure the real-time performance of dirt detection.

[0132] Please see Figure 4 This application embodiment also provides a camera dirt detection device 400, which can implement the above-mentioned camera dirt detection method. The device 400 includes a dirt detection device including an acquisition module 401, a dirt probability threshold module 402 and a dirt detection module 403.

[0133] The acquisition module 401 is used to respond to the target camera acquiring a first image, determine the probability of dirtiness of the target camera based on the first image, and acquire the target weather state when the target camera acquires the first image; wherein, the probability of dirtiness is used to characterize the probability of dirtiness in the target camera.

[0134] The dirt probability threshold module 402 is used to determine the dirt probability threshold corresponding to the target weather state;

[0135] The dirt detection module 403 is used to determine the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold.

[0136] The camera dirt detection device provided in this application embodiment can implement all the steps of the above-described camera dirt detection method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0137] This application also provides an electronic device, including a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the various steps of the above-described camera dirt detection method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0138] Figure 5 To illustrate the hardware structure of the electronic device according to the embodiments of this application, the electronic device includes:

[0139] The processor 501 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0140] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called by the processor 501 to execute the camera dirt detection method of the embodiments of this application.

[0141] The input / output interface 503 is used to implement information input and output;

[0142] The communication interface 504 is used to enable communication and interaction between this electronic device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0143] Bus 505 transmits information between various components of an electronic device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0144] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the electronic device via bus 505.

[0145] The electronic device provided in this application embodiment can implement all the steps of the above-described camera dirt detection method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0146] This application also provides a computer-readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various steps of the above-described camera dirt detection method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0147] The processor is the processor in the electronic device described in the above embodiments. The computer-readable storage medium includes computer-readable storage media such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0148] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various steps of the above-described camera dirt detection method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0150] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various steps of the above-described camera dirt detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0151] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not delete other identical elements present in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0153] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for detecting dirt in a camera, characterized in that, include: Training samples are input into the neural network model to be trained to obtain the dirt prediction probability and weather state prediction results corresponding to the sample images. The training samples include the sample images and the dirt label probability and weather state label results corresponding to the sample images. Based on the weather state prediction results and the weather state labels, a first loss function value of the neural network model to be trained is determined, and a second loss function value of the neural network model to be trained is determined based on the dirt prediction probability and the dirt label probability. A target loss function value is determined based on the first loss function value and the second loss function value. Based on the target loss function value, the network parameters of the first prediction network and the second prediction network in the neural network model to be trained are updated to train the target neural network model. In response to the target camera acquiring a first image, the first image is input into the trained target neural network model. Based on the first image, the first prediction network outputs the dirt probability of the target camera, and the second prediction network outputs the target weather state based on the first image; wherein, the dirt probability is used to characterize the probability that the target camera is dirty. Determine the dirt probability threshold corresponding to the target weather state; The dirt detection result of the target camera is determined based on the comparison between the dirt probability and the dirt probability threshold.

2. The dirt detection method as described in claim 1, characterized in that, The training samples also include dirt label box information corresponding to the sample images; The step of determining the target loss function value based on the first loss function value and the second loss function value includes: The third loss function value of the neural network model to be trained is determined based on the dirt prediction box information and the dirt label box information corresponding to the sample image; the dirt prediction box information is obtained by inputting the training sample into the neural network model to be trained. The target loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value.

3. The dirt detection method as described in claim 1, characterized in that, The target camera is mounted on the vehicle; The dirt detection method also includes: The image acquisition frequency of the target camera is determined based on the preset range of the vehicle's driving parameters; The first image is acquired using the target camera at the specified image acquisition frequency.

4. The dirt detection method as described in claim 1, characterized in that, Determining the dirt probability threshold corresponding to the target weather state includes: Based on the correspondence between weather conditions and dirt probability thresholds, a dirt probability threshold corresponding to the target weather condition is obtained. The correspondence is determined based on the false detection rate of dirt and grime under different weather conditions.

5. The dirt detection method as described in claim 1, characterized in that, Determining the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold includes: If the dirt probability is greater than or equal to the dirt probability threshold, the dirt detection results corresponding to the target number of second images are obtained; the target number of second images are consecutive frame images captured by the target camera that are located before the first image. If the dirt detection result corresponding to the second image of the target number indicates that the target camera is dirty, then the dirt detection result of the target camera is determined to be that the target camera is dirty.

6. A camera dirt detection device, employing the camera dirt detection method as described in claim 1, characterized in that, The dirt detection device includes an acquisition module, a dirt probability threshold module, and a dirt detection module. The acquisition module is configured to, in response to the target camera acquiring a first image, determine the probability of dirtiness of the target camera based on the first image, and acquire the target weather state when the target camera acquired the first image; wherein, the probability of dirtiness is used to characterize the probability that the target camera is dirty; The dirt probability threshold module is used to determine the dirt probability threshold corresponding to the target weather state; The dirt detection module is used to determine the dirt detection result of the target camera based on the comparison result between the dirt probability and the dirt probability threshold.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the dirt detection method for the camera as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the dirt detection method for the camera as described in any one of claims 1 to 5.

Images