Image acquisition device shielding detection method and device, vehicle, medium and program product

Abstract

The invention relates to an image acquisition device shielding detection method and device, a vehicle, a medium and a program product in the technical field of automatic driving. The method comprises the steps that a to-be-detected image acquired by an image acquisition device is acquired; the image to be detected is input into a detection model, a detection result that whether the image acquisition device is shielded or not is obtained, the detection model comprises a plurality of detection sub-models, and when sample images of different detection sub-models are collected, illumination intensity and image acquisition device shielding conditions are different. Therefore, the accuracy of detecting whether the image acquisition device is shielded or not can be improved, the risk of misinformation of shielding of the image acquisition device is reduced, and the user experience is improved. Meanwhile, instability of the automatic driving system caused by false alarm is avoided, and the safety of the automatic driving system is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an image acquisition device occlusion detection method, device, vehicle, medium, and program product. Background Art

[0002] Autonomous driving relies on various onboard sensors, such as cameras, millimeter-wave radar, and lidar. These sensors acquire information about the external environment, use it to make decisions, and issue driving commands. Cameras are the input data for perception algorithms. Any anomalies with onboard cameras can directly impact the functionality of the perception algorithms and the safety of autonomous driving. For example, if a camera is obstructed and the user is not promptly notified to inspect and clear the camera, the perception algorithm will not function, posing a high risk of causing a vehicle accident. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an image acquisition device occlusion detection method, device, vehicle, medium and program product.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for detecting occlusion of an image acquisition device is provided, comprising: Acquiring the image to be detected captured by the image acquisition device; The image to be detected is input into the detection model to obtain a detection result of whether the image acquisition device is blocked, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device blocking conditions are different when collecting sample images of different detection sub-models.

[0005] Optionally, inputting the image to be detected into a detection model to obtain a detection result of whether the image acquisition device is blocked includes: Inputting the image to be detected into the first detection sub-model of the detection model, obtaining a first detection result output by the first detection sub-model as to whether the image acquisition device is blocked; If the first detection result indicates that the image acquisition device is blocked, inputting the image to be detected into the second detection sub-model of the detection model to obtain a second detection result of whether the illumination intensity exceeds a preset threshold when the second detection sub-model outputs the image to be detected; According to the second detection result, the detection result of whether the image acquisition device is blocked is obtained.

[0006] Optionally, obtaining the detection result of whether the image acquisition device is blocked according to the second detection result includes: If the second detection result indicates that the light intensity does not exceed the preset threshold when the image to be detected is collected, determining whether a night condition is met; When the dark night condition is met, inputting the image to be detected into the third detection sub-model of the detection model to obtain a detection result output by the third detection sub-model as to whether the image acquisition device is blocked; Among them, the sample images for training the first detection sub-model are collected by the image acquisition device under obscured conditions and the light intensity exceeds the preset threshold, and the sample images for training the third detection sub-model are collected by the image acquisition device under unobstructed conditions and the light intensity does not exceed the preset threshold.

[0007] Optionally, the first detection sub-model and the third detection sub-model each include a plurality of unit models, each of the unit models independently taking the image to be detected as input and outputting a detection result of whether the image acquisition device is blocked; The detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by each unit model.

[0008] Optionally, the detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by each of the unit models, including: Determining, based on the accuracy of each unit model for the detection results of different categories, a confidence level corresponding to each unit model and a confidence level threshold corresponding to the detection results of each category; Determining a target unit model according to the confidence level corresponding to each unit model and the corresponding confidence level threshold; The detection result of whether the image acquisition device is blocked output by the detection sub-model is obtained according to the detection result of whether the image acquisition device is blocked output by the target unit model.

[0009] Optionally, determining the confidence corresponding to each unit model and the confidence threshold corresponding to each category of the detection results based on the accuracy of each unit model for the detection results of different categories includes: Determining the total accuracy of the detection results of each category according to the accuracy of the detection results of each unit model for different categories; Determining the confidence level corresponding to each unit model according to the accuracy of the detection results of each category corresponding to each unit model and the total accuracy rate corresponding to the category; The confidence mean corresponding to each unit model in the category is used as the confidence threshold corresponding to the detection result of the category.

[0010] Optionally, obtaining the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by the target unit model includes: Determining the total number of the target unit models in the detection sub-model; The detection result of whether the image acquisition device is blocked output by the detection sub-model is determined based on the ratio of the number of the target unit models that output the detection result that the image acquisition device is blocked to the total number.

[0011] Optionally, determining the target unit model according to the confidence level corresponding to each unit model and the corresponding confidence level threshold includes: In a case where the confidence corresponding to the unit model is greater than the confidence threshold of the corresponding category, the unit model is determined to be the target unit model.

[0012] Optionally, if the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, determining whether a night condition is met includes: If the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, obtaining a collection time of collecting the image to be detected; Whether the darkness condition is met is determined based on the collection time and a preset time threshold.

[0013] Optionally, obtaining the detection result of whether the image acquisition device is blocked according to the second detection result includes: If the second detection result indicates that the light intensity exceeds the preset threshold when the image to be detected is collected, the detection result that the image acquisition device is blocked is obtained.

[0014] Optionally, the sample images for training the second detection sub-model include a first sample image captured by the image acquisition device under unobstructed conditions and with the light intensity exceeding the preset threshold, and a second sample image captured by the image acquisition device under unobstructed conditions and with the light intensity not exceeding the preset threshold.

[0015] According to a second aspect of an embodiment of the present disclosure, there is provided an image acquisition device occlusion detection device, comprising: an acquisition module, configured to acquire the image to be detected acquired by the image acquisition device; The determination module is configured to input the image to be detected into the detection model to obtain a detection result of whether the image acquisition device is blocked, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device blocking conditions are different when collecting sample images of different detection sub-models.

[0016] According to a third aspect of an embodiment of the present disclosure, there is provided a vehicle, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement any one of the methods described in the first aspect.

[0017] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, which implements the steps of any one of the methods described in the first aspect when executed by a processor.

[0018] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of any one of the methods in the first aspect when executed by a processor.

[0019] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects: The method involves obtaining an image to be detected captured by the image acquisition device; inputting the image to be detected into a detection model to obtain a detection result indicating whether the image acquisition device is obscured, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device obscuration conditions are different when collecting sample images for different detection sub-models. The multiple detection sub-models are trained using different illumination intensities and obscuration conditions, and can accurately determine whether the image acquisition device is obscured in various complex environments. By taking into account the effects of different illumination intensities and obscuration conditions, a high detection accuracy rate can be maintained even in poor lighting conditions or when obscurations are relatively hidden. This improves the accuracy of detecting whether the image acquisition device is obscured, reduces the risk of false alarms indicating that the image acquisition device is obscured, and enhances the user experience. At the same time, it avoids instability of the autonomous driving system due to false alarms, thereby improving the safety of the autonomous driving system.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 The present invention is a flowchart of an occlusion detection method for an image acquisition device according to an exemplary embodiment.

[0023] Figure 2 An implementation according to an exemplary embodiment is shown Figure 1 Flowchart of step S12 in FIG.

[0024] Figure 3 An implementation according to an exemplary embodiment is shown Figure 2 Flowchart of step S123 in FIG.

[0025] Figure 4 The figure is a flowchart showing how a detection sub-model determines a detection result according to an exemplary embodiment.

[0026] Figure 5 An implementation according to an exemplary embodiment is shown Figure 4 Flowchart of step S51 in FIG.

[0027] Figure 6 The figure is a flow chart showing another method for detecting occlusion in an image acquisition device according to an exemplary embodiment.

[0028] Figure 7 The present invention is a block diagram of an occlusion detection device for an image acquisition device according to an exemplary embodiment.

[0029] Figure 8 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0031] The embodiments described in the following examples of the present disclosure do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0033] Before introducing the image acquisition device occlusion detection method, device, vehicle, medium, and program product provided by the embodiments of the present disclosure, the following related technologies in related scenarios are introduced. In the related scenarios, camera occlusion is detected based on a deep learning model. Since the image performance difference between some abnormal scenes and normal scenes is small, false alarms may occur. In particular, the images captured by the camera in low-light conditions such as basements and at night are very similar to the images captured in occluded conditions, which can easily cause misrecognition of the model, resulting in false alarms of camera occlusion when driving in low-light conditions, false alarms of obstacles that cannot be passed, or forced exit from the autonomous driving system, resulting in a reduced user experience.

[0034] In view of this, the present disclosure provides an image acquisition device occlusion detection method, which aims to solve the problem of false alarm of image acquisition device occlusion caused by insufficient light in basements, at night, etc., thereby improving the stability of vehicles such as automatic driving and obstacle detection, and further enhancing the user experience.

[0035] Figure 1 FIG. 1 is a flow chart showing an occlusion detection method for an image acquisition device according to an exemplary embodiment. Figure 1 As shown, the method can be applied to an automatic driving system of a vehicle and includes the following steps.

[0036] In step S11, the image to be detected acquired by the image acquisition device is acquired.

[0037] Among them, the image acquisition device can be a camera installed on the vehicle facing the vehicle's driving environment, used to collect environmental images in the surrounding environment of the vehicle, for example, continuously capturing environmental images in front of the vehicle, including roads, pedestrians, other vehicles, road traffic signs, etc.

[0038] It can also be a camera installed on the vehicle and facing the interior of the vehicle, used to collect images of the interior environment of the vehicle, such as the driver's status, the passengers' conditions, etc.

[0039] In step S12, the image to be detected is input into the detection model to obtain a detection result of whether the image acquisition device is blocked, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device blocking conditions are different when collecting sample images of different detection sub-models.

[0040] Among them, multiple detection sub-models all take the image to be detected as input, but the multiple detection sub-models are set in a linear order, and the output of the previous detection sub-model is the trigger condition for whether the next detection sub-model needs to perform image recognition. For example, when the output of the previous detection sub-model meets the enabling conditions of the next detection sub-model, the next detection sub-model takes the image to be detected as input and performs classification and recognition of the image to be detected. In other words, except for the first detection sub-model which only takes the image to be detected as input, the other detection sub-models all take the output of the previous detection sub-model and the image to be detected as input, and the output of the previous detection sub-model is used as input to determine the trigger condition for performing image recognition, and the image to be detected is used as input for image recognition.

[0041] In the disclosed embodiment, when an image to be detected is input into the first sub-model, it performs a preliminary analysis of the image based on its trained parameters and algorithm and outputs a result. This result may be a classification label (such as "unoccluded," "partially occluded," or "completely occluded") or a numerical value (such as the probability or degree of occlusion). The system then determines whether the conditions for enabling the next sub-model are met based on this output. If so, the image to be detected is passed to the next sub-model for further analysis. If not, the sub-model is skipped and the final detection result is directly output.

[0042] It can be explained that the sample images used in training different detection sub-models are collected under different lighting intensities and image acquisition device occlusion conditions. In this way, the detection sub-model can detect whether the image acquisition device is blocked based on different lighting intensities and different occlusion conditions, thereby improving its applicability in various complex environments and reducing the risk of false alarms due to occlusion of the image acquisition device.

[0043] In the embodiment of the present disclosure, when it is determined that the image acquisition device is blocked, it can be reported to the vehicle system in a timely manner to inform the user that the image acquisition device is blocked, and the user can be reminded to clean the image acquisition device, or the automatic cleaning device configured on the vehicle can be started to clean the image acquisition device.

[0044] The above technical solution obtains the image to be detected captured by the image acquisition device; inputs the image to be detected into the detection model to obtain a detection result of whether the image acquisition device is obscured, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device obscuration conditions are different when collecting sample images of different detection sub-models. The multiple detection sub-models are trained with different illumination intensities and obscuration conditions, and can accurately determine whether the image acquisition device is obscured in various complex environments. Because the influence of different illumination intensities and obscuration conditions is taken into account, a high detection accuracy rate can be maintained even in poor lighting conditions or when the obstruction is relatively hidden. This can improve the accuracy of detecting whether the image acquisition device is obscured, reduce the risk of false alarms of the image acquisition device being obscured, and enhance the user experience. At the same time, it avoids instability of the autonomous driving system due to false alarms, thereby improving the safety of the autonomous driving system.

[0045] Alternatively, see Figure 2 As shown, in step S12, the image to be detected is input into the detection model to obtain a detection result of whether the image acquisition device is blocked, including: In step S121, the image to be detected is input into the first detection sub-model of the detection model, and the first detection sub-model outputs a first detection result of whether the image acquisition device is blocked; The first detection sub-model is the initial stage of the detection model. Its primary task is to preliminarily determine whether the image acquisition device (e.g., camera) is obstructed. This sub-model may use computer vision techniques such as image segmentation, edge detection, and object recognition to identify the presence of obstructions in the image. It performs feature extraction and pattern recognition on the input image to be detected, outputting a preliminary determination of whether the image acquisition device is obstructed.

[0046] For example, the first detection sub-model can be a deep learning-based object detection model that has been trained to identify occluders (such as dust, raindrops, and other objects) in images. When an image to be detected (such as an image of the road environment in front of the vehicle) is input into this sub-model, the model extracts image features such as color, texture, and shape and compares them with the occluder features learned during training. If the model determines that an occluder is present in the image and that it may be blocking the camera, it outputs an "occluded" result.

[0047] In step S122, if the first detection result indicates that the image acquisition device is blocked, the image to be detected is input into the second detection sub-model of the detection model to obtain a second detection result of whether the illumination intensity exceeds a preset threshold when the second detection sub-model outputs the image to be detected; In the disclosed embodiment, if the first detection sub-model determines that the image acquisition device is obscured, the system needs to further determine whether the obscuration is caused by insufficient or excessive light intensity. Therefore, the main task of the second detection sub-model is to analyze the light intensity in the image and determine whether it exceeds a preset threshold. This sub-model may use image processing techniques such as histogram equalization and light estimation to accurately estimate the light intensity in the image.

[0048] For example, if the first detection sub-model outputs an "occluded" result, the system will input the image to be detected into the second detection sub-model. This sub-model will analyze the brightness distribution and contrast in the image to estimate the light intensity. It may calculate the brightness histogram of the image or use other light estimation algorithms to obtain a specific value for the light intensity. This value is then compared with a preset light intensity threshold. If the light intensity is lower than the lower threshold (indicating insufficient light) or higher than the upper threshold (indicating excessive light), the second detection sub-model will output a "lighting anomaly" result.

[0049] In step S123, the detection result of whether the image acquisition device is blocked is obtained according to the second detection result.

[0050] In the embodiment of the present disclosure, after obtaining the result of the second detection sub-model, a comprehensive judgment can be made based on this result whether the image acquisition device is actually blocked. If the second detection sub-model judges that the light intensity is normal (i.e., exceeds the preset threshold), then the system will consider the "blocked" result of the first detection sub-model to be accurate, and thus ultimately judge that the image acquisition device is blocked. If the second detection sub-model judges that the light intensity is abnormal (i.e., does not exceed the preset threshold), then it can be considered that such blockage may be caused by lighting conditions rather than a real obstruction. Therefore, in this case, a result of "blockage caused by abnormal lighting" may be output instead of simply judging it as "blocked".

[0051] For example, if the second detection sub-model outputs a "normal lighting" result, the system will assume that the first detection sub-model's "occluded" result is accurate. Therefore, it will ultimately determine that the image acquisition device (e.g., the camera in front of the vehicle) is blocked, prompting the system to clean the image acquisition device (e.g., the camera in front of the vehicle). This may mean that the camera needs cleaning or repair. However, if the second detection sub-model outputs a "abnormal lighting" result, the system will assume that the obstruction is likely due to lighting conditions rather than a true obstruction.

[0052] Alternatively, see Figure 3As shown, in step S123, obtaining the detection result of whether the image acquisition device is blocked according to the second detection result includes: In step S1231, if the second detection result indicates that the light intensity does not exceed the preset threshold when the image to be detected is collected, determining whether a dark night condition is met; When determining whether the image acquisition device is obscured, if the second detection sub-model analyzes that the light intensity when capturing the image to be detected does not exceed the preset threshold, two possible scenarios may exist: one is that the actual ambient light is indeed insufficient (such as dusk, dawn, or cloudy days), and the other is that although the ambient light is sufficient, the captured image appears dark for some reason (such as camera malfunction, lens contamination, etc.). To distinguish between these two situations, it is necessary to further determine whether the current night condition is met.

[0053] In the disclosed embodiments, determining whether a dark environment is satisfied typically relies on the timestamp of the captured image and a preset time threshold (set based on local latitude and longitude and season). If the capture time exceeds the time threshold, the current environment is considered dark; otherwise, it is considered daytime.

[0054] In step S1232, when the dark night condition is met, the image to be detected is input into the third detection sub-model of the detection model to obtain a detection result of whether the image acquisition device is blocked, which is output by the third detection sub-model.

[0055] Among them, the sample images for training the first detection sub-model are collected by the image acquisition device under obscured conditions and the light intensity exceeds the preset threshold, and the sample images for training the third detection sub-model are collected by the image acquisition device under unobstructed conditions and the light intensity does not exceed the preset threshold.

[0056] In the disclosed embodiment, if it is determined that the dark condition is not currently met, the second detection sub-model can determine whether the image acquisition device is obscured. If it is determined that the dark condition is currently met, the image to be detected can be input into the third detection sub-model of the detection model to obtain a more accurate occlusion detection result. The third detection sub-model may be a model specifically optimized for image occlusion detection in daytime or nighttime environments. It can better process images in low-light conditions and improve the accuracy of occlusion detection.

[0057] For example, continuing with the above example, if the autonomous driving system determines that it is currently dark, it inputs the image to be detected into the third detection sub-model. This model is specially trained to identify possible obstructions in dark environments, such as artifacts caused by reflected light from headlights and road signs, as well as actual obstructions such as raindrops and snowflakes. After analysis, the third detection sub-model outputs a judgment result: whether the image acquisition device (front camera) is obstructed. If the result indicates obstruction, the system can take appropriate warning measures. If the image acquisition device (front camera) is not obstructed, the calculation ends.

[0058] For example, in a case of sufficient light (such as daytime), the first detection sub-model training is performed using the first image captured when the camera is blocked, wherein: , x is the first sample image, and the first detection sub-model outputs the first model classification results of unobstructed and obstructed images. In the case of insufficient light (such as at night), the third detection sub-model is trained using the second image captured when the camera is not obstructed, where , x is the second sample image to obtain the unoccluded and occluded third model classification results.

[0059] Optionally, the first detection sub-model and the third detection sub-model each include a plurality of unit models, each of the unit models independently taking the image to be detected as input and outputting a detection result of whether the image acquisition device is blocked; The unit models are the fundamental components of the detection sub-model. Each unit model can independently process the image to be detected and output its own detection results. This independence between the unit models ensures that even if one model misjudges an image, it will not directly affect other models, thereby improving the robustness of the overall system.

[0060] The detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by each unit model.

[0061] The detection sub-model relies not only on the output of a single unit model but also integrates the outputs of multiple unit models through specific strategies or algorithms. This integration process may involve weighting, voting, or other forms of fusion of the unit model outputs. By integrating the results of multiple unit models, the detection sub-model can more comprehensively evaluate the image being detected, reducing the possibility of false positives and missed detections.

[0062] Alternatively, see Figure 4As shown, the detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by each of the unit models, including: In step S51, according to the accuracy of each unit model for the detection results of different categories, the confidence corresponding to each unit model and the confidence threshold corresponding to each category of the detection results are determined; Among them, taking the two categories of image acquisition device without occlusion and occlusion as an example, the accuracy index of each unit model in each category is ,in Represents the accuracy of the i-th category of the k-th unit model. When there are only two categories in this task, when i is 0, it means that the image acquisition device is not blocked, and when i is 1, it means that the image acquisition device is blocked.

[0063] In step S52, a target unit model is determined according to the confidence level corresponding to each unit model and the corresponding confidence level threshold; It can be understood that if the confidence corresponding to the unit model is greater than the corresponding confidence threshold, it can be determined that the unit model is the target unit model; if the confidence corresponding to the unit model is less than or equal to the corresponding confidence threshold, it can be determined that the unit model is not the target unit model.

[0064] In step S53, the detection result of whether the image acquisition device is blocked output by the detection sub-model is obtained according to the detection result of whether the image acquisition device is blocked output by the target unit model.

[0065] In the disclosed embodiment, the detection results output by the target unit model can be voted, and the detection result of the target unit model that ultimately receives the most votes will be used as the detection result of the corresponding detection sub-model. If the detection results are the same, the votes of the target unit models can be combined and calculated.

[0066] For example, there are 5 unit models in the first detection sub-model, among which there are 3 unit models whose confidence of output detection results is greater than the confidence threshold, and the detection results of 2 unit models are that the image acquisition device is blocked, and the detection result of 1 unit model is that the image acquisition device is not blocked. Then, voting can be performed among these 3 models, for example, random voting is performed according to a preset number of votes (for example, 100 votes). Finally, the two target unit models whose detection results are that the image acquisition device is blocked received 46 and 40 votes respectively, and the target unit models whose detection results are that the image acquisition device is not blocked received 14 votes respectively. Since the target unit model whose image acquisition device is blocked received a total of 86 votes, the detection sub-model outputs the detection result that the image acquisition device is blocked.

[0067] Alternatively, see Figure 5 As shown, in step S51, the confidence corresponding to each unit model and the confidence threshold corresponding to each category of the detection result are determined based on the accuracy of each unit model for the detection results of different categories, including: In step S511, the total accuracy of the detection results of each category is determined according to the accuracy of the detection results of each unit model for different categories; It can be understood that the total accuracy of the detection results of each category is determined by calculating the sum of the accuracy of the unit models in the detection results of each category.

[0068] In step S512, the confidence level corresponding to each unit model is determined according to the accuracy of the detection result of each category corresponding to each unit model and the total accuracy of the category; In the above embodiment, the confidence of each unit model in each category is , Represents the confidence of the i-th category of the k-th model. is the total accuracy of the unit model corresponding to each category.

[0069] In step S513, the confidence mean corresponding to each of the unit models in the category is used as the confidence threshold corresponding to the detection result of the category.

[0070] In the embodiment of the present disclosure, the sum of the confidences corresponding to the unit models in the category can be calculated, and then the confidence mean can be determined based on the number of unit models in the category to obtain the confidence threshold corresponding to the detection result of the category.

[0071] Among them, following the above embodiment, the confidence threshold corresponding to each category is .

[0072] Optionally, in step S53, obtaining the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by the target unit model includes: Determining the total number of the target unit models in the detection sub-model; It can be understood that the total number of target unit models in the detection sub-model is obtained by counting the number of target unit models in the detection sub-model.

[0073] The detection result of whether the image acquisition device is blocked output by the detection sub-model is determined based on the ratio of the number of the target unit models that output the detection result that the image acquisition device is blocked to the total number.

[0074] In the embodiment of the present disclosure, it is possible to determine the ratio of the number of target unit models whose detection result is that the image acquisition device is blocked to the total number, and to determine the ratio of the number of target unit models whose detection result is that the image acquisition device is not blocked to the total number, and then determine the detection result of whether the image acquisition device output by the detection sub-model is blocked based on these two ratios.

[0075] In other words, the larger of the ratio of the number of target unit models whose detection results are that the image acquisition device is occluded to the total number and the ratio of the number of target unit models whose detection results are that the image acquisition device is not occluded to the total number is used to determine the detection result output by the detection sub-model as to whether the image acquisition device is occluded. For example, there are 5 unit models in the first detection sub-model, of which 3 unit models have a confidence level greater than the confidence threshold for outputting detection results, while the detection results of 2 unit models are that the image acquisition device is occluded and the detection result of 1 unit model is that the image acquisition device is not occluded. Since the detection result that the image acquisition device is occluded accounts for a larger ratio, the detection result output by the first detection sub-model is that the image acquisition device is occluded.

[0076] Optionally, in step S52, determining the target unit model according to the confidence level corresponding to each unit model and the corresponding confidence level threshold includes: In a case where the confidence corresponding to the unit model is greater than the confidence threshold of the corresponding category, the unit model is determined to be the target unit model.

[0077] In the disclosed embodiment, if the confidence level corresponding to the unit model is less than or equal to the confidence level threshold of the corresponding category, it can be determined that the accuracy of the unit model is low, the recognition result of the image to be detected is inaccurate, and the detection result of the unit model can be discarded. If the confidence level corresponding to the unit model is greater than the confidence level threshold of the corresponding category, it can be determined that the accuracy of the unit model is high, the recognition result of the image to be detected is accurate, and the detection result of the unit model can be used as a reference for the final detection result.

[0078] In this way, the target unit model is screened through the confidence and confidence threshold of the unit model, avoiding the use of unit models with low accuracy in determining the detection results, which can improve the accuracy of the detection model output results.

[0079] Optionally, in step S1231, if the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, determining whether a dark night condition is met includes: If the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, obtaining a collection time of collecting the image to be detected; It is understandable that the image acquisition device usually has an acquisition timestamp when acquiring the image to be detected. Therefore, the acquisition timestamp of each image to be detected can be used as the acquisition moment of the image to be detected.

[0080] Whether the darkness condition is met is determined based on the collection time and a preset time threshold.

[0081] In the disclosed embodiment, a time threshold T can be set based on the sunset and darkness conditions in the city where the vehicle is operating. If the acquisition time of the image to be detected exceeds the time threshold T, it is considered to be a dark environment, and the night condition is determined to be met. If the acquisition time of the image to be detected does not exceed the time threshold T, it is considered to be a daytime environment, and the night condition is determined to be not met. The time threshold T is related to the longitude and latitude of the location and the seasons, and can be set based on the current longitude and latitude coordinates and the seasons.

[0082] See also Figure 6 As shown, after the image to be detected is acquired by the camera, the image to be detected is first input into the first detection sub-model, and then it is determined whether the detection result output by the first detection sub-model is that the camera is blocked; and if the detection result output by the first detection sub-model is that the camera is not blocked, a detection result that the camera is not blocked can be obtained.

[0083] Furthermore, if the detection result output by the first detection sub-model is that the camera is blocked, the image to be detected is input into the second detection sub-model, and then it is determined whether the detection result output by the second detection sub-model is night. If the detection result output by the second detection sub-model is not night, the detection result of the camera being blocked can be obtained.

[0084] Furthermore, if the detection result output by the second detection sub-model is night, it is determined whether the acquisition time of the image to be detected exceeds the time threshold. If the acquisition time of the image to be detected does not exceed the time threshold, a detection result of the camera being blocked can be obtained.

[0085] Furthermore, if the acquisition time of the image to be detected exceeds the time threshold, the image to be detected is input into the third detection sub-model, and then it is determined whether the detection result output by the third detection sub-model is that the camera is blocked.

[0086] By analyzing the light intensity at the time the image to be detected is captured and combining it with the image's timestamp, the system can intelligently determine whether the current environment meets darkness conditions. This judgment method not only relies on direct image information (light intensity) but also incorporates time information, making the judgment more accurate and reliable. Furthermore, by setting a time threshold T based on the location's longitude and latitude and the seasons, the system can adapt to changes in sunlight in different locations and seasons. This ensures that the system can accurately determine whether it is currently dark in various environments.

[0087] Optionally, in step S123, obtaining the detection result of whether the image acquisition device is blocked according to the second detection result includes: If the second detection result indicates that the light intensity exceeds the preset threshold when the image to be detected is collected, the detection result that the image acquisition device is blocked is obtained.

[0088] Among them, the second detection result indicates that the illumination intensity exceeds the preset threshold when the image to be detected is collected. It can be considered that when the image to be detected is collected, it is in an environment with sufficient light, such as during the day. At this time, since the first detection result indicates that the image acquisition device is blocked, it can be reasonably determined that the image acquisition device is blocked and not due to insufficient or abnormal lighting conditions, and thus a detection result of the image acquisition device being blocked can be obtained.

[0089] By combining the results of the first and second detection sub-models, this technical solution can more accurately determine whether the image acquisition device is obscured and distinguish whether the obscuration is caused by lighting conditions. This improves detection accuracy and reduces the risk of false positives due to obscured image acquisition devices.

[0090] Optionally, the sample images for training the second detection sub-model include a first sample image captured by the image acquisition device under unobstructed conditions and with the light intensity exceeding the preset threshold, and a second sample image captured by the image acquisition device under unobstructed conditions and with the light intensity not exceeding the preset threshold.

[0091] For example, in the case of sufficient light (daytime) and insufficient light (nighttime), the second detection sub-model training is performed respectively on the third image captured when the camera is not blocked, wherein, , x are sample images corresponding to sufficient light (daytime) and insufficient light (nighttime), and each time the third model classification results of sufficient light (daytime) and insufficient light (nighttime) can be output.

[0092] By training with sample images under two different lighting conditions (above the threshold and below), the second detection sub-model can learn the impact of changes in light intensity on image quality and how these changes affect the accuracy of occlusion detection. This enables the model to more accurately determine whether light intensity is abnormal in real environments, thus avoiding misjudgments due to lighting conditions.

[0093] The present disclosure also provides an image acquisition device occlusion detection device, see Figure 7 As shown, it includes: an acquisition module 710 and a determination module 720.

[0094] The acquisition module 710 is configured to acquire the image to be detected acquired by the image acquisition device; The determination module 720 is configured to input the image to be detected into the detection model to obtain a detection result of whether the image acquisition device is blocked, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device blocking conditions are different when collecting sample images of different detection sub-models.

[0095] Optionally, the determination module 720 includes: a first input submodule configured to input the image to be detected into a first detection submodel of the detection model, and obtain a first detection result output by the first detection submodel indicating whether the image acquisition device is blocked; A second input submodule is configured to input the image to be detected into the second detection submodel of the detection model if the first detection result indicates that the image acquisition device is blocked, and obtain a second detection result of whether the illumination intensity exceeds a preset threshold when the second detection submodel outputs the image to be detected; The determination submodule is configured to obtain the detection result of whether the image acquisition device is blocked according to the second detection result.

[0096] Optionally, the determining submodule is configured to: If the second detection result indicates that the light intensity does not exceed the preset threshold when the image to be detected is collected, determining whether a night condition is met; When the dark night condition is met, inputting the image to be detected into the third detection sub-model of the detection model to obtain a detection result output by the third detection sub-model as to whether the image acquisition device is blocked; Among them, the sample images for training the first detection sub-model are collected by the image acquisition device under obscured conditions and the light intensity exceeds the preset threshold, and the sample images for training the third detection sub-model are collected by the image acquisition device under unobstructed conditions and the light intensity does not exceed the preset threshold.

[0097] Optionally, the first detection sub-model and the third detection sub-model each include a plurality of unit models, each of the unit models independently taking the image to be detected as input and outputting a detection result of whether the image acquisition device is blocked; The detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by each unit model.

[0098] Optionally, the detection sub-model is configured as follows: Determining, based on the accuracy of each unit model for the detection results of different categories, a confidence level corresponding to each unit model and a confidence level threshold corresponding to the detection results of each category; Determining a target unit model according to the confidence level corresponding to each unit model and the corresponding confidence level threshold; The detection result of whether the image acquisition device is blocked output by the detection sub-model is obtained according to the detection result of whether the image acquisition device is blocked output by the target unit model.

[0099] Optionally, the detection sub-model is configured as follows: Determining the total accuracy of the detection results of each category according to the accuracy of the detection results of each unit model for different categories; Determining the confidence level corresponding to each unit model according to the accuracy of the detection results of each category corresponding to each unit model and the total accuracy rate corresponding to the category; The confidence mean corresponding to each unit model in the category is used as the confidence threshold corresponding to the detection result of the category.

[0100] Optionally, the detection sub-model is configured as follows: Determining the total number of the target unit models in the detection sub-model; The detection result of whether the image acquisition device is blocked output by the detection sub-model is determined based on the ratio of the number of the target unit models that output the detection result that the image acquisition device is blocked to the total number.

[0101] Optionally, the detection sub-model is configured as follows: In a case where the confidence corresponding to the unit model is greater than the confidence threshold of the corresponding category, the unit model is determined to be the target unit model.

[0102] Optionally, the determining submodule is configured to: If the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, obtaining a collection time of collecting the image to be detected; Whether the darkness condition is met is determined based on the collection time and a preset time threshold.

[0103] Optionally, the determining submodule is configured to: If the second detection result indicates that the light intensity exceeds the preset threshold when the image to be detected is collected, the detection result that the image acquisition device is blocked is obtained.

[0104] Optionally, the sample images for training the second detection sub-model include a first sample image captured by the image acquisition device under unobstructed conditions and with the light intensity exceeding the preset threshold, and a second sample image captured by the image acquisition device under unobstructed conditions and with the light intensity not exceeding the preset threshold.

[0105] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0106] The present disclosure also provides a vehicle, including: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement the method described in any one of the aforementioned embodiments.

[0107] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0108] An embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned embodiments when executed by a processor.

[0109] Figure 8 FIG6 is a block diagram illustrating a vehicle 600 according to an exemplary embodiment. For example, vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0110] Reference Figure 8 Vehicle 600 may include various subsystems, such as an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. Vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 600 may be interconnected via wired or wireless means.

[0111] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, a navigation system, and the like.

[0112] The perception system 620 may include several sensors for sensing information about the environment surrounding the vehicle 600. For example, the perception system 620 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0113] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0114] The drive system 640 may include components that provide power to the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0115] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652. The processor 651 may execute instructions 653 stored in the memory 652.

[0116] The processor 651 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0117] The memory 652 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0118] In addition to instructions 653 , memory 652 may also store data, such as road maps, route information, and vehicle location, direction, speed, etc. The data stored in memory 652 may be used by computing platform 650 .

[0119] In the embodiment of the present disclosure, the processor 651 may execute the instruction 653 to complete all or part of the steps of the above-mentioned image acquisition device occlusion detection method.

[0120] Although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. With particular regard to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. In addition, although particular features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include," "have," "have," "have," or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0121] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for detecting occlusion of an image acquisition device, characterized in that: include: Acquiring the image to be detected captured by the image acquisition device; The image to be detected is input into the detection model to obtain a detection result of whether the image acquisition device is blocked, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device blocking conditions are different when collecting sample images of different detection sub-models.

2. The method according to claim 1, characterized in that Inputting the image to be detected into the detection model to obtain a detection result of whether the image acquisition device is blocked includes: Inputting the image to be detected into the first detection sub-model of the detection model, obtaining a first detection result output by the first detection sub-model as to whether the image acquisition device is blocked; If the first detection result indicates that the image acquisition device is blocked, inputting the image to be detected into the second detection sub-model of the detection model to obtain a second detection result of whether the illumination intensity exceeds a preset threshold when the second detection sub-model outputs the image to be detected; According to the second detection result, the detection result of whether the image acquisition device is blocked is obtained.

3. The method according to claim 2, characterized in that Obtaining the detection result of whether the image acquisition device is blocked according to the second detection result includes: If the second detection result indicates that the light intensity does not exceed the preset threshold when the image to be detected is collected, determining whether a night condition is met; When the dark night condition is met, inputting the image to be detected into the third detection sub-model of the detection model to obtain a detection result output by the third detection sub-model as to whether the image acquisition device is blocked; Among them, the sample images for training the first detection sub-model are collected by the image acquisition device under obscured conditions and the light intensity exceeds the preset threshold, and the sample images for training the third detection sub-model are collected by the image acquisition device under unobstructed conditions and the light intensity does not exceed the preset threshold.

4. The method according to claim 3, characterized in that The first detection sub-model and the third detection sub-model each include a plurality of unit models, each of the unit models independently taking the image to be detected as input and outputting a detection result of whether the image acquisition device is blocked; The detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model based on the detection result of whether the image acquisition device is blocked output by each unit model.

5. The method according to claim 4, characterized in that The detection sub-model obtains the detection result of whether the image acquisition device is blocked output by the detection sub-model according to the detection result of whether the image acquisition device is blocked output by each of the unit models, including: Determining, based on the accuracy of each unit model for the detection results of different categories, a confidence level corresponding to each unit model and a confidence level threshold corresponding to the detection results of each category; Determining a target unit model according to the confidence level corresponding to each unit model and the corresponding confidence level threshold; The detection result of whether the image acquisition device is blocked output by the detection sub-model is obtained according to the detection result of whether the image acquisition device is blocked output by the target unit model.

6. The method according to claim 5, characterized in that Determining the confidence level corresponding to each unit model and the confidence level threshold corresponding to each category of the detection results based on the accuracy of each unit model for the detection results of different categories includes: Determining the total accuracy of the detection results of each category according to the accuracy of the detection results of each unit model for different categories; Determining the confidence level corresponding to each unit model according to the accuracy of the detection results of each category corresponding to each unit model and the total accuracy rate corresponding to the category; The confidence mean corresponding to each unit model in the category is used as the confidence threshold corresponding to the detection result of the category.

7. The method according to claim 5, characterized in that The obtaining, based on the detection result of whether the image acquisition device is blocked output by the target unit model, the detection result of whether the image acquisition device is blocked output by the detection sub-model, includes: Determining the total number of the target unit models in the detection sub-model; The detection result of whether the image acquisition device is blocked output by the detection sub-model is determined based on the ratio of the number of the target unit models that output the detection result that the image acquisition device is blocked to the total number.

8. The method according to claim 5, characterized in that Determining the target unit model according to the confidence level corresponding to each unit model and the corresponding confidence level threshold includes: In a case where the confidence corresponding to the unit model is greater than the confidence threshold of the corresponding category, the unit model is determined to be the target unit model.

9. The method according to claim 3, characterized in that If the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, determining whether the night condition is met includes: If the second detection result indicates that the light intensity is insufficient when the image to be detected is collected, obtaining a collection time of collecting the image to be detected; Whether the darkness condition is met is determined based on the collection time and a preset time threshold.

10. The method according to any one of claims 2 to 9, characterized in that Obtaining the detection result of whether the image acquisition device is blocked according to the second detection result includes: If the second detection result indicates that the light intensity exceeds the preset threshold when the image to be detected is collected, the detection result that the image acquisition device is blocked is obtained.

11. The method according to any one of claims 2 to 9, characterized in that The sample images for training the second detection sub-model include a first sample image captured by the image acquisition device under unobstructed conditions and with the light intensity exceeding the preset threshold, and a second sample image captured by the image acquisition device under unobstructed conditions and with the light intensity not exceeding the preset threshold.

12. An image acquisition device occlusion detection device, characterized in that: include: an acquisition module, configured to acquire the image to be detected acquired by the image acquisition device; The determination module is configured to input the image to be detected into the detection model to obtain a detection result of whether the image acquisition device is blocked, wherein the detection model includes multiple detection sub-models, and the illumination intensity and image acquisition device blocking conditions are different when collecting sample images of different detection sub-models.

13. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 11 when the computer program is executed by a processor.

Images