Methods, devices, and vehicles for identifying abnormal states of vehicle-mounted camera lenses

Abstract

This invention provides a method, device, and vehicle for identifying abnormal states of an in-vehicle camera lens, relating to the fields of intelligent driving assistance systems and in-vehicle visual perception technology. The method includes: acquiring image frames captured by an in-vehicle camera; preprocessing the image frames to obtain a standard image; inputting the standard image into an abnormal state recognition model, wherein the abnormal state recognition model is used to extract features from the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, with each dimension component corresponding to an abnormal state, and independently activating and mapping each dimension component to output the confidence scores corresponding to multiple abnormal states; and determining the abnormal state recognition result of the in-vehicle camera lens based on the multiple confidence scores. This invention enables the simultaneous identification of multi-dimensional abnormal states of the lens and is adaptable to complex operating conditions.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving assistance systems and vehicle-mounted visual perception technology, and in particular to a method, device, and vehicle for identifying abnormal states of vehicle-mounted camera lenses. Background Technology

[0002] With the development of autonomous driving and intelligent driving assistance technologies, vehicle cameras, especially surround-view cameras, have become one of the important sensors in vehicle environmental perception systems. Vehicle cameras collect images of the vehicle's surrounding environment, providing basic perception information for functions such as target detection, obstacle recognition, parking assistance, surround-view stitching display, and assisted driving decision-making. Therefore, the reliability of the vehicle camera lens directly affects image quality and the accuracy of subsequent perception results. If the lens malfunctions, it can easily cause image distortion, leading to environmental perception failure or incorrect vehicle decisions, adversely affecting driving safety.

[0003] The inventors discovered that during actual vehicle operation, vehicle-mounted camera lenses may experience various abnormalities. For example, the lens may become completely or partially obstructed by foreign objects; it may become blurry due to dirt, fogging, or decreased optical performance; it may freeze in low temperatures; it may splash due to rain or mud; it may lose focus due to focusing system malfunctions or vibrations; it may be affected by backlighting or solar radiation from strong light; and it may even exhibit a "green image" phenomenon due to sensor, data transmission link, or image processing abnormalities. All of these abnormalities will reduce the effectiveness and reliability of the images output by the vehicle-mounted camera to varying degrees.

[0004] Further research by the inventors revealed that while existing technologies already include detection schemes for anomalies in vehicle-mounted camera lenses, they still suffer from the following shortcomings: First, existing schemes typically only identify a single type of anomaly, such as whether the lens is obstructed or dirty, which is insufficient to meet the multi-dimensional anomaly monitoring needs of surround-view vehicle-mounted cameras. Second, some schemes rely on the difference between visual perception and radar perception results for judgment, while LiDAR data is susceptible to environmental noise, and rasterization processing may also cause the loss of detailed information, leading to misidentification or missed identification. Furthermore, such schemes are highly dependent on specific perception architectures and have limited scalability. Third, some existing methods employ image pyramid construction and multi-raster region analysis, which, especially in vehicle-mounted scenarios with parallel processing of multiple surround-view cameras, suffer from high computational load and insufficient real-time performance, making it difficult to meet the real-time decision-making requirements of autonomous driving or assisted driving systems.

[0005] Therefore, there is an urgent need to provide a method for identifying abnormal states of vehicle-mounted camera lenses that can simultaneously identify multi-dimensional abnormal states of the lens and adapt to complex working conditions. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device, and vehicle for identifying abnormal states of vehicle-mounted camera lenses, in order to solve the aforementioned technical problems in the prior art.

[0007] On the one hand, in order to achieve the above objectives, the present invention provides a method for identifying abnormal states of vehicle-mounted camera lenses.

[0008] The method for identifying abnormal states of a vehicle-mounted camera lens includes: acquiring image frames captured by the vehicle-mounted camera; preprocessing the image frames to obtain a standard image; inputting the standard image into an abnormal state recognition model, wherein the abnormal state recognition model is used to extract features from the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, with each dimension component corresponding to an abnormal state, and independently activating and mapping each dimension component to output the confidence scores corresponding to multiple abnormal states; and determining the abnormal state recognition result of the vehicle-mounted camera lens based on the multiple confidence scores.

[0009] Furthermore, the abnormal state recognition model includes a feature extraction network, a feature mapping layer, and an activation layer. The steps of inputting a standard image into the abnormal state recognition model include: inputting the standard image into the feature extraction network, wherein the feature extraction network is used to extract features from the standard image and output a global feature vector; inputting the global feature vector into the feature mapping layer, wherein the feature mapping layer is used to map the global feature vector into a multi-dimensional initial prediction vector; and inputting the multi-dimensional initial prediction vector into the activation layer, wherein the activation layer is used to perform independent activation mapping on each dimension component to output the confidence level corresponding to each abnormal state.

[0010] Furthermore, several anomalous states include full occlusion, partial occlusion, blur, icing, splashing, out of focus, backlighting, sun rays, and green map.

[0011] Furthermore, the feature extraction network adopts the ResNet18 residual network, which is used to output a 512-dimensional global feature vector. The feature mapping layer is used to map the 512-dimensional global feature vector into a 9-dimensional initial prediction vector. The activation layer adopts the Sigmoid function, and Focal Loss is used as the training loss function during the training process of the abnormal state recognition model.

[0012] Furthermore, for a surround-view vehicle camera, after determining the abnormal state recognition result of the vehicle camera lens based on multiple confidence levels, the method further includes: when the abnormal state of the vehicle camera lens includes complete occlusion, icing, or a green map, outputting a prompt message to exit the perception function; when the abnormal state of the vehicle camera lens includes partial occlusion, out-of-focus, blur, or splashing, outputting a prompt message to check the lens state; and when the abnormal state of the vehicle camera lens includes backlighting or sunlight, outputting a prompt message that the current light is affecting camera performance.

[0013] Furthermore, the step of determining the abnormal state recognition result of the vehicle-mounted camera lens based on multiple confidence levels includes: presetting different confidence thresholds for different abnormal states; traversing multiple confidence levels and determining whether the confidence level is greater than or equal to the corresponding confidence threshold; when the confidence level is greater than or equal to the corresponding confidence threshold, determining that the first image recognition result includes the presence of an abnormal state corresponding to the confidence level in the image frame; when the confidence level is less than the corresponding confidence threshold, determining that the first image recognition result includes the absence of an abnormal state corresponding to the confidence level in the image frame; and determining the abnormal state recognition result based on the first image recognition result.

[0014] Further, the step of determining the abnormal state identification result based on the first image recognition result includes: traversing the abnormal states of the image frames identified by the first image recognition result, counting the number of image frames with abnormal states within the sliding time window where the image frame is located; determining whether the number of image frames is greater than or equal to a preset frame number threshold; when the number of image frames is greater than or equal to the preset frame number threshold, determining that the second image recognition result includes the presence of abnormal states in the image frames; when the number of image frames is less than the preset frame number threshold, determining that the second image recognition result includes the absence of abnormal states in the image frames; and determining the abnormal state identification result based on the second image recognition result.

[0015] Further, the step of determining the abnormal state identification result based on the second image recognition result includes: determining whether the second image recognition result satisfies the abnormal state association rules, wherein the abnormal state association rules include conflict rules and / or co-occurrence rules, the conflict rules are used to define abnormal states that conflict, and the co-occurrence rules are used to define abnormal states that occur simultaneously; and when the second image recognition result does not satisfy the abnormal state association rules, verifying the abnormal states with low confidence in the unsatisfied abnormal state association rules according to preset external parameters; and determining the abnormal state identification result based on the verification result.

[0016] On the other hand, in order to achieve the above objectives, the present invention provides a vehicle-mounted camera lens abnormality identification device.

[0017] The vehicle-mounted camera lens abnormal state recognition device includes: an acquisition module for acquiring image frames captured by the vehicle-mounted camera; a preprocessing module for preprocessing the image frames to obtain a standard image; a recognition module for inputting the standard image into an abnormal state recognition model, wherein the abnormal state recognition model is used to extract features from the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, with each dimension component corresponding to an abnormal state, and perform independent activation mapping on each dimension component to output the confidence level corresponding to each of the multiple abnormal states; and a determination module for determining the abnormal state recognition result of the vehicle-mounted camera lens based on the multiple confidence levels.

[0018] On the other hand, to achieve the above objectives, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0019] On the other hand, to achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0020] The present invention provides a method, apparatus, and vehicle for identifying abnormal states of vehicle-mounted camera lenses. First, image frames captured by the vehicle-mounted camera are acquired and preprocessed to obtain standard images that meet the recognition requirements. Then, the standard images are input into an abnormal state recognition model, which extracts features from the standard images and generates a multi-dimensional initial prediction vector based on the extracted features. Each dimension corresponds to an abnormal state. Each dimension is then independently activated and mapped to output the confidence scores corresponding to multiple abnormal states. Finally, the abnormal state recognition result of the vehicle-mounted camera lens is determined based on the multiple output confidence scores. This invention directly uses lens images captured by vehicle-mounted cameras as the identification basis. Lens anomaly identification can be completed by extracting features and judging abnormal states from the lens images themselves, without the need for auxiliary judgment based on LiDAR data or other heterogeneous sensing results. Therefore, it can reduce dependence on external sensing architecture, avoid the susceptibility of LiDAR data to environmental noise, and avoid the misjudgment problems that may arise from the comparison of sensing results. This makes it more suitable for lens anomaly identification needs under complex vehicle operating conditions such as rain, snow, fog, strong light, and night. At the same time, based on the anomaly state identification model, a multi-dimensional initial prediction vector is generated and each dimension component is independently activated and mapped. It can output the confidence scores corresponding to multiple anomaly states, so that multi-dimensional anomaly states that may exist simultaneously in the same image frame, such as full occlusion, partial occlusion, blur, splashing, and backlighting, can be identified in parallel. This improves the comprehensiveness and accuracy of lens anomaly identification by vehicle-mounted cameras, and provides more effective technical support for vehicle environmental perception reliability monitoring and subsequent safety decisions. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of the vehicle-mounted camera lens abnormality identification method provided in Embodiment 1 of the present invention; Figure 2This is a flowchart of the vehicle-mounted camera lens abnormality identification method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the abnormal state recognition model in the vehicle-mounted camera lens abnormal state recognition method provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the post-processing steps in the vehicle-mounted camera lens abnormality identification method provided in Embodiment 2 of the present invention; Figure 5 This is a block diagram of the vehicle-mounted camera lens abnormality recognition device provided in Embodiment 3 of the present invention; Figure 6 This is a hardware structure diagram of a computer device provided in Embodiment 5 of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0023] Example 1 Embodiment 1 of the present invention provides a method for identifying abnormal states of a vehicle-mounted camera lens. This method enables parallel identification of multiple abnormal states of the lens based on image frames captured by the vehicle-mounted camera, and outputs the corresponding abnormal state identification results. Specifically, Figure 1 This is a flowchart of the vehicle-mounted camera lens abnormality recognition method provided in Embodiment 1 of the present invention, as follows: Figure 1 As shown, the vehicle-mounted camera lens abnormality identification method provided in this embodiment includes the following steps S101 to S104.

[0024] Step S101: Acquire image frames captured by the vehicle-mounted camera.

[0025] Among them, the vehicle-mounted camera is a camera device installed on the vehicle for environmental perception. For example, the vehicle-mounted camera is a surround-view vehicle-mounted camera installed around the vehicle body. Accordingly, the acquired image frames can be a single frame image captured by a single camera at a certain moment, or image frames captured separately by multiple surround-view cameras. For image frames captured by multiple cameras, they can be input separately into the subsequent recognition process to judge the abnormal state of the lens, thereby realizing the monitoring of the lens state from multiple perspectives of the vehicle.

[0026] In the field of vehicle-related technologies, malfunctions in vehicle-mounted camera lenses directly affect the results of vehicle environmental perception. For example, when a lens is completely or partially obstructed, the effective environmental information in the acquired image is reduced; when a lens is blurry, out of focus, iced, or splashed, image details, edge contours, or local areas will be distorted; when a lens is in a backlit or sun-ray scene, the image brightness distribution will be abnormal; and the appearance of a green image often indicates a malfunction in the sensor, image transmission link, or image processing stage. Based on this, this embodiment uses the actual image frames acquired by the vehicle-mounted camera as the recognition input, rather than relying on other external perception results as the main recognition basis, so as to directly determine the lens status from the image performance.

[0027] Step S102: Preprocess the image frames to obtain standard images.

[0028] A standard image refers to image data that meets the input requirements of an anomaly recognition model. In this embodiment, the purpose of preprocessing the image frames is twofold: firstly, to unify the image representation from different vehicle-mounted cameras or under different acquisition conditions; and secondly, to reduce the impact of differences in original image scale, color distribution, etc., on subsequent anomaly recognition, thereby improving recognition stability.

[0029] Specifically, preprocessing includes image resizing and color space standardization. Image resizing refers to scaling the input image to a preset size to fit the input specifications of the subsequent recognition model, while also considering the limitations of the computing resources on the vehicle. In one implementation, bilinear interpolation is used to resize the image to maintain image proportions and local details as much as possible, reducing feature distortion caused by stretching. Color space standardization refers to converting the image to a unified color space, such as the RGB color space, and standardizing each color channel to reduce color channel differences between different vehicle camera imaging devices.

[0030] Step S103: Input the standard image into the abnormal state recognition model.

[0031] Among them, the abnormal state recognition model is used to extract features of the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, each dimension component corresponds to an abnormal state, and perform independent activation mapping on each dimension component to output the confidence level corresponding to each of the multiple abnormal states.

[0032] In this embodiment, the abnormal state recognition model is an image recognition model. It takes an image to be recognized as input and outputs the confidence scores corresponding to multiple abnormal states. Specifically, after training, the model receives a preprocessed standard image, first extracting features from the image's texture, edges, brightness distribution, color distribution, and local region structure. Then, it generates a multi-dimensional initial prediction vector based on the extracted features. Each component in the multi-dimensional initial prediction vector corresponds to an abnormal state. After independent activation mapping, each component finally outputs the confidence score corresponding to the abnormal state.

[0033] Optionally, when training the abnormal state recognition model, a sample set is first constructed. Specifically, image frames are collected by a test vehicle equipped with an onboard camera under different climates, time periods, and road conditions, such as sunny days, rainy days, snowy days, foggy days, daytime, dawn, dusk, and nighttime, as well as different scenarios like urban roads, highways, and rural roads. This allows the recognition model to cover a richer range of vehicle operating environments and improves the adaptability of lens abnormal state recognition under complex onboard conditions. Based on the collected image frames, a sample set is established, and abnormal states appearing in the images are labeled. In scenarios where multiple abnormal states coexist, multiple abnormal state labels can be assigned to the same image frame.

[0034] Optionally, during the model training phase, data augmentation can be performed on the image frames in the sample set to improve the model's adaptability to changes in vehicle posture, ambient light, and imaging perturbations during vehicle movement. For example, random flipping, rotation, and scaling can be performed on the images to simulate minute changes in camera angle during vehicle movement; or brightness, contrast, and saturation perturbations can be performed to simulate differences in image imaging under different lighting conditions. It should be noted that the above data augmentation is primarily used for sample construction during the model training phase; it is not necessary to perform the above augmentation processing on the image frames to be recognized during the online recognition phase.

[0035] In this embodiment, the abnormal states refer to a number of predefined abnormal states of the vehicle-mounted camera lens. For example, abnormal states include complete occlusion, partial occlusion, blur, icing, splatter, out of focus, backlight, sunlight, and green image. Complete occlusion manifests as an area of ​​the image frame being completely covered by foreign objects, meaning the entire lens area is covered by foreign objects. Partial occlusion manifests as a localized area of ​​the image frame exhibiting obvious dark areas, color blocks, or texture breaks, meaning a portion of the lens is covered by foreign objects. Blur indicates a decrease in the overall sharpness of the image frame, meaning there are stains on the lens surface, lens fogging, or a decline in optical performance. Icing manifests as an image frame accompanied by white opaque or semi-transparent textures, meaning an ice or frost layer forms on the lens surface. Splatter manifests as an image frame containing droplets, watermarks, or mud stains. Uneven areas of light and shadow are caused by rain, mud, puddles, or other liquids or semi-fluid substances splashing onto the lens surface; backlighting manifests as strong overexposure and significant contrast between dark and bright areas, caused by strong light sources directly in front or to the side leading to severe overexposure; out-of-focus appears as a general or localized "blurred" image, caused by a malfunction in the camera's focusing system or vibration causing the image focus to deviate; sunlight rays appear as radial bright lines in the image frame, caused by sunlight refracting into the image when the lens is in a strong light environment; and abnormal green tones appear in localized or overall areas, caused by camera hardware or software malfunctions. All of these abnormal conditions are closely related to lens reliability in vehicle operation scenarios.

[0036] In this embodiment, the abnormal state recognition model performs independent activation mapping on each component to output the confidence score corresponding to each abnormal state. Therefore, two or more abnormal states can coexist in the same image frame, and each outputs its corresponding confidence score. For example, in a rainy driving scenario, splashing, blurring, and partial occlusion may occur simultaneously in the same image frame. In this case, the model does not force multiple abnormal states to be mutually exclusive, but allows for parallel judgment of multiple abnormal states. That is, the multiple confidence scores output in this embodiment do not represent a competitive relationship between single categories, but rather represent the probability of multiple abnormal states existing separately.

[0037] Optionally, the anomaly recognition model employs a lightweight convolutional neural network combined with a multi-label classification head to balance recognition accuracy and inference efficiency when deployed in vehicles, making it more suitable for real-time recognition requirements in vehicle environment perception systems. Furthermore, during the model training phase, considering the high proportion of normal samples and the relative scarcity of some anomalous samples in the vehicle camera sample set, sample imbalance processing is applied during training to enhance the model's ability to learn anomalous state features. Further, optionally, during the training data construction phase, data augmentation methods such as random flipping, rotation, scaling, and brightness, contrast, and saturation perturbations are used to improve sample distribution imbalance; during the model training phase, the Focal Loss loss function is employed to reduce the dominant role of majority class samples in the training process and increase the model's attention to scarce anomalous samples.

[0038] Step S104: Determine the abnormal state recognition result of the vehicle-mounted camera lens based on multiple confidence levels.

[0039] The anomaly identification result includes identification conclusions for multiple anomaly states, used to characterize whether there are any anomalies in the vehicle-mounted camera lens corresponding to the current image frame, and what those anomalies are. In this embodiment, when determining the anomaly identification result based on multiple confidence levels, the judgment rules corresponding to each anomaly state can be combined to perform post-processing on the multiple confidence levels, thereby improving the stability and usability of the result.

[0040] Optionally, the confidence levels are sorted from largest to smallest, and the abnormal states corresponding to the top few confidence levels that are greater than the preset confidence threshold are selected as the abnormal state identification results, that is, to confirm that the vehicle-mounted camera lens has these abnormal states.

[0041] Optionally, different confidence thresholds can be set for different abnormal states. Based on the comparison between the confidence level of each abnormal state and the corresponding threshold, it can be determined that abnormal states with a confidence level greater than the corresponding confidence threshold exist in the current image frame, thus confirming the existence of these abnormal states in the vehicle-mounted camera lens. Using differentiated thresholds can adapt to the differences in sample distribution, visual complexity, and recognition difficulty of different abnormal states, which is beneficial to improving the classification accuracy of each abnormal state.

[0042] Optionally, during continuous vehicle operation, the onboard camera outputs a continuous frame sequence. Since lens anomalies typically exhibit temporal continuity, time-series smoothing can be performed based on the recognition results of these continuous frames. For example, when the confidence level of an anomaly exceeds a preset threshold, it is determined that the anomaly occurs in the current image frame. Furthermore, the occurrence of this anomaly in consecutive frames is statistically analyzed within a sliding time window. If the anomaly persists within the window and meets a preset condition, it is determined that the anomaly is valid within the current time period. Conversely, if the anomaly only occasionally appears in a small number of frames, it can be considered as being caused by noise, transient disturbances, or single-frame misidentification and filtered out from the recognition results. This processing method reduces misidentification caused by factors such as vehicle body vibration, instantaneous changes in illumination, or short-term reflections from local water droplets.

[0043] Optionally, after the identification result determines that some abnormal states exist in the current image frame while others do not, the identification result is further verified based on the correlation between different abnormal states. For example, certain combinations of abnormal states should not occur simultaneously; when an abnormal state combination that violates such correlation rules appears in the identification result, a secondary verification can be performed on the abnormal states with lower confidence, and contradictory results can be eliminated if necessary to further improve the reasonableness of the final identification result. For vehicle-mounted scenarios, this type of correlation verification helps avoid conflicts in abnormal state combinations caused by complex weather, strong light reflection, or local image anomalies.

[0044] Optionally, based on the final abnormal state identification results, further abnormal state information can be generated for use by downstream vehicle functions. For example, the identification results can be provided to the vehicle's environmental perception system, driver assistance system, or vehicle alert system to perform function degradation, status alerts, or lens check reminders. For example, complete occlusion, icing, and green maps can be considered as severely affected abnormal states; partial occlusion, out-of-focus, blur, and splashes can be considered as abnormal states where some functions are affected; and backlighting and sun rays can be considered as abnormal states that affect imaging performance but still provide some useful information. Corresponding processing signals or alert information can then be output for different levels of abnormal states.

[0045] In the vehicle-mounted camera lens abnormal state recognition method provided in this embodiment, image frames captured by the vehicle-mounted camera are first acquired, and the image frames are preprocessed to obtain a standard image that meets the recognition requirements. Then, the standard image is input into the abnormal state recognition model, which extracts the features of the standard image and generates a multi-dimensional initial prediction vector based on the extracted features. Each dimension component corresponds to an abnormal state. Then, each dimension component is independently activated and mapped to output the confidence scores corresponding to multiple abnormal states. Finally, based on the multiple output confidence scores, the abnormal state recognition result of the vehicle-mounted camera lens is determined. The vehicle-mounted camera lens anomaly identification method provided in this embodiment directly uses the lens images captured by the vehicle-mounted camera as the identification basis. Lens anomaly identification can be completed by extracting features and judging anomalies in the lens images themselves, without the need to combine LiDAR data or other heterogeneous sensing results for auxiliary judgment. Therefore, it can reduce the dependence on external sensing architecture, avoid the problem that LiDAR data is easily affected by environmental noise and the misjudgment problem that may be caused by the comparison of sensing results. Thus, it is more suitable for lens anomaly identification needs under complex vehicle working conditions such as rain, snow, fog, strong light, and night. At the same time, based on the anomaly identification model, a multi-dimensional initial prediction vector is generated and each dimension component is independently activated and mapped. It can output the confidence scores corresponding to multiple anomalies, so that multi-dimensional anomalies such as full occlusion, partial occlusion, blur, splash, and backlight that may exist simultaneously in the same image frame can be identified in parallel. This improves the comprehensiveness and accuracy of vehicle-mounted camera lens anomaly identification and provides more effective technical support for vehicle environmental perception reliability monitoring and subsequent safety decisions.

[0046] Optionally, in one embodiment, the abnormal state recognition model includes a feature extraction network, a feature mapping layer, and an activation layer; the step of inputting a standard image into the abnormal state recognition model includes: inputting the standard image into the feature extraction network, which extracts features from the standard image and outputs a global feature vector; inputting the global feature vector into the feature mapping layer, which maps the global feature vector into a multi-dimensional initial prediction vector; and inputting the multi-dimensional initial prediction vector into the activation layer, which performs independent activation mapping on each dimension component to output the confidence level corresponding to each abnormal state.

[0047] Specifically, the anomaly recognition model comprises three functional layers: a feature extraction network, a feature mapping layer, and an activation layer. The feature extraction network extracts high-level visual representations related to the lens state from the standard image, such as edge sharpness, local occlusion texture, abnormal brightness distribution, abnormal color regions, and local morphological changes. These representations can be used to distinguish different types of anomalies in vehicle scenes, such as lens smudges, occlusion, backlighting, and out-of-focus conditions. The global feature vector output by the feature extraction network is a centralized representation of the lens state information for the entire frame, and is then fed into the feature mapping layer.

[0048] The feature mapping layer transforms the global feature vector into a multi-dimensional initial prediction vector. Each component in this multi-dimensional initial prediction vector corresponds to an anomalous state to be identified. Therefore, this layer converts general image features into predicted values ​​for each anomalous state. Subsequently, the activation layer performs independent activation mapping on each component, enabling each dimension to output the confidence level of the corresponding anomalous state. Because a sequential independent mapping approach is used, different anomalous states do not need to satisfy mutual exclusion. Therefore, it can adapt to situations where multiple anomalous states occur simultaneously in the same image frame during vehicle operation, such as splashing, blurring, and partial occlusion in rainy conditions, which can coexist in the same frame.

[0049] The vehicle-mounted camera lens abnormality recognition method provided in this embodiment defines the abnormality recognition model as including a feature extraction network, a feature mapping layer, and an activation layer, and clarifies the data flow path within the model. This ensures that the prediction results for different abnormalities are all based on a unified image feature representation and a dimension-by-dimensional independent mapping mechanism, thereby improving the interpretability, feasibility, and adaptability to scenarios with multiple coexisting abnormalities in the lens abnormality recognition process.

[0050] Optionally, in one embodiment, the multiple abnormal states include full occlusion, partial occlusion, blur, icing, splashing, out of focus, backlight, sun rays, and green map.

[0051] In this embodiment, the types of abnormal states are further defined to ensure that the output dimensions of the abnormal state recognition model correspond to the needs of vehicle lens anomaly monitoring. Specifically, "complete occlusion" characterizes a state where the entire imaging area of ​​the lens is completely covered by foreign objects; "partial occlusion" characterizes a state where there is partial occlusion in the lens imaging area; "blurring" characterizes a state where overall sharpness is reduced due to dirt, fog, or decreased optical performance on the lens surface; "icing" characterizes a state where ice or frost forms on the lens surface in low-temperature environments; "splashing" characterizes a state where rainwater, mud, or accumulated water splashes onto the lens surface; "out of focus" characterizes a state of defocusing caused by abnormal focusing system conditions or vibration; "backlighting" characterizes a state of severe overexposure caused by strong light sources directly in front or to the side; "sunlight rays" characterizes a state where strong light is refracted through the lens optical structure to form radial bright lines; and "green map" characterizes an abnormal green tone state caused by sensor malfunction, abnormal data transmission, or abnormal image processing chip conditions. These anomaly types cover lens anomalies that directly affect the perception results during vehicle operation.

[0052] Furthermore, by limiting the abnormal states to the aforementioned nine categories, a clear one-to-one correspondence is established between the output space of the abnormal state recognition model and the abnormal lens scenarios. During the training phase, corresponding samples are collected or labeled for each type of abnormal state; during the inference phase, the current abnormal state of the lens is determined based on the confidence output corresponding to the nine types of abnormal states. Especially in surround-view vehicle camera scenarios, different cameras are located in different positions and have different external environmental exposure conditions, making it easier for various types of lens anomalies to occur. Using the aforementioned nine-category abnormal state definition can cover multiple sources of anomalies, from physical occlusion and optical distortion to image link failures.

[0053] The vehicle-mounted camera lens abnormality identification method provided in this embodiment further limits the abnormality to nine types of lens abnormalities with clear vehicle scene meaning, which enables the model output results to directly correspond to lens problems in vehicle operation, thereby improving the correspondence between the identification results and the vehicle environment perception fault types.

[0054] Optionally, in one embodiment, the feature extraction network adopts a residual network ResNet18. The feature extraction network outputs a 512-dimensional global feature vector, and the feature mapping layer maps the 512-dimensional global feature vector into a 9-dimensional initial prediction vector. In this embodiment, ResNet18 is selected as the feature extraction network because it has a residual connection structure, which can control the parameter scale and computational complexity while maintaining good feature extraction capabilities. Furthermore, ResNet18 has good compatibility in vehicle-side deployment scenarios, making it easy to adapt to in-vehicle inference acceleration frameworks and mainstream computing platforms. The 512-dimensional global feature vector output by this network is used to centrally represent the lens state information in the entire frame image. The feature mapping layer then compresses and maps this 512-dimensional feature vector into a 9-dimensional initial prediction vector, corresponding to the initial prediction values ​​for nine abnormal states. In addition, this model structure is compatible with INT8 (Integer 8-bit) quantization and TensorRT (NVIDIA Tensor Runtime) acceleration, which can meet the real-time inference requirements of parallel processing of surround-view cameras.

[0055] The activation layer uses the Sigmoid function (also known as the S-shaped function) to achieve independent probability mapping for the predicted value of each anomalous state, rather than competitive normalization among multiple categories as in mutually exclusive classification. Therefore, when the same frame image contains compound anomalous states such as splattering, blurring, and partial occlusion, the model can still output high confidence scores for each.

[0056] During the training of the abnormal state recognition model, Focal Loss is used as the training loss function to alleviate the class imbalance problem caused by the high proportion of normal samples and the relative scarcity of abnormal samples in vehicle lens abnormal samples, so that the training process pays more attention to the abnormal samples that are more difficult to identify or have a small number of abnormal samples.

[0057] The vehicle-mounted camera lens abnormality recognition method provided in this embodiment combines ResNet18, 512-dimensional global features, 9-dimensional initial prediction vector, Sigmoid activation, and Focal Loss training loss function. This enables the model to maintain high feature extraction capability and multi-anomaly parallel recognition capability in vehicle-mounted resource-constrained environments, while also mitigating the impact of sample imbalance on training results. Thus, it balances recognition accuracy, inference speed, and deployment feasibility in real-time vehicle operation scenarios.

[0058] Optionally, in one embodiment, the vehicle-mounted camera is a surround-view vehicle-mounted camera. After determining the abnormal state identification result of the vehicle-mounted camera lens based on multiple confidence levels, the method further includes: when the abnormal state of the vehicle-mounted camera lens includes complete occlusion, icing, or a green map, outputting a prompt message to exit the perception function; when the abnormal state of the vehicle-mounted camera lens includes partial occlusion, out-of-focus, blur, or splatter, outputting a prompt message to check the lens state; when the abnormal state of the vehicle-mounted camera lens includes backlighting or sunlight, outputting a prompt message that the current light is affecting camera performance.

[0059] In this embodiment, the obtained lens abnormality identification results are further linked with downstream vehicle functions. For surround-view vehicle camera scenarios, the linkage function can improve the safety and reliability of downstream functions such as vehicle parking assistance, low-speed environment perception, surround-view stitching display, or assisted driving visual perception. Specifically, when the identification result includes complete occlusion, icing, or a green map, it indicates that the lens can no longer provide effective image information, or the image link itself has experienced a serious abnormality. In this case, a prompt message to exit the perception function can be output to prevent the system from continuing to make perception decisions based on distorted images. When the identification result includes partial occlusion, out-of-focus, blurry, or splatter, it indicates that the lens may still retain some image information, but the perception quality has been significantly affected. In this case, a prompt message to check the lens status can be output to remind the driver to pay attention to the lens surface condition or the external environment of the vehicle. When the identification result includes backlighting or sunlight, it indicates that the lens is mainly affected by strong light conditions, and the image performance degrades at a specific moment but is not completely ineffective. Therefore, a prompt message indicating that the current light is affecting camera performance can be output.

[0060] The vehicle-mounted camera lens abnormality identification method provided in this embodiment combines the abnormality results with the downstream perception and prompting requirements of the surround-view vehicle-mounted camera, so that lens abnormalities of different severity can trigger different types of vehicle prompts or functional response strategies, thereby enabling the lens abnormality identification results to directly serve the perception reliability assurance and driving prompts during vehicle operation.

[0061] Optionally, in one embodiment, the step of determining the abnormal state recognition result of the vehicle-mounted camera lens based on multiple confidence levels includes: presetting different confidence thresholds for different abnormal states; traversing multiple confidence levels and determining whether the confidence level is greater than or equal to the corresponding confidence threshold; when the confidence level is greater than or equal to the corresponding confidence threshold, determining that the first image recognition result includes the presence of an abnormal state corresponding to the confidence level of the image frame; when the confidence level is less than the corresponding confidence threshold, determining that the first image recognition result includes the absence of an abnormal state corresponding to the confidence level of the image frame; and determining the abnormal state recognition result based on the first image recognition result.

[0062] In this embodiment, the process of determining the abnormal state identification result based on multiple confidence levels in the aforementioned embodiments is further defined as a differentiated threshold judgment process. Setting different confidence thresholds for different abnormal states can adapt to the differences in the visual representation of different abnormal states in vehicle scenarios. For example, complete occlusion and green maps often have strong overall characteristics, while partial occlusion, splashing, backlighting, and other abnormalities may appear more complex in local areas or under specific lighting conditions. Therefore, this method can avoid the judgment bias caused by using a uniform threshold for all abnormal states.

[0063] Specifically, after the model outputs the confidence scores corresponding to multiple abnormal states, the confidence scores for each abnormal state can be iterated one by one and compared with the corresponding pre-set thresholds. If the confidence score of an abnormal state reaches or exceeds the corresponding threshold, it is determined that the current image frame contains the abnormal state; otherwise, it is determined that the current image frame does not contain the abnormal state. The resulting first image recognition result serves as the initial judgment result based on the current single-frame image. This initial judgment result can be directly used as the final recognition result, or it can be used for further correction by combining it with other processing logic.

[0064] The vehicle-mounted camera lens abnormal state recognition method provided in this embodiment sets confidence thresholds for different abnormal states and forms a first image recognition result. This makes the judgment criteria for each abnormal state more consistent with its own visual feature distribution and recognition difficulty, thereby improving the accuracy of the initial judgment of abnormal states at the single frame level and improving classification accuracy.

[0065] Optionally, in one embodiment, the step of determining the abnormal state identification result based on the first image recognition result includes: traversing the abnormal states of the image frames determined by the first image recognition result, and counting the number of image frames with abnormal states within the sliding time window where the image frames are located; determining whether the number of image frames is greater than or equal to a preset frame number threshold; when the number of image frames is greater than or equal to the preset frame number threshold, determining that the second image recognition result includes the presence of abnormal states in the image frames; when the number of image frames is less than the preset frame number threshold, determining that the second image recognition result includes the absence of abnormal states in the image frames; and determining the abnormal state identification result based on the second image recognition result.

[0066] In this embodiment, the first image recognition result obtained in the aforementioned embodiment is further smoothed in the temporal dimension. During vehicle operation, the onboard camera captures a continuous sequence of image frames. Lens anomalies typically do not occur and disappear instantly at a single point in time, but rather often exhibit a certain degree of temporal continuity. Simultaneously, factors such as vehicle vibration, localized water droplet reflections, short-term changes in intense light, and road surface splashes momentarily passing the lens edge can cause occasional misjudgments at the single-frame level. Therefore, this embodiment further confirms the temporal continuity of the anomalies identified in the first image recognition result of the current image frame by setting a sliding time window.

[0067] Specifically, a sliding time window is constructed around the current image frame, and the number of image frames within this window that are determined to contain the same abnormal state is counted. When the number of image frames reaches a preset frame threshold, it indicates that the abnormal state has sufficient persistence in consecutive frames and can be retained in the second image recognition result; conversely, when the number of image frames does not reach the preset frame threshold, the abnormal state can be considered unstable in the time dimension and thus excluded from the second image recognition result. Optionally, a sliding window voting process is performed on 10 consecutive frames. If an abnormal state appears 6 times or more in 10 consecutive frames, it is finally determined to exist. If it only appears in 1-2 frames, it can be considered a model misjudgment and filtered out.

[0068] The vehicle-mounted camera lens abnormality identification method provided in this embodiment confirms the persistence of abnormality by sliding time windows. This effectively reduces the impact of single-frame noise, transient interference, or local sporadic image abnormalities on the identification results, making the final retained abnormality more consistent with the real lens state during continuous vehicle operation, thereby improving the temporal stability and robustness of the abnormality identification results.

[0069] Optionally, in one embodiment, the step of determining the abnormal state identification result based on the second image recognition result includes: determining whether the second image recognition result satisfies the abnormal state association rule, wherein the abnormal state association rule includes conflict rule and / or co-occurrence rule, the conflict rule is used to define abnormal states that conflict, and the co-occurrence rule is used to define abnormal states that occur simultaneously; when the second image recognition result does not satisfy the abnormal state association rule, verifying the abnormal states with low confidence in the unsatisfied abnormal state association rule according to preset external parameters; and determining the abnormal state identification result based on the verification result.

[0070] In this embodiment, based on the second image recognition result, association rules between abnormal states are further introduced for logical consistency verification. The vehicle operating environment is complex; the camera lens may be affected by multiple factors such as weather, mud, vibration, and strong light. Therefore, even after smoothing through a time window, some combinations of abnormal states may still appear semantically unreasonable. To address this, abnormal state association rules are pre-constructed to review the reasonableness of the second image recognition result.

[0071] The conflict rule characterizes combinations of anomalous states that should not typically occur simultaneously in real-world vehicle scenarios. For example, icing is usually associated with low temperatures and would not occur simultaneously with splashing; defocusing and greening patterns typically do not occur together. When such conflicting combinations appear in the second image recognition result, it is considered that the recognition result of at least one of the anomalous states requires further verification. The co-occurrence rule characterizes combinations of anomalous states that may occur simultaneously under specific vehicle operating conditions. For example, in rainy or slippery road scenarios, splashing, blurring, and partial occlusion may co-occur; in strong light scenarios, backlighting and sunlight rays may also have a high correlation. Through the conflict rule and / or co-occurrence rule, the second image recognition result is subjected to more realistic constraints on the reasonableness of combinations that closely resemble vehicle scenarios.

[0072] Furthermore, when the second image recognition result does not meet the abnormal state association rules, the system combines preset external parameters to verify the abnormal states with lower confidence among the combinations of abnormal states that do not meet the association rules. These preset external parameters may include parameters related to the vehicle's operating environment, such as weather parameters, ambient temperature parameters, lighting conditions parameters, and parameters indicating whether the vehicle is in a rainy or snowy road environment. For example, if the recognition result shows both icing and splashing, and the vehicle's current external environmental parameters indicate that the temperature is in the low-temperature icing range and there is no rainfall, then the icing state can be retained first; conversely, if the external parameters indicate that the vehicle is in rainy road conditions and the ambient temperature is high, then the splashing state can be retained first. This method can further eliminate abnormal state results that contradict each other or are inconsistent with external operating conditions.

[0073] The vehicle-mounted camera lens abnormality recognition method provided in this embodiment verifies the rationality of abnormality combination by using abnormality association rules and external environmental parameters. This reduces the situation where conflicting or inconsistent abnormalities with the actual vehicle operating conditions are retained simultaneously, making the final abnormality recognition result not only have temporal continuity, but also better scene consistency and logical rationality, thereby further improving the credibility of lens abnormality recognition results in vehicle environmental perception and safety decision-making.

[0074] In this embodiment, by first determining a confidence threshold, abnormal states with initial credibility can be screened based on the recognition confidence of a single frame image, preventing obviously low-confidence abnormal states from entering subsequent processing. Then, by using a time window to determine the temporal continuity of the initial recognition results, misjudgments in a single frame caused by factors such as instantaneous changes in illumination, vehicle body vibration, localized reflections, and short-term water droplet interference can be filtered out, making the retained abnormal states more consistent with the actual characteristic of continuous lens anomalies during vehicle operation. Based on this, association rule verification is performed on the abnormal states after temporal continuity screening, further eliminating combinations of conflicting or inconsistent abnormal states with actual operating conditions, improving the logical consistency and scenario rationality of the final recognition result. Therefore, performing the combined judgment in the order of initial confidence screening, temporal continuity confirmation, and association rule verification not only reduces misjudgments and result jumps layer by layer, but also avoids miscorrection caused by premature association rule judgment before single-frame noise is eliminated. This results in an abnormal state recognition result that simultaneously possesses high recognition accuracy, temporal stability, and combined rationality, making it more suitable for anomaly recognition of vehicle-mounted camera lenses under complex vehicle operating conditions.

[0075] Example 2 Embodiment 2 of this invention provides a method for identifying abnormal states of vehicle-mounted camera lenses. This method uses a lightweight convolutional network structure to perform real-time analysis of image data acquired by vehicle-mounted surround-view cameras, achieving accurate identification of various abnormal states. This provides technical support for reliability monitoring, fault warning, and decision-making safety of vehicle surround-view perception systems, and can be widely applied in the research and application scenarios of automotive electronics, intelligent transportation, and autonomous driving technologies, involving fields such as autonomous driving, deep learning, and image processing. This method uses images acquired by surround-view cameras mounted on the vehicle body as input. Based on a lightweight convolutional neural network structure with a multi-label classification head, it achieves rapid identification of different abnormal state signals and outputs the corresponding confidence scores. This method supports the simultaneous identification of multiple typical abnormal states, including complete occlusion, partial occlusion, blurring, icing, splashing, defocusing, backlighting, solar radiation, and green images, effectively improving the reliability of vehicle-mounted perception systems.

[0076] Figure 2This is a flowchart of the vehicle-mounted camera lens abnormality recognition method provided in Embodiment 2 of the present invention, as follows: Figure 2 As shown, in the embodiment provided by this invention, images captured by multiple cameras are first acquired. After obtaining relevant data, fixed-size data is obtained through preprocessing and input into the backbone network. The features extracted from each image by the backbone network are sent to a multi-label classification head to obtain the confidence level of each abnormal signal. Finally, the output results are post-processed to obtain the final abnormal state information, which is provided to downstream systems for security decision-making.

[0077] This method supports the identification of nine types of camera lens anomalous signals, including complete occlusion, partial occlusion, blur, icing, splashing, out of focus, backlight, sun rays, and green map. The definition and main coverage scenarios of each anomalous signal are as follows: 1. Complete Obstruction: The entire imaging area of ​​the camera lens is completely covered by foreign objects (such as plastic bags, paper, leaves, etc.), and the image has no effective environmental information. It appears as a single color (black, gray or the color of the foreign object itself) or is completely blurry and unrecognizable. 2. Partial occlusion: 10% or more of the imaging area of ​​the camera lens is obscured by a foreign object. The unobstructed area can present effective environmental information, but the obstructed area forms obvious dark areas, color blocks or texture breaks, such as the lens edge being obscured by tree branches or paper. 3. Blurry: Stains on the lens surface (such as dust and oil), lens fogging, or decreased optical performance lead to a reduction in the overall clarity of the image, blurred edge contours, loss of detail information, and no significant change in the overall brightness uniformity of the image even without obvious obstructions. 4. Icing: In low-temperature environments, ice or frost forms on the lens surface, resulting in a white, opaque or semi-transparent texture in the image. This is accompanied by image distortion caused by light refraction, and the area covered by ice lacks environmental details. 5. Splashing: During driving, rainwater, mud, puddles and other liquids or semi-fluid substances splash onto the lens surface, forming irregular droplets, watermarks or mud spots. There are scattered areas of uneven brightness in the image. In dynamic scenes, the splash marks may appear as motion blur. 6. Out of focus: A malfunction in the camera's focusing system or vibration causes the image focus to deviate, resulting in a "out-of-focus" phenomenon in the whole or part of the image, with no clear outlines for both distant and close-up scenes, and low contrast. 7. Backlighting: The sun or a strong light source is located directly in front of or to the side of the camera's shooting direction. The light shines directly into the lens, causing severely overexposed areas in the image. The overexposed areas appear as bright white highlights, while the unexposed areas appear dark due to insufficient light, resulting in a clear contrast between light and dark areas. 8. Sun rays: In strong light conditions, sunlight rays are refracted through the lens's optical structure and form radial bright lines in the image. The bright lines spread outward from the light source, covering part of the effective imaging area, but do not affect the detail in areas not covered by the rays. 9. Green image: Camera sensor malfunction, abnormal data transmission, or image processing chip error causes the image to appear green in parts or the whole.

[0078] To improve model training efficiency and recognition accuracy, this invention designs a standardized data preprocessing workflow to optimize the acquired surround-view vehicle camera image data in multiple dimensions. The specific steps are as follows: 1. Data Acquisition and Labeling: Raw image data was collected by test vehicles equipped with surround-view cameras under different weather conditions (sunny, rainy, snowy, foggy), time of day (daytime, dawn, dusk, night), and road conditions (urban roads, highways, rural roads), ensuring data coverage of 9 abnormal and normal states. A combination of pre-labeling and manual labeling was used to label each image with anomaly state tags (in multi-label scenarios, all existing anomaly types were labeled), thus constructing a dataset.

[0079] 2. Unified Image Size: Considering the limitations of computing resources in vehicle terminals and the requirements of model input, all input images need to be reduced to a uniform size. Bilinear interpolation is used to maintain image proportions and detail information, and to avoid feature distortion caused by stretching and deformation.

[0080] 3. Color space standardization: Convert the image to the RGB color space to eliminate color channel differences between different camera acquisition devices, and then standardize the three RGB channels separately. 4. Data Augmentation: To address the imbalanced sample distribution and improve the model's generalization ability, targeted data augmentation strategies are designed to augment the samples during the training phase. Specifically, geometric transformations and / or illumination perturbations can be employed. Geometric transformations mainly include random flipping, rotation, and scaling to simulate minute changes in the camera angle during vehicle movement; illumination perturbations mainly involve randomly adjusting image brightness, contrast, and saturation to simulate imaging differences under different lighting conditions.

[0081] This method employs a lightweight feature extraction backbone network superimposed with an efficient multi-label classification head. It combines a Sigmoid classification head with a Focal Loss loss function designed for scenarios with multiple anomalies coexisting in vehicle cameras. At the same time, it selects the deployment-friendly ResNet18 residual network as the backbone network to achieve a balance between recognition performance and engineering feasibility. The overall model structure is compact and computationally efficient, making it suitable for real-time inference in resource-constrained environments on vehicle devices. Figure 3This is a schematic diagram of the structure of the abnormal state recognition model in the vehicle-mounted camera lens abnormal state recognition method provided in Embodiment 2 of the present invention, as shown below. Figure 3 As shown, the various parts of this abnormal state recognition model structure are described below: 1. ResNet18 Backbone Network: The ResNet18 residual network is selected as the basis for feature extraction. Its core advantages lie in its small parameter size and low computational complexity, while ensuring feature extraction capabilities through residual connections, making it suitable for the limited computing resources of in-vehicle devices. Its input is a preprocessed image, which, through a series of convolution and pooling operations, ultimately outputs a 512-dimensional global feature vector, providing efficient representation support for subsequent classification tasks. This backbone network can be directly deployed without complex modifications and is compatible with various in-vehicle inference acceleration frameworks, exhibiting excellent deployment compatibility.

[0082] 2. Sigmoid multi-label classification head: In response to the typical features of multiple anomalies coexisting in abnormal scenes of vehicle lenses (such as splashing, blurring and partial occlusion that may occur at the same time in rainy weather), an independent and parallel Sigmoid multi-label classification head is designed to replace the traditional single-label classification structure. The core design is as follows: (1) Feature mapping layer: The feature vector output by ResNet18 is mapped to a 9-dimensional vector through a lightweight fully connected layer, which corresponds to the initial prediction value of 9 abnormal states respectively; (2) Sigmoid activation mechanism: Abandoning the "probability mutual exclusion" characteristic of Softmax activation, the Sigmoid function is used to independently map the prediction value of each abnormal type to the [0,1] interval. The output value of each dimension directly represents the existence probability of the corresponding abnormal state, realizing the parallel judgment of multiple anomalies.

[0083] 3. Focal Loss function: To address the imbalance problem of a high proportion of normal samples and a scarcity of some abnormal samples in vehicle-mounted abnormal samples, Focal Loss is introduced as the training loss function to replace the traditional binary cross-entropy loss, thereby improving training stability and effectively solving the problem that the model tends to predict the majority class under the traditional loss function. This allows the model training process to focus on learning the key abnormal sample features.

[0084] This model structure is compatible with INT8 quantization and TensorRT acceleration. The quantized parameters have low accuracy errors, which can meet performance requirements. The inference speed can meet the real-time requirements of parallel processing of surround-view cameras. The model can be adapted to mainstream automotive chips without reconstruction, resulting in low engineering deployment costs.

[0085] To further improve the reliability of the recognition results and reduce false positives, false negatives, and jumps in confidence levels, this embodiment performs post-processing on the prediction results output by the model. Figure 4 This is a schematic diagram of the post-processing steps in the vehicle-mounted camera lens abnormality recognition method provided in Embodiment 2 of the present invention, as shown below. Figure 4As shown, the specific processing steps are as follows: 1. Differentiated Threshold Judgment: Different thresholds are set for the predicted scores of different abnormal states. When the predicted score is higher than the corresponding threshold, the abnormal state is determined to exist; when it is lower than the threshold, it is determined not to exist. The differentiated threshold setting takes into account the sample distribution and recognition difficulty of different abnormal states, thereby improving classification accuracy.

[0086] 2. Window Smoothing: Since the images from the vehicle-mounted camera are continuous frame sequences, the appearance and disappearance of abnormal states are continuous. Therefore, time-series smoothing is performed on the recognition results of 10 consecutive frames. A sliding window voting mechanism is used: if an abnormal state appears 6 or more times in 10 consecutive frames, it is ultimately determined to be an anomaly; if it only appears in 1-2 frames, it is determined to be a model misjudgment, and the result is filtered out. This processing can effectively eliminate misidentification caused by noise in single-frame images or transient interference.

[0087] 3. Abnormal State Correlation Validation: Some abnormal states are correlated (e.g., icing is usually accompanied by low temperatures and will not occur simultaneously with splashing; defocusing and greening patterns usually do not occur simultaneously). Based on this, abnormal state correlation rules are constructed. If a combination that violates the correlation rules appears in the identification results (e.g., icing and splashing exist simultaneously), the abnormal state with the lower prediction score in that combination is subject to secondary validation. Further judgment can be made in conjunction with weather conditions to eliminate contradictory results.

[0088] 4. Result Output and Anomaly Classification: The final identification results are classified into three levels according to the severity of the anomaly: Level 1 anomaly (complete occlusion, icing, green image), at which point the camera cannot provide any useful information at all, and it is recommended that the system's perception function be deactivated; Level 2 anomaly (partial occlusion, out of focus, blur, splash), some functions of the camera are malfunctioning, and the driver can be prompted to check the lens status; Level 3 anomaly (backlight, sun rays), the camera performance is affected but can still provide some useful information, and a prompt message can be output, without the need for emergency handling.

[0089] The vehicle-mounted camera lens anomaly recognition method provided by this invention supports the identification of multiple anomaly signals, covering various lens anomalies that may occur during driving, thus solving the problem of limited identification types in existing methods. The overall framework design is simple and efficient, possessing high universality and good scalability. The modular architecture facilitates subsequent function iterations and anomaly type expansion, supporting rapid transfer learning to add new anomaly categories with a small number of samples, meeting the continuous evolution and diverse deployment needs of intelligent driving systems. The lightweight algorithm structure balances high accuracy and real-time inference performance, ensuring that anomaly events can be identified promptly and trigger warning mechanisms, guaranteeing driving decision safety. The overall architecture is engineering-friendly and can be deployed and run on various mainstream computing platforms.

[0090] Example 3 Corresponding to Embodiment 1 above, Embodiment 3 of the present invention provides a vehicle-mounted camera lens abnormality identification device. The corresponding technical features and effects can be referred to Embodiment 1 above, and will not be repeated in this embodiment. Figure 5 This is a block diagram of the vehicle-mounted camera lens abnormality recognition device provided in Embodiment 3 of the present invention, as shown below. Figure 5 As shown, the device includes: an acquisition module 201, a preprocessing module 202, an identification module 203, and a determination module 204.

[0091] The acquisition module 201 is used to acquire image frames captured by the vehicle-mounted camera; the preprocessing module 202 is used to preprocess the image frames to obtain a standard image; the recognition module 203 is used to input the standard image into an abnormal state recognition model, wherein the abnormal state recognition model is used to extract features of the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, each dimension component corresponds to an abnormal state, and performs independent activation mapping on each dimension component to output the confidence level corresponding to each of the multiple abnormal states; and the determination module 204 is used to determine the abnormal state recognition result of the vehicle-mounted camera lens based on the multiple confidence levels.

[0092] Optionally, in one embodiment, the abnormal state recognition model includes a feature extraction network, a feature mapping layer, and an activation layer. The recognition module includes: a first input unit for inputting the standard image into the feature extraction network, wherein the feature extraction network is used to extract features from the standard image and output a global feature vector; a second input unit for inputting the global feature vector into the feature mapping layer, wherein the feature mapping layer is used to map the global feature vector into the multidimensional initial prediction vector; and a third input unit for inputting the multidimensional initial prediction vector into the activation layer, wherein the activation layer is used to perform independent activation mapping on each dimension component to output the confidence level corresponding to each abnormal state.

[0093] Optionally, in one embodiment, the plurality of abnormal states include full occlusion, partial occlusion, blur, icing, splashing, out of focus, backlight, sun rays, and green map.

[0094] Optionally, in one embodiment, the feature extraction network adopts a residual network ResNet18, the feature extraction network is used to output a 512-dimensional global feature vector, the feature mapping layer is used to map the 512-dimensional global feature vector into a 9-dimensional initial prediction vector, the activation layer adopts a Sigmoid function, and Focal Loss is used as the training loss function during the training process of the abnormal state recognition model.

[0095] Optionally, in one embodiment, the vehicle-mounted camera is a surround-view vehicle-mounted camera, and the device further includes an output module, configured to, after the determining module determines the abnormal state identification result of the vehicle-mounted camera lens based on the plurality of confidence levels, output a prompt message to exit the perception function when the abnormal state of the vehicle-mounted camera lens includes complete obstruction, icing, or a green map; output a prompt message to check the lens status when the abnormal state of the vehicle-mounted camera lens includes partial obstruction, out-of-focus, blur, or splatter; and output a prompt message that the current light affects the camera performance when the abnormal state of the vehicle-mounted camera lens includes backlight or sunlight.

[0096] Optionally, in one embodiment, the determining module includes: a storage unit for storing different preset confidence thresholds for different abnormal states; a first traversal judgment unit for traversing the plurality of confidence levels and determining whether the confidence level is greater than or equal to the corresponding confidence threshold; a first determining unit for determining that the first image recognition result includes the presence of an abnormal state corresponding to the confidence level in the image frame when the confidence level is greater than or equal to the corresponding confidence threshold; a second determining unit for determining that the first image recognition result includes the absence of an abnormal state corresponding to the confidence level in the image frame when the confidence level is less than the corresponding confidence threshold; and a third determining unit for determining the abnormal state recognition result based on the first image recognition result.

[0097] Optionally, in one embodiment, when the third determining unit determines the abnormal state identification result based on the first image recognition result, the specific execution steps include: traversing the abnormal states of the image frames determined by the first image recognition result, and counting the number of image frames with the abnormal state within the sliding time window where the image frame is located; determining whether the number of image frames is greater than or equal to a preset frame number threshold; when the number of image frames is greater than or equal to the preset frame number threshold, determining that the second image recognition result includes the presence of the abnormal state in the image frame; when the number of image frames is less than the preset frame number threshold, determining that the second image recognition result includes the absence of the abnormal state in the image frame; and determining the abnormal state identification result based on the second image recognition result.

[0098] Optionally, in one embodiment, the step of determining the abnormal state identification result based on the second image recognition result includes: determining whether the second image recognition result satisfies an abnormal state association rule, wherein the abnormal state association rule includes a conflict rule and / or a co-occurrence rule, the conflict rule is used to define an abnormal state that conflicts, and the co-occurrence rule is used to define an abnormal state that occurs simultaneously; when the second image recognition result does not satisfy the abnormal state association rule, verifying the abnormal state with low confidence among the unsatisfied abnormal state association rules according to preset external parameters; and determining the abnormal state identification result based on the verification result.

[0099] Example 4 This fourth embodiment provides a vehicle, an in-vehicle camera, and a processor. The processor executes any of the in-vehicle camera lens abnormality identification methods provided by this invention to identify abnormalities, possessing relevant technical features and corresponding technical effects, which will not be elaborated here.

[0100] Example 5 This embodiment also provides a computer device, such as an in-vehicle controller, in-vehicle computing platform, domain controller, edge computing unit, or server capable of executing programs. Figure 6 As shown, the computer device 01 in this embodiment includes, but is not limited to, a memory 012 and a processor 011 that are communicatively connected to each other via a system bus. It should be noted that... Figure 6 Only a computer device 01 with component memory 012 and processor 011 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0101] In this embodiment, the memory 012 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 012 may be an internal storage unit of the computer device 01, such as the hard disk or memory of the computer device 01. In other embodiments, the memory 012 may also be an external storage device of the computer device 01, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 01. Of course, the memory 012 may include both the internal storage unit and its external storage device of the computer device 01. In this embodiment, the memory 012 is typically used to store the operating system and various application software installed on the computer device 01, such as the program code of the vehicle-mounted camera lens abnormality recognition device in Embodiment 3. In addition, memory 012 can also be used to temporarily store various types of data that have been output or will be output.

[0102] In some embodiments, processor 011 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 011 is typically used to control the overall operation of computer device 01. In this embodiment, processor 011 is used to run program code stored in memory 012 or process data, such as a method for identifying abnormal states of a vehicle-mounted camera lens.

[0103] Example 6 This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. The computer-readable storage medium of this embodiment is used to store a vehicle-mounted camera lens abnormal state recognition device, and when executed by a processor, it implements the vehicle-mounted camera lens abnormal state recognition method of Embodiment 1 or 2.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0105] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0107] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for identifying abnormal states of a vehicle-mounted camera lens, characterized in that, include: Acquire image frames captured by the vehicle-mounted camera; The image frames are preprocessed to obtain standard images; The standard image is input into the abnormal state recognition model, wherein the abnormal state recognition model is used to extract the features of the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, each dimension component corresponds to an abnormal state, and each dimension component is independently activated and mapped to output the confidence level corresponding to each of the multiple abnormal states. as well as The abnormal state identification result of the vehicle-mounted camera lens is determined based on the multiple confidence levels.

2. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 1, characterized in that, The abnormal state recognition model includes a feature extraction network, a feature mapping layer, and an activation layer. The step of inputting the standard image into the abnormal state recognition model includes: The standard image is input into the feature extraction network, wherein the feature extraction network is used to extract features from the standard image and output a global feature vector; The global feature vector is input to the feature mapping layer, wherein the feature mapping layer is used to map the global feature vector into the multidimensional initial prediction vector; The multidimensional initial prediction vector is input to the activation layer, wherein the activation layer is used to perform independent activation mapping on each dimension component to output the confidence level corresponding to each abnormal state.

3. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 2, characterized in that, The various abnormal states include full occlusion, partial occlusion, blur, icing, splashing, out of focus, backlighting, solar radiation, and green map.

4. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 2 or 3, characterized in that, The feature extraction network adopts the ResNet18 residual network, which is used to output a 512-dimensional global feature vector. The feature mapping layer is used to map the 512-dimensional global feature vector into a 9-dimensional initial prediction vector. The activation layer adopts the Sigmoid function. Focal Loss is used as the training loss function during the training process of the abnormal state recognition model.

5. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 3, characterized in that, The vehicle-mounted camera is a surround-view vehicle-mounted camera. After the step of determining the abnormal state recognition result of the vehicle-mounted camera lens based on the multiple confidence levels, the method further includes: When the abnormal state of the vehicle-mounted camera lens includes complete obstruction, icing, or a green image, a prompt message indicating that the perception function should be exited is output. When the abnormal condition of the vehicle-mounted camera lens includes partial obstruction, out-of-focus, blurriness, or splattering, a prompt message to check the lens condition is output; and When the abnormal state of the vehicle-mounted camera lens includes backlighting or sunlight, a prompt message is output indicating that the current light is affecting the camera performance.

6. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 1, characterized in that, The steps for determining the abnormal state recognition result of the vehicle-mounted camera lens based on the multiple confidence levels include: Different confidence thresholds are preset for different abnormal states; Iterate through the multiple confidence levels and determine whether each confidence level is greater than or equal to the corresponding confidence threshold. When the confidence level is greater than or equal to the corresponding confidence threshold, the first image recognition result is determined to include the presence of an abnormal state corresponding to the confidence level in the image frame; When the confidence level is less than the corresponding confidence threshold, the first image recognition result is determined to include the absence of an abnormal state corresponding to the confidence level in the image frame; and The abnormal state recognition result is determined based on the first image recognition result.

7. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 6, characterized in that, The steps for determining the abnormal state recognition result based on the first image recognition result include: Iterate through the abnormal states of the image frames determined by the first image recognition result, and count the number of image frames with the abnormal states within the sliding time window where the image frame is located. Determine whether the number of image frames is greater than or equal to a preset frame count threshold; When the number of image frames is greater than or equal to the preset frame number threshold, the second image recognition result is determined to include the presence of the abnormal state in the image frame; When the number of image frames is less than the preset frame number threshold, the second image recognition result is determined to include the absence of the abnormal state in the image frame; and The abnormal state recognition result is determined based on the second image recognition result.

8. The method for identifying abnormal states of vehicle-mounted camera lenses according to claim 7, characterized in that, The step of determining the abnormal state recognition result based on the second image recognition result includes: Determine whether the second image recognition result satisfies the abnormal state association rule, wherein the abnormal state association rule includes conflict rule and / or co-occurrence rule, the conflict rule is used to define abnormal states that conflict, and the co-occurrence rule is used to define abnormal states that occur simultaneously; When the second image recognition result does not satisfy the abnormal state association rule, the abnormal states with low confidence in the abnormal state association rule that are not satisfied are verified according to preset external parameters; and The abnormal state identification result is determined based on the verification result.

9. A device for identifying abnormal states of a vehicle-mounted camera lens, characterized in that, include: The acquisition module is used to acquire image frames captured by the vehicle-mounted camera; The preprocessing module is used to preprocess the image frames to obtain a standard image; The recognition module is used to input the standard image into the abnormal state recognition model, wherein the abnormal state recognition model is used to extract the features of the standard image, generate a multi-dimensional initial prediction vector based on the features of the standard image, each dimension component corresponds to an abnormal state, and perform independent activation mapping on each dimension component to output the confidence level corresponding to each of the multiple abnormal states. as well as The determination module is used to determine the abnormal state identification result of the vehicle-mounted camera lens based on the multiple confidence levels.

10. A vehicle, characterized in that, include: A vehicle-mounted camera and a processor, wherein the processor executes the vehicle-mounted camera lens abnormality identification method according to any one of claims 1 to 8 to identify an abnormality.

Images