A camera contamination detection system and method for a vehicle

Abstract

The application discloses a kind of camera pollutant detection systems and methods for car, it is related to automobile technical field, by real-time acquisition video stream around vehicle and carry out pretreatment, utilize multi-task self-supervised learning from a large number of unlabeled data Extraction multi-modal feature.System uses attention mechanism dynamically adjusts the area of focus, and combined with the generation of specific types and degree of pollution image, to enhance feature representation.Feature fusion module is integrated by high-dimensional feature, multi-scale feature map, enhanced feature representation, form comprehensive feature vector, by multilayer perceptron classification output pollution degree probability value, and then control cleaning device.Feedback and optimization module dynamically adjusts strategy according to detection result and cleaning effect, ensure that camera can maintain clear vision under various environmental conditions, improve driving safety.The application has advantages in improving detection accuracy, response speed and cleaning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automotive technology, and in particular to a camera contamination detection system and method for automobiles. BACKGROUND

[0002] Automotive cameras are key components in intelligent driving and assisted driving systems, used to capture image information of the vehicle's surroundings. These image information is crucial for the vehicle's navigation, obstacle avoidance, lane keeping, and other functions. However, contamination on the camera surface (such as dust, mud, raindrops, etc.) can severely affect image quality, and thus the performance of the system.

[0003] In the prior art, camera contamination detection systems have insufficient detection accuracy in complex and variable environmental conditions (such as rainy days, snowy days, night, etc.), and are prone to false positives and false negatives, affecting the reliability and safety of the system. In addition, these systems perform poorly in terms of real-time performance and response speed, and cannot complete detection and cleaning control within a short time, affecting the real-time monitoring and timely processing capabilities of the system. At the same time, existing methods have poor robustness and generalization ability in different environments and conditions, making it difficult to adapt to diverse application scenarios. For example, during high-speed driving, existing cleaning strategies may not be flexible enough, affecting cleaning effectiveness.

[0004] The present application aims to overcome the shortcomings of the prior art by combining self-supervised learning, attention mechanisms, and generative adversarial networks, and proposes a camera contamination detection system and method for automobiles. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art by combining self-supervised learning, attention mechanisms, and generative adversarial networks, and proposes a camera contamination detection system and method for automobiles.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] A camera contamination detection system for automobiles, comprising:

[0008] An image preprocessing module: real-time acquisition of video streams around the vehicle, wherein each video stream includes multiple images, and preprocessing of the acquired images;

[0009] A self-supervised learning module: receiving processed image data from the image preprocessing module, extracting multi-modal features from a large amount of unlabeled data through multi-task self-supervised learning, and generating high-dimensional feature representation F SSL , shape (H, W, 64);

[0010] Attention mechanism module: receives high-dimensional feature representation F from self-supervised learning module SSL , dynamically adjusts the area of attention using attention mechanism, generates multi-scale feature map F with attention weight AM , shape (H, W, 32);

[0011] Generative adversarial network module: accepts multi-scale feature map F with attention weight from attention mechanism module AM , generates specific type and degree of pollution image using generative adversarial network, generates enhanced feature representation F GAN , shape (H, W, 48);

[0012] Feature fusion module: concatenates high-dimensional feature representation F extracted by self-supervised learning module SSL , multi-scale feature map F with attention weight generated by attention mechanism module AM , and enhanced feature representation F generated by generative adversarial network module GAN along the channel dimension, forming a comprehensive feature vector F combined , shape (H, W, 144); then F combined is flattened into a one-dimensional vector F flat , shape (N, 144), where N equals height H times width W; F flat is processed through two fully connected layers to generate the final feature representation F final , shape (1, 64), as input to the pollution detection module;

[0013] Pollution detection module: uses multi-layer perceptron to classify the final feature representation F final , outputs the probability value P of the pollution degree, and judges the pollution condition of the camera according to the set threshold T, outputs the decision result of the cleaning required;

[0014] Cleaning control module: according to the pollution degree judgment result provided by the pollution detection module, starts the corresponding cleaning device, starts the air jet cleaning device for light pollution, starts the water jet cleaning device for heavy pollution, and dynamically adjusts the cleaning strategy considering the vehicle driving state, outputs the cleaning instruction to the cleaning device;

[0015] Feedback and optimization module: optimizes and feeds back the detection result and cleaning effect.

[0016] Preferably, the self-supervised learning module receives preprocessed image data as input and designs multiple self-supervised tasks through multi-task self-supervised learning, including rotation prediction, jigsaw puzzle, and color channel separation tasks. These tasks are combined to extract richer features. Contamination is detected by comparing changes in adjacent frames using temporal information from consecutive frames. Enhanced features F are generated by optimizing the feature representation through a comparative loss function. enhanced The features extracted by self-supervised learning are mapped to a high-dimensional space to facilitate effective fusion with features from other modules; a high-dimensional feature representation F is generated. SSL The shape is (H,W,64), where H and W are the height and width of the feature map, respectively.

[0017] Preferably, the attention mechanism module receives the high-dimensional feature representation F extracted by the self-supervised learning module. SSL As input, by calculating the attention weight for each pixel, the model can automatically adjust the attention region based on the current image content; multi-scale feature maps are extracted, and attention mechanisms at different scales are used to detect pollutants of different sizes and types; a multi-scale feature map F with attention weights is generated. AM The shape is (H,W,32).

[0018] Preferably, the generative adversarial network module receives a multi-scale feature map F with attention weights generated by the attention mechanism module. AM As input, specific types and degrees of contamination images are generated; semi-supervised GANs are used to improve the model's generalization ability by combining a small amount of labeled data and a large amount of unlabeled data; multimodal contamination images are generated by combining infrared and visible light images to enhance the model's ability to detect contamination under complex environmental conditions; domain adaptation techniques are used to reduce domain shift under different environmental conditions to improve the model's generalization ability; contamination images and enhanced feature representations F are generated. GAN The shape is (H,W,48).

[0019] Preferably, the feature fusion module receives the high-dimensional feature representation F extracted by the self-supervised learning module. SSL The shape is (H, W, 64); the attention mechanism module generates a multi-scale feature map F with attention weights. AM The shape is (H, W, 32); the enhanced feature representation F generated by the generative adversarial network module is... GAN The shape is (H, W, 48); these features are concatenated according to the channel dimension to form a comprehensive feature vector F. combined The shape is (H, W, 144); the comprehensive feature vector F is processed through a fully connected layer. combined The process is performed to generate the final feature representation F. finalThe shape is (1, 64), used for subsequent pollution detection and classification in this system. The specific steps are as follows:

[0020] 1. F SSL F AM and F GAN The vectors are concatenated along the channel dimension to form a comprehensive feature vector F. combined The shape is (H,W,144);

[0021] 2. F combined Flattened into a one-dimensional vector F flat The shape is (N, 144), where N equals the height H multiplied by the width W;

[0022] 3. Through the fully connected layer FC1 to F flat Dimensionality reduction is performed to generate an intermediate feature vector F. mid The weight matrix W1 has a shape of (144, 128), the bias vector b1 has a shape of (128), and the intermediate eigenvector F mid It is the weight matrix W1 and the flattened eigenvector F flat The dot product plus the bias vector b1, followed by the ReLU activation function, is given by the formula: F mid =ReLU(W1+F flat +b1);

[0023] 4. F is connected via the fully connected layer FC2. mid Further processing generates the final feature representation F. final The weight matrix W2 has a shape of (128, 64), the bias vector b2 has a shape of (64), and the final feature representation F final It is the weight matrix W2 and the intermediate eigenvector F mid The dot product of the two vectors, plus the bias vector b2, and then the result after passing through the ReLU activation function, is given by the formula: F final =ReLU(W2+F mid +b2).

[0024] Preferably, the pollution detection module receives the final feature representation F. final The input is a graph with shape ((1,64). The results from the self-supervised learning module, attention mechanism module, and generative adversarial network module are combined to determine whether the camera is contaminated. The features extracted by each module are fused, and a fully connected layer or multilayer perceptron (MLP) is used for the final classification decision. The final feature representation F is then processed using a multilayer perceptron. final The system performs classification and outputs a probability value P representing the degree of pollution; the probability value P represents the output layer weight matrix W. out The shape is (64,1), with a bias of b. outThe shape is (1), and the probability value P of the pollution level is the output layer weight matrix W. out With the final feature representation F final dot product plus bias b out Then, the result is processed by the Sigmoid function; the probability value of the pollution level is compared with a preset threshold to determine the pollution level. The preset threshold T = 0.5 and can be adjusted according to actual application and environmental conditions; if P ≥ T, the camera is considered to be heavily polluted; otherwise, the camera is considered to be lightly polluted.

[0025] Preferably, the cleaning control module includes receiving pollution detection results as input, activating corresponding cleaning devices according to the degree of pollution, activating a jet cleaning device for pollution levels less than T, using high-pressure gas to blow away light pollutants on the camera surface; activating a water spray cleaning device for pollution levels greater than or equal to T, using jet water to clean heavily pollutants on the camera surface; and dynamically adjusting the cleaning strategy according to the vehicle's driving status.

[0026] Preferably, the feedback and optimization module receives pollution detection results and cleaning effects as inputs and performs online optimization; it uses reinforcement learning algorithms, including Qlearning and DeepQ-Network, to dynamically adjust the cleaning strategy and improve the system's response speed and accuracy; and it uses OTA (Over-the-Air) update technology to periodically update the detection model on the vehicle-mounted device.

[0027] A method for detecting pollutants using a car camera includes the following steps:

[0028] Step 1: Real-time acquisition of video streams around the vehicle and preprocessing of the acquired images;

[0029] Step 2: Extract features from a large amount of unlabeled data through multi-task self-supervised learning to generate high-dimensional features;

[0030] Step 3: Use an attention mechanism to dynamically adjust the attention region and generate a multi-scale feature map with attention weights;

[0031] Step 4: Use a generative adversarial network to generate images of a specific type and degree of contamination;

[0032] Step 5: Concatenate the high-dimensional feature representation extracted by the self-supervised learning module, the multi-scale feature map with attention weights generated by the attention mechanism module, and the enhanced feature representation generated by the generative adversarial network module according to the channel dimension to form a comprehensive feature vector; process the comprehensive feature vector through a fully connected layer to generate the final feature.

[0033] Step 6: Use a multilayer perceptron to classify the final features and output the probability value P of the degree of contamination; if P≥T, the camera is considered to be heavily contaminated; otherwise, the camera is considered to be slightly contaminated.

[0034] Step 7: Activate the corresponding cleaning device according to the degree of pollution, and dynamically adjust the cleaning strategy according to the vehicle's driving status;

[0035] Step 8: Feed back the test results and cleaning effect to the system for online optimization.

[0036] The present invention has the following beneficial effects:

[0037] 1. In this invention, the feature fusion module enables efficient detection and analysis of contaminants from camera feeds. By deeply fusing high-dimensional features extracted through self-supervised learning, multi-scale feature maps generated by an attention mechanism, and feature representations enhanced by generative adversarial networks, a comprehensive feature vector is generated. This not only improves the accuracy of contaminant detection but also enhances the system's robustness under complex environmental conditions, ensuring that the camera provides clear images under various lighting and weather conditions, thereby significantly improving driving safety.

[0038] 2. In this invention, the feature fusion module, through processing the final feature representation generated by the fully connected layer, provides rich information to the pollution detection module, enabling it to accurately determine the degree of pollution at the camera. This not only optimizes the response strategy of the cleaning control module but also achieves real-time monitoring and strategy adjustment of the cleaning effect through the feedback and optimization module. This allows the system to intelligently and dynamically adjust the working mode of the cleaning device based on the degree of pollution and the vehicle's driving status, thereby reducing energy consumption and maintenance costs while ensuring cleaning effectiveness. Attached Figure Description

[0039] Fig. 1 This is a system architecture diagram of a car camera pollutant detection system proposed in this invention;

[0040] Fig. 2 This is a flowchart of a method for a vehicle camera pollutant detection system according to the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] like Figs. 1-2As shown, the present invention proposes a vehicle camera pollutant detection system, comprising:

[0043] Image preprocessing module: Real-time acquisition of video streams around the vehicle, where each video stream contains multiple frames of images, and preprocesses the acquired images;

[0044] The self-supervised learning module receives image data processed by the image preprocessing module and extracts multimodal features from a large amount of unlabeled data through multi-task self-supervised learning to generate a high-dimensional feature representation F. SSL The shape is (H,W,64);

[0045] Attention mechanism module: Receives high-dimensional feature representation F from the self-supervised learning module. SSL By dynamically adjusting the region of focus using an attention mechanism, a multi-scale feature map F with attention weights is generated. AM The shape is (H,W,32);

[0046] Generative Adversarial Network Module: Receives multi-scale feature maps F with attention weights from the attention mechanism module. AM Generative adversarial networks are used to generate images with specific types and degrees of contamination, generating enhanced feature representations F. GAN The shape is (H,W,48);

[0047] Feature fusion module: This module integrates the high-dimensional feature representation F extracted by the self-supervised learning module. SSL The attention mechanism module generates a multi-scale feature map F with attention weights. AM The enhanced feature representation F generated by the generative adversarial network module GAN The vectors are concatenated along the channel dimension to form a comprehensive feature vector F. combined The shape is (H, W, 144); then F combined Flattened into a one-dimensional vector F flat The shape is (N, 144), where N equals the height H multiplied by the width W; F is processed through two fully connected layers. flat Generate the final feature representation F final The shape is (1, 64), which serves as the input to the pollution detection module;

[0048] Pollution detection module: Uses a multilayer perceptron for the final feature representation F final Classify the contamination level and output the probability value P. Determine the contamination status of the camera based on the set threshold T and output the decision result that it needs to be cleaned.

[0049] Cleaning control module: Based on the pollution level judgment results provided by the pollution detection module, the corresponding cleaning device is activated. When the pollution is light, the jet cleaning device is activated, and when the pollution is heavy, the water spray cleaning device is activated. At the same time, the cleaning strategy is dynamically adjusted considering the vehicle's driving status, and the cleaning command is output to the cleaning device.

[0050] Feedback and optimization module: Feeds back the detection results and cleaning effect to the system.

[0051] In one embodiment, such as Figs. 1-2 As shown, an autonomous vehicle is driving in the rain. Due to the mixture of rainwater and mud, the front camera becomes contaminated, affecting the normal operation of the vision sensors. We need to promptly detect the contamination of the camera and take appropriate cleaning measures to ensure the safety and reliability of the autonomous driving system.

[0052] 1. Real-time video stream acquisition and preprocessing

[0053] Step 1: When a car is driving in the rain, the images captured by the camera have low brightness and poor contrast. The system automatically adjusts the image brightness and contrast through preprocessing steps to ensure image clarity. For example, if the system detects that the ambient light is dim, it automatically increases the image brightness, making the image brighter and clearer, while the car's camera captures the video stream from ahead in real time.

[0054] Hardware: Use a high-resolution, high-frame-rate camera (e.g., 1080p, 30fps) to ensure clear images are captured under various lighting conditions.

[0055] Preprocessing: The acquired images are preprocessed, including noise reduction, brightness adjustment, and contrast enhancement, to ensure image quality.

[0056] Denoising: Use bilateral filtering or nonlocal mean filtering to remove image noise.

[0057] Brightness adjustment: Adaptive histogram equalization (CLAHE) and adaptive gamma correction methods are used to enhance the contrast and brightness of the image.

[0058] Contrast Enhancement: Improves image contrast through histogram stretching and gamma correction.

[0059] 2. Multi-task self-supervised learning

[0060] Step 2: As the car drives in the rain, the camera surface gradually becomes covered with mud and sand. The system uses multi-task self-supervised learning to extract features from consecutive frames and detect changes in contaminants on the camera surface. For example, through a rotation prediction task, the system identifies whether there are rotating stains on the camera surface, thus determining the level of contamination. Rich features are extracted from a large amount of unlabeled data through multi-task self-supervised learning.

[0061] Self-monitoring task:

[0062] Rotation prediction task: Randomly rotate an image by a certain angle (e.g., 0°, 90°, 180°, 270°) and train a model to predict the rotation angle.

[0063] Jigsaw puzzle task: Divide the image into 9 pieces, shuffle them, and then train the model to rearrange them.

[0064] Color channel separation task: After separating the color channels of an image, train a model to recover the original image.

[0065] Feature extraction: These tasks are used to extract richer features and generate high-dimensional feature representations F. SSL The shape is (H, W, 64). Assume that the feature vector extracted from the same image using a convolutional neural network is a 256-dimensional vector, where each dimension represents the value of a certain high-level abstract feature of the image, such as object edges, textures, etc.

[0066] Temporal information utilization: Contamination is detected by comparing changes in adjacent frames using temporal information from consecutive frames.

[0067] Contrastive Loss Function: The feature representation is optimized by using a contrastive loss function to generate enhanced features F. enhanced .

[0068] 3. Attention Mechanism

[0069] Step 3: When a car is driving in the rain, the camera surface has contaminants of different sizes and types. Through an attention mechanism, the system can focus on potentially contaminated areas, improving detection accuracy. For example, the system identifies larger clumps of mud and smaller water droplets and processes them separately, thus detecting contaminants more precisely. The attention mechanism dynamically adjusts the attention area.

[0070] Attention weight calculation: Calculate the attention weight for each pixel so that the model can automatically adjust the attention area based on the current image content.

[0071] Calculation formula: The attention weight A is calculated using the Softmax function:

[0072]

[0073] Where, α i It is the attention score of the i-th pixel.

[0074] Multi-scale feature maps: Extract multi-scale feature maps and use attention mechanisms at different scales to detect pollutants of different sizes and types.

[0075] Generate attention feature maps: Generate multi-scale feature maps F with attention weights.AM The shape is (H,W,32).

[0076] 4. Generative Adversarial Networks

[0077] Step 4: When a car is driving in the rain, the system needs to process pollution images under different humidity and temperature conditions. Through generative adversarial networks (GANs), the system generates different types of pollution images, increasing the diversity of the training data. For example, the system generates pollution images from rainy and foggy days, improving the model's detection capabilities under various weather conditions, and uses GANs to generate pollution images of specific types and degrees.

[0078] Data diversity: Increase the diversity of training data to generate enhanced feature representations F. GAN The shape is (H,W,48).

[0079] Semi-supervised learning: Combining a small amount of labeled data with a large amount of unlabeled data, semi-supervised GANs are used to improve the generalization ability of the model.

[0080] Multimodal images: Combining infrared and visible light images to generate multimodal pollution images enhances the model's ability to detect pollution under complex environmental conditions.

[0081] Domain adaptation techniques: These techniques reduce domain shift under different environmental conditions and improve the model's generalization ability.

[0082] 5. Feature fusion

[0083] Step 5: When the car is driving in the rain, a comprehensive feature representation is generated by integrating multiple features, thereby more accurately detecting contamination on the camera surface. The high-dimensional feature representation F extracted by the self-supervised learning module is then used. SSL The attention mechanism module generates a multi-scale feature map F with attention weights. AM The enhanced feature representation F generated by the generative adversarial network module GAN The vectors are concatenated along the channel dimension to form a comprehensive feature vector F. combined The shape is (H,W,144).

[0084] Flattening operation: F combined Flattened into a one-dimensional vector F flat The shape is (N, 144), where N equals the height H multiplied by the width W.

[0085] Fully connected layer processing:

[0086] First fully connected layer: F is connected via the fully connected layer FC1. flat Dimensionality reduction is performed to generate an intermediate feature vector F. midThe weight matrix W1 has a shape of (144, 128), and the bias vector b1 has a shape of (128).

[0087] F mid =ReLU(W1+F flat +b1);

[0088] Second fully connected layer: F is connected via the fully connected layer FC2. mid Further processing generates the final feature representation F. final The shape of the weight matrix W2 is (128, 64), and the shape of the bias vector b2 is (64).

[0089] F final =ReLU(W2+F mid +b2).

[0090] 6. Pollution detection

[0091] Step 6: Use a multilayer perceptron to process the final feature representation F. final Classify the pollution levels and output the probability value P of the pollution level.

[0092] Classification decision: The probability value P of the degree of pollution is the output layer weight matrix W. out With the final feature representation F final dot product plus bias b out Then, the result is processed by the Sigmoid function.

[0093] P=σ(W out +F final +b out )

[0094] Threshold determination: Based on the experimental results, a reasonable threshold T is set, which is set to 0.5.

[0095] If P ≥ 0.5, the camera is considered to be severely contaminated.

[0096] If P < 0.5, the camera is considered to be slightly contaminated.

[0097] For example, if the system detects a pollution probability value P ≥ 0.7 on the current camera surface, it determines that the camera is heavily polluted; if it detects P = 0.3, it determines that the camera is lightly polluted.

[0098] 7. Start the cleaning device

[0099] Step 7: Start the appropriate cleaning device according to the degree of contamination.

[0100] Slight contamination: If P < 0.5, activate the jet cleaning device and use high-pressure gas to blow away the slight contaminants on the camera surface.

[0101] The jetting device uses a compressed air tank and controls the gas injection via a solenoid valve. The injection pressure is 0.5 MPa.

[0102] Moderate or heavy contamination: If P ≥ 0.5, activate the water spray cleaning device and use the water jet to clean the moderate or heavy contaminants on the camera surface.

[0103] Water spraying device: It uses a water pump and nozzles, and controls the water flow through a solenoid valve. The spraying pressure is 0.3MPa and the water flow rate is 2L / min.

[0104] Dynamic adjustment: The cleaning strategy is dynamically adjusted according to the vehicle's driving status. For example, water spray cleaning is avoided when the vehicle speed is greater than 80km / h.

[0105] Vehicle speed detection: Vehicle speed information is obtained through onboard sensors. When the vehicle speed is greater than 80km / h, only the jet system is activated.

[0106] For example, when a car is driving in the rain, the system detects that the camera is moderately contaminated, with a contamination probability value P = 0.7. The system activates the water spray system to clean the contaminants from the camera surface using a jet of water. Simultaneously, the system detects that the current vehicle speed is 90 km / h. To avoid safety hazards associated with water spraying, the system only activates the air jet system, using high-pressure gas to blow away the contaminants. The system activates the water spray system at a vehicle speed of 70 km / h and only the air jet system at a vehicle speed of 90 km / h.

[0107] 8. Online optimization

[0108] Step 8: When the car is driving in the rain, the system continuously learns and improves its cleaning strategy through online optimization. For example, the system uses reinforcement learning algorithms to adjust the cleaning strategy based on the effectiveness of each cleaning cycle, thereby improving cleaning efficiency. The system records the cleaning results after each cycle and adjusts the cleaning strategy based on this data to improve future cleaning results. Simultaneously, the system uses OTA (Over-The-Air) update technology to regularly update the detection model, ensuring the system is always in optimal condition.

[0109] Reinforcement learning: Using reinforcement learning algorithms, including Qlearning and Deep Q-Network, the cleaning strategy is dynamically adjusted to improve the system's response speed and accuracy.

[0110] Reward function: Define a reward function that gives a positive reward when the camera's cleanliness improves and a negative reward when it deteriorates.

[0111] Status indication: Status includes camera contamination level, vehicle speed, cleaning device status, etc.

[0112] OTA Update: The detection model on the vehicle equipment is updated regularly through OTA (Over-the-Air) update technology to ensure that it is always up-to-date and most accurate.

[0113] Update frequency: The model is updated monthly, with new model files downloaded wirelessly and installed automatically.

[0114] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A vehicle-mounted camera-based pollutant detection system, characterized in that, include: Image preprocessing module: Real-time acquisition of video streams around the vehicle, where each video stream contains multiple frames of images, and preprocesses the acquired images; The self-supervised learning module receives image data processed by the image preprocessing module and extracts multimodal features from a large amount of unlabeled data through multi-task self-supervised learning to generate high-dimensional feature representations. , shape is ; Attention mechanism module: Receives high-dimensional feature representations from the self-supervised learning module. By dynamically adjusting the region of focus using an attention mechanism, multi-scale feature maps with attention weights are generated. , shape is ; Generative Adversarial Network Module: Receives multi-scale feature maps with attention weights from the attention mechanism module. Generative adversarial networks are used to generate images with specific types and degrees of contamination, resulting in enhanced feature representations. , shape is ; Feature fusion module: Represents the high-dimensional features extracted by the self-supervised learning module. The attention mechanism module generates multi-scale feature maps with attention weights. Enhanced feature representations generated by generative adversarial network modules The features are concatenated along the channel dimension to form a comprehensive feature vector. Shape Then Flattened into a one-dimensional vector , shape is ,in equal to height Multiply by the width W; Through the fully connected layer right Dimensionality reduction is performed to generate intermediate feature vectors. Weight matrix The shape is Bias vector The shape is intermediate feature vector It is a weight matrix with flattened eigenvectors dot product plus bias vector Then, the result after passing through the ReLU activation function is given by the following formula: ; Through the fully connected layer right Further processing generates the final feature representation. Weight matrix The shape is Bias vector The shape is The final feature representation It is a weight matrix With intermediate feature vectors dot product plus bias vector Then, the result after passing through the ReLU activation function is given by the following formula: , shape is As input to the pollution detection module; Pollution detection module: Uses a multilayer perceptron for the final feature representation. Classify the pollution levels and output probability values. According to the set threshold Determine the extent of camera contamination and output a decision that requires cleaning; Cleaning control module: Based on the pollution level judgment results provided by the pollution detection module, the corresponding cleaning device is activated. When the pollution is light, the jet cleaning device is activated, and when the pollution is heavy, the water spray cleaning device is activated. At the same time, the cleaning strategy is dynamically adjusted considering the vehicle's driving status, and the cleaning command is output to the cleaning device. Feedback and Optimization Module: Optimizes and provides feedback on the test results and cleaning effects.

2. The automotive camera pollutant detection system according to claim 1, characterized in that, The self-supervised learning module receives preprocessed image data as input and designs multiple self-supervised tasks through multi-task self-supervised learning, including rotation prediction, jigsaw puzzle, and color channel separation. These tasks are combined to extract further features. Contamination is detected by comparing changes in adjacent frames using temporal information from consecutive frames. Enhanced features are generated by optimizing the feature representation through a comparative loss function. The features extracted by self-supervised learning are mapped to a high-dimensional space to facilitate effective fusion with features from other modules, generating high-dimensional feature representations. , shape is ,in and These are the height and width of the feature map, respectively.

3. The automotive camera pollutant detection system according to claim 1, characterized in that, The attention mechanism module receives high-dimensional feature representations extracted by the self-supervised learning module. As input, by calculating the attention weight for each pixel, the model can automatically adjust the attention region according to the current image content; Extract multi-scale feature maps and use attention mechanisms at different scales to detect pollutants of different sizes and types; Generate multi-scale feature maps with attention weights , shape is .

4. The automotive camera pollutant detection system according to claim 1, characterized in that, The generative adversarial network module receives multi-scale feature maps with attention weights generated by the attention mechanism module. As input, specific types and degrees of contamination images are generated; semi-supervised GANs are used to improve the model's generalization ability by combining a small amount of labeled data with a large amount of unlabeled data; multimodal contamination images are generated by combining infrared and visible light images to enhance the model's ability to detect contamination under complex environmental conditions; domain adaptation techniques are used to reduce domain shifts under different environmental conditions, thereby improving the model's generalization ability; contamination images and enhanced feature representations are generated. , shape is .

5. The automotive camera pollutant detection system according to claim 1, characterized in that, The pollution detection module receives the final feature representation. , shape is As input, the results from the self-supervised learning module, attention mechanism module, and generative adversarial network module are combined to determine whether the camera is contaminated; the features extracted by each module are fused, and a fully connected layer or multilayer perceptron (MLP) is used for the final classification decision; the MLP is then used to process the final feature representation. Classify the pollution levels and output probability values. ; probability value of pollution level It is the output layer weight matrix The shape is bias The shape is The probability value of the degree of pollution It is the output layer weight matrix With the final feature representation dot product plus bias After that, and then The result of the function; the probability value of the pollution level is compared with a preset threshold to determine the pollution level, wherein the preset threshold... And it can be adjusted according to actual application and environmental conditions; if If the condition is met, the camera is considered to be heavily contaminated; otherwise, it is considered to be lightly contaminated.

6. The automotive camera pollutant detection system according to claim 1, characterized in that, The cleaning control module includes receiving contamination detection results as input and activating corresponding cleaning devices based on the degree of contamination. For contamination levels less than [a certain value], [the module will] [activate the cleaning device]. In cases where the air jet cleaning system is activated, high-pressure gas is used to blow away light contaminants from the camera surface; for contamination levels greater than or equal to... In such cases, the water spray cleaning device is activated to use a jet of water to clean the heavily contaminant surface of the camera. The cleaning strategy is dynamically adjusted based on the vehicle's driving status.

7. The automotive camera pollutant detection system according to claim 1, characterized in that, The feedback and optimization module receives pollution detection results and cleaning effects as input and performs online optimization; it uses reinforcement learning algorithms to dynamically adjust the cleaning strategy to improve the system's response speed and accuracy; and it uses OTA update technology to regularly update the detection model on the vehicle-mounted equipment.

8. A method for detecting pollutants using a car camera, applied to a car camera pollutant detection system according to any one of claims 1-7, characterized in that, Includes the following steps: Step 1: Real-time acquisition of video streams around the vehicle and preprocessing of the acquired images; Step 2: Extract features from a large amount of unlabeled data through multi-task self-supervised learning to generate high-dimensional features; Step 3: Use an attention mechanism to dynamically adjust the attention region and generate a multi-scale feature map with attention weights; Step 4: Use a generative adversarial network to generate images of a specific type and degree of contamination; Step 5: Concatenate the high-dimensional feature representation extracted by the self-supervised learning module, the multi-scale feature map with attention weights generated by the attention mechanism module, and the enhanced feature representation generated by the generative adversarial network module according to the channel dimension to form a comprehensive feature vector; process the comprehensive feature vector through a fully connected layer to generate the final feature. Step 6: Use a multilayer perceptron to classify the final features and output the probability value of the degree of contamination. ;if If the condition is met, the camera is considered to be heavily contaminated; otherwise, the camera is considered to be slightly contaminated. Step 7: Activate the corresponding cleaning device according to the degree of pollution, and dynamically adjust the cleaning strategy according to the vehicle's driving status; Step 8: Feed back the test results and cleaning effect to the system for online optimization.

Images