Camera pollutant detection system and method for automobile

Abstract

The invention discloses an automobile camera pollutant detection system and method, relates to the technical field of automobiles, and is characterized in that a video stream around an automobile is collected in real time and preprocessed, and multi-modal features are extracted from a large amount of unlabeled data by using multi-task self-supervised learning. The system adopts an attention mechanism to dynamically adjust a region of interest, and combines with a generative adversarial network to generate a pollution image of a specific type and degree to enhance feature representation. And the feature fusion module forms a comprehensive feature vector by integrating the high-dimensional features, the multi-scale feature map and the enhanced feature representation, and outputs probability values of pollution degrees in a classified manner through a multi-layer perceptron so as to control the cleaning device. And the feedback and optimization module dynamically adjusts a strategy according to a detection result and a cleaning effect, so that the camera can keep a clear view under various environmental conditions, and the driving safety is improved. The method has the advantages of improving the detection accuracy, the response speed and the cleaning efficiency.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a camera pollutant detection system and method for an automobile. Background Art

[0002] Automotive cameras are key components in intelligent driving and assisted driving systems, used to collect image information of the vehicle's surroundings. This image information is essential for vehicle navigation, obstacle avoidance, lane keeping and other functions. However, contaminants on the camera surface (such as dust, mud, raindrops, etc.) can seriously affect the image quality and thus the performance of the system.

[0003] In the existing technology, the camera pollutant detection system has insufficient detection accuracy under complex and changeable environmental conditions (such as rainy days, snowy days, nighttime, etc.), which is prone to false alarms and missed alarms, affecting the reliability and safety of the system. In addition, these systems perform poorly in terms of real-time performance and response speed, and cannot complete detection and cleaning control in a short time, affecting the system's real-time monitoring and timely processing capabilities. At the same time, the existing methods have poor robustness and generalization capabilities under different environments and conditions, and are difficult to adapt to a variety of application scenarios. For example, when driving at high speeds, the existing cleaning strategies may not be flexible enough, affecting the cleaning effect.

[0004] The present invention aims to overcome the shortcomings of the prior art by combining self-supervised learning, attention mechanism and generative adversarial network, and proposes a system and method for detecting pollutants using a camera for automobiles. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of insufficient detection accuracy, poor real-time performance and response speed, and poor robustness and generalization ability of existing camera pollutant detection systems under complex and changeable environmental conditions, and to propose a camera pollutant detection system and method for automobiles.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A camera pollutant detection system for an automobile, comprising:

[0008] Image preprocessing module: collects video streams around the vehicle in real time, where each video stream includes multiple frames of images, and preprocesses the collected images;

[0009] Self-supervised learning module: receives the processed image data from the image preprocessing module, extracts multimodal features from a large amount of unlabeled data through multi-task self-supervised learning, and generates a high-dimensional feature representation F SSL , shape is (H,W,64);

[0010] Attention mechanism module: receives the high-dimensional feature representation F from the self-supervised learning module SSL , using the attention mechanism to dynamically adjust the area of ​​interest and generate a multi-scale feature map F with attention weights AM , shape is (H,W,32);

[0011] Generate adversarial network module: accept the multi-scale feature map F with attention weights from the attention mechanism module AM , using a generative adversarial network to generate images of specific types and degrees of pollution and generate enhanced feature representations F GAN , shape is (H,W,48);

[0012] Feature fusion module: The high-dimensional features extracted by the self-supervised learning module are represented as F SSL , the multi-scale feature map F with attention weights generated by the attention mechanism module AM and the enhanced feature representation F generated by the generative adversarial network module GAN Spliced ​​together according to the channel dimension to form a comprehensive feature vector F combined , the shape is (H, W, 144); then F combined Flatten into a one-dimensional vector F flat , with a shape of (N,144), where N equals the height H times the width W; processed by two fully connected layers F flat , generate the final feature representation F final , with a shape of (1, 64), as the input of the pollution detection module;

[0013] Pollution detection module: Use a multi-layer perceptron to represent the final feature F final Perform classification and output the probability value P of the pollution degree. According to the set threshold T, judge whether the camera is polluted and output the decision result that needs to be cleaned.

[0014] Cleaning control module: according to the pollution degree judgment result provided by the pollution detection module, the corresponding cleaning device is started. When the pollution is light, the jet cleaning device is started, and when the pollution is heavy, the water spray cleaning device is started. At the same time, the cleaning strategy is dynamically adjusted considering the driving status of the vehicle, and the cleaning instruction is output to the cleaning device;

[0015] Feedback and optimization module: optimize and provide feedback on test results and cleaning effects.

[0016] Preferably, the self-supervised learning module receives the preprocessed image data as input, designs multiple self-supervised tasks through multi-task self-supervised learning, including a rotation prediction task, a jigsaw task, and a color channel separation task, and combines these tasks to extract richer features; uses the time information of consecutive frames to detect contamination by comparing the changes of adjacent frames; optimizes the feature representation through the contrast loss function to generate an enhanced feature F enhanced ; Map the features extracted by self-supervised learning to a high-dimensional space so that they can be effectively integrated with the features of other modules; Generate a high-dimensional feature representation F SSL , the shape is (H,W,64), where H and W are the height and width of the feature map respectively.

[0017] Preferably, the attention mechanism module receives the high-dimensional feature representation F extracted from the supervised learning module SSL As input, by calculating the attention weight of each pixel, the model can automatically adjust the attention area according to the current image content; extract multi-scale feature maps, use attention mechanisms of different scales to detect pollutants of different sizes and types; generate multi-scale feature maps F with attention weights AM , the shape is (H,W,32).

[0018] Preferably, the generative adversarial network module receives a multi-scale feature map F with attention weights generated by the attention mechanism module AM As input, generate pollution images of specific types and degrees; combine a small amount of labeled data with a large amount of unlabeled data, and use semi-supervised GANs to improve the generalization ability of the model; combine infrared images and visible light images to generate multimodal pollution images, and enhance the model's pollution detection ability under complex environmental conditions; reduce domain shift under different environmental conditions through domain adaptation technology, and improve the generalization ability of the model; generate pollution images and enhanced feature representation F GAN , the shape is (H,W,48).

[0019] Preferably, the feature fusion module receives the high-dimensional feature representation F extracted from the supervised learning module SSL , shape is (H, W, 64); the multi-scale feature map F with attention weights generated by the attention mechanism module AM , shape is (H, W, 32); the enhanced feature representation F generated by the generative adversarial network module GAN , with a shape of (H, W, 48); these features are concatenated together according to the channel dimension to form a comprehensive feature vector F combined , the shape is (H, W, 144); through the fully connected layer, the comprehensive feature vector F combined Processing is performed to generate the final feature representation F final, with a shape of (1,64), is used for subsequent pollution detection and classification in the system. The specific steps are as follows:

[0020] 1. F SSL 、F AM and F GAN Spliced ​​together according to the channel dimension to form a comprehensive feature vector F combined , shape is (H,W,144);

[0021] 2. F combined Flatten into a one-dimensional vector F flat , shape is (N,144), where N is equal to the height H times the width W;

[0022] 3. Through the fully connected layer FC1 to F flat Perform dimensionality reduction to generate the intermediate feature vector F mid , the shape of the weight matrix W1 is (144,128), the shape of the bias vector b1 is (128), and the intermediate eigenvector F mid is the weight matrix W1 and the flattened eigenvector F flat The dot product of the bias vector b1 is added, and then the result of the ReLU activation function is: mid =ReLU(W1+F flat +b1);

[0023] 4. Through the fully connected layer FC2, F mid Further processing generates the final feature representation F final , the shape of the weight matrix W2 is (128,64), the shape of the bias vector b2 is (64), and the final feature representation F final is the weight matrix W2 and the intermediate eigenvector F mid The dot product of the bias vector b2 is added, and then the result of the ReLU activation function is: final =ReLU(W2+F mid +b2).

[0024] Preferably, the pollution detection module receives the final feature representation F final , with a shape of ((1,64)) as input, the results of the self-supervised learning module, the attention mechanism module and the generative adversarial network module are integrated to determine whether the camera is contaminated; the features extracted by each module are fused, and the final classification decision is made using a fully connected layer or a multi-layer perceptron (MLP); the final feature representation F is used final Classify and output the probability value P of the pollution degree; the probability value P of the pollution degree is the output layer weight matrix W out The shape is (64,1), bias b outThe shape is (1), and the probability value P of the pollution degree is the output layer weight matrix W out And the final feature representation F final The dot product of plus the bias b out After that, the result of the Sigmoid function is passed; the probability value of the pollution degree is compared with a preset threshold to determine the pollution degree, the preset threshold T = 0.5, and can be adjusted according to actual applications and environmental conditions; if P ≥ T, the camera is considered to be severely polluted; otherwise, the camera is considered to be slightly polluted.

[0025] Preferably, the cleaning control module includes receiving pollution detection results as input, starting a corresponding cleaning device according to the degree of pollution, starting a jet cleaning device when the pollution degree is less than T, using high-pressure gas to blow away light pollutants on the camera surface; starting a water spray cleaning device when the pollution degree is greater than or equal to T, using a jet of water to clean heavy pollutants on the camera surface; and dynamically adjusting the cleaning strategy according to the driving status of the vehicle.

[0026] Preferably, the feedback and optimization module receives the pollution detection results and cleaning effects as input and performs online optimization; uses reinforcement learning algorithms, including Ql earning and DeepQ-Network, to dynamically adjust the cleaning strategy to improve the response speed and accuracy of the system; and regularly updates the detection model on the vehicle-mounted equipment through OTA (Over-the-Ai r) update technology.

[0027] A method for detecting pollutants using a camera for an automobile comprises the following steps:

[0028] Step 1: Collect the video stream around the vehicle in real time and pre-process the collected images;

[0029] Step 2: Extract features from a large amount of unlabeled data through multi-task self-supervised learning to generate high-dimensional features;

[0030] Step 3: Use the attention mechanism to dynamically adjust the attention area and generate a multi-scale feature map with attention weights;

[0031] Step 4: Generate images of specific types and degrees of pollution using a generative adversarial network;

[0032] Step 5: The high-dimensional feature representation extracted by the self-supervised learning module, the multi-scale feature map with attention weights generated by the attention mechanism module, and the enhanced feature representation generated by the generative adversarial network module are spliced ​​together according to the channel dimension to form a comprehensive feature vector; the comprehensive feature vector is processed through the fully connected layer to generate the final feature;

[0033] Step 6: Use a multi-layer perceptron to classify the final features and output the probability value P of the pollution degree; if P ≥ T, the camera is considered to be severely polluted; otherwise, the camera is considered to be slightly polluted;

[0034] Step 7: Start the corresponding cleaning device according to the degree of pollution, and dynamically adjust the cleaning strategy according to the vehicle driving status;

[0035] Step 8: Feedback the test results and cleaning effects to the system for online optimization.

[0036] The present invention has the following beneficial effects:

[0037] 1. In the present invention, the feature fusion module realizes efficient detection and analysis of camera pollutants. A comprehensive feature vector is generated by deeply fusing the high-dimensional features extracted by self-supervised learning, the multi-scale feature graph generated by the attention mechanism, and the feature representation enhanced by the generative adversarial network. It not only improves the accuracy of pollutant detection, but also enhances the robustness of the system under complex environmental conditions, ensuring that the camera can provide clear images under various lighting and weather conditions, thereby significantly improving driving safety.

[0038] 2. In the present invention, the final feature representation generated by the feature fusion module through the fully connected layer processing provides rich information for the pollution detection module, enabling it to accurately judge the degree of pollution of the camera. It not only optimizes the response strategy of the cleaning control module, but also realizes real-time monitoring and strategy adjustment of the cleaning effect through the feedback and optimization module. The system can intelligently adjust the working mode of the cleaning device dynamically according to the degree of pollution and the driving status of the vehicle, thereby reducing energy consumption and maintenance costs while ensuring the cleaning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a system architecture diagram of a camera pollutant detection system for an automobile proposed by the present invention;

[0040] Figure 2 The present invention is a flowchart of a method for a camera pollutant detection system for an automobile. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] like Figure 1-Figure 2As shown, the present invention proposes a camera pollutant detection system for an automobile, comprising:

[0043] Image preprocessing module: collects video streams around the vehicle in real time, where each video stream includes multiple frames of images, and preprocesses the collected images;

[0044] Self-supervised learning module: receives the processed image data from the image preprocessing module, extracts multimodal features from a large amount of unlabeled data through multi-task self-supervised learning, and generates a high-dimensional feature representation F SSL , shape is (H,W,64);

[0045] Attention mechanism module: receives the high-dimensional feature representation F from the self-supervised learning module SSL , using the attention mechanism to dynamically adjust the area of ​​interest and generate a multi-scale feature map F with attention weights AM , shape is (H,W,32);

[0046] Generate adversarial network module: accept the multi-scale feature map F with attention weights from the attention mechanism module AM , using a generative adversarial network to generate images of specific types and degrees of pollution and generate enhanced feature representations F GAN , shape is (H,W,48);

[0047] Feature fusion module: The high-dimensional features extracted by the self-supervised learning module are represented as F SSL , the multi-scale feature map F with attention weights generated by the attention mechanism module AM and the enhanced feature representation F generated by the generative adversarial network module GAN Spliced ​​together according to the channel dimension to form a comprehensive feature vector F combined , the shape is (H, W, 144); then F combined Flatten into a one-dimensional vector F flat , with a shape of (N,144), where N equals the height H times the width W; processed by two fully connected layers F flat , generate the final feature representation F final , with a shape of (1, 64), as the input of the pollution detection module;

[0048] Pollution detection module: Use a multi-layer perceptron to represent the final feature F final Perform classification and output the probability value P of the pollution degree. According to the set threshold T, judge whether the camera is polluted and output the decision result that needs to be cleaned.

[0049] Cleaning control module: according to the pollution degree judgment result provided by the pollution detection module, the corresponding cleaning device is started. When the pollution is light, the jet cleaning device is started, and when the pollution is heavy, the water spray cleaning device is started. At the same time, the cleaning strategy is dynamically adjusted considering the driving status of the vehicle, and the cleaning instruction is output to the cleaning device;

[0050] Feedback and optimization module: Feedback the test results and cleaning effects to the system.

[0051] In one embodiment, Figure 1-Figure 2 As shown in the figure, an autonomous vehicle is driving in the rain. Due to the mixture of rain and mud, the front camera of the vehicle is contaminated, affecting the normal operation of the visual sensor. We need to detect the contamination of the camera in time and take corresponding cleaning measures to ensure the safety and reliability of the autonomous driving system.

[0052] 1. Real-time video stream acquisition and preprocessing

[0053] Step 1: When the car is driving in the rain, the image captured by the camera has low brightness and poor contrast. The system automatically adjusts the image brightness and contrast through preprocessing steps to ensure a clear image. For example, if the system detects that the current ambient light is dim, it automatically increases the image brightness to make the image brighter and clearer. The car camera collects the video stream in front in real time.

[0054] Hardware: Use a high-resolution, high-frame-rate camera (e.g., 1080p, 30fps) to ensure clear images in all lighting conditions.

[0055] Preprocessing: Preprocess the acquired images, including denoising, brightness adjustment, and contrast enhancement, to ensure image quality.

[0056] Denoising: Use bilateral filtering or non-local means filtering to remove image noise.

[0057] Brightness adjustment: Use adaptive histogram equalization (CLAHE) and adaptive gamma correction methods to enhance the contrast and brightness of the image.

[0058] Contrast Enhancement: Improve image contrast through histogram stretching and gamma correction.

[0059] 2. Multi-task self-supervised learning

[0060] Step 2: When the car is driving in the rain, the camera surface is gradually covered with mud and sand. The system extracts features from continuous frames through multi-task self-supervised learning to detect changes in pollutants on the camera surface. For example, the system uses the rotation prediction task to identify whether there are rotating stains on the camera surface, thereby determining the pollution situation. Multi-task self-supervised learning can extract rich features from a large amount of unlabeled data.

[0061] Self-supervised tasks:

[0062] Rotation prediction task: randomly rotate the image by a certain angle (e.g. 0°, 90°, 180°, 270°) and train the model to predict the rotation angle.

[0063] Jigsaw task: split the image into 9 blocks, shuffle the order, and then train the model to rearrange them.

[0064] Color channel separation task: Separate the color channels of the image and then train the model to restore the original image.

[0065] Feature extraction: Use these tasks to extract richer features and generate high-dimensional feature representations F SSL , the shape is (H, W, 64). Assume that the feature vector extracted from the same image using a convolutional neural network is a 256-dimensional vector, and each dimension represents the value of the image at a certain high-level abstract feature, such as object edges, textures, etc.

[0066] Utilization of temporal information: Utilize the temporal information of consecutive frames to detect contamination by comparing the changes in adjacent frames.

[0067] Contrastive loss function: The contrastive loss function is used to optimize the feature representation and generate enhanced features F enhanced .

[0068] 3. Attention Mechanism

[0069] Step 3: When the car is driving in the rain, there are pollutants of different sizes and types on the camera surface. Through the attention mechanism, the system is able to focus on potential polluted areas and improve detection accuracy. For example, the system identifies larger mud clumps and smaller water droplets and processes them separately, thereby detecting pollution more accurately and using the attention mechanism to dynamically adjust the attention area.

[0070] Attention weight calculation: Calculate the attention weight of each pixel so that the model can automatically adjust the attention area according to the current image content.

[0071] Calculation formula: Use the Softmax function to calculate the attention weight A:

[0072]

[0073] Among them, α i is the attention score of the i-th pixel.

[0074] Multi-scale feature maps: Multi-scale feature maps are extracted and attention mechanisms at different scales are used to detect pollutants of different sizes and types.

[0075] Generate attention feature map: Generate a multi-scale feature map F with attention weightsAM , the shape is (H,W,32).

[0076] 4. Generative Adversarial Networks

[0077] Step 4: When the car is driving in the rain, the system needs to process pollution images under different humidity and temperature conditions. Through the generative adversarial network, the system generates different types of pollution images to increase the diversity of training data. For example, the system generates pollution images on rainy and foggy days to improve the model's detection ability in various weather conditions, and uses the generative adversarial network to generate pollution images of specific types and degrees.

[0078] Data diversity: Increase the diversity of training data and generate enhanced feature representation F GAN , the shape is (H,W,48).

[0079] Semi-supervised learning: Combining a small amount of labeled data with a large amount of unlabeled data, semi-supervised GANs are used to improve the generalization ability of the model.

[0080] Multimodal images: Combine infrared images and visible light images to generate multimodal pollution images, enhancing the model's ability to detect pollution under complex environmental conditions.

[0081] Domain adaptation technology: Domain adaptation technology can be used to reduce domain shift under different environmental conditions and improve the generalization ability of the model.

[0082] 5. Feature Fusion

[0083] Step 5: When the car is driving in the rain, multiple features are integrated to generate a comprehensive feature representation, so as to more accurately detect the pollution on the camera surface. The high-dimensional feature representation F extracted by the self-supervised learning module is SSL , the multi-scale feature map F with attention weights generated by the attention mechanism module AM and the enhanced feature representation F generated by the generative adversarial network module GAN Spliced ​​together according to the channel dimension to form a comprehensive feature vector F combined , the shape is (H,W,144).

[0084] Flattening operation: F combined Flatten into a one-dimensional vector F flat , with shape (N, 144), where N is equal to the height H times the width W.

[0085] Fully connected layer processing:

[0086] The first layer is fully connected: F is connected through the fully connected layer FC1 flat Perform dimensionality reduction to generate the intermediate feature vector F mid, the shape of the weight matrix W1 is (144, 128), and the shape of the bias vector b1 is (128).

[0087] F mid =ReLU(W1+F flat +b1);

[0088] The second layer is fully connected: F is connected through the fully connected layer FC2 mid Further processing generates the final feature representation F final , the shape of the weight matrix W2 is (128,64), and the shape of the bias vector b2 is (64).

[0089] F final =ReLU(W2+F mid +b2).

[0090] 6. Pollution detection

[0091] Step 6: Use a multi-layer perceptron to represent the final feature F final Classify and output the probability value P of the pollution degree.

[0092] Classification decision: The probability value P of the pollution degree is the output layer weight matrix W out And the final feature representation F final The dot product of plus the bias b out After that, the result is passed through the Sigmoid function.

[0093] P=σ(W out +F final +b out )

[0094] Threshold judgment: A reasonable threshold T is set based on experimental results, and the threshold T is set to 0.5.

[0095] If P ≥ 0.5, the camera is considered to be severely polluted.

[0096] If P < 0.5, the camera is considered to be slightly contaminated.

[0097] For example, if the system detects that the pollution probability value P of the current camera surface is ≥ 0.7, it determines that the camera is severely polluted. If it detects that P = 0.3, it determines that the camera is slightly polluted.

[0098] 7. Start the cleaning device

[0099] Step 7: Start the corresponding cleaning device according to the degree of contamination.

[0100] Mild contamination: If P<0.5, start the jet cleaning device and use high-pressure gas to blow away the mild contaminants on the camera surface.

[0101] Jet device: Use a compressed air tank to control the gas jet through a solenoid valve, and the jet pressure is 0.5MPa.

[0102] Moderate or severe contamination: If P ≥ 0.5, start the water spray cleaning device to use a jet of water to clean moderate or severe contaminants on the camera surface.

[0103] Water spray device: Use a water pump and a nozzle, and control the water spray through a solenoid valve. The spray pressure is 0.3MPa and the water flow rate is 2L / min.

[0104] Dynamic adjustment: Dynamically adjust the cleaning strategy according to the vehicle's driving status, for example, avoid starting water spray cleaning when the vehicle speed is greater than 80km / h.

[0105] Vehicle speed detection: The vehicle speed information is obtained through the vehicle-mounted sensor. When the vehicle speed is greater than 80km / h, only the jet device is activated.

[0106] For example, when the car is driving in the rain, the system detects that the camera is moderately polluted, with a pollution probability value of P = 0.7. The system activates the water spray cleaning device and uses a jet of water to clean the pollutants on the surface of the camera. At the same time, the system detects that the current vehicle speed is 90km / h. In order to avoid the safety hazards caused by water spray cleaning, the system only activates the jet device and uses high-pressure gas to blow away the pollutants. The system activates the water spray device when the vehicle speed is 70km / h, and only activates the jet device when the vehicle speed is 90km / h.

[0107] 8. Online Optimization

[0108] Step 8: When the car is driving in the rain, the system continuously learns and improves the cleaning strategy through online optimization. For example, the system uses a reinforcement learning algorithm to adjust the cleaning strategy based on the effect of each cleaning to improve cleaning efficiency. The system records the cleaning effect after each cleaning and adjusts the cleaning strategy based on this data to improve future cleaning effects. At the same time, the system regularly updates the detection model through OTA update technology to ensure that the system is always in the best condition.

[0109] Reinforcement learning: Use reinforcement learning algorithms, including Qlearning and Deep Q-Network, to dynamically adjust cleaning strategies and improve the system's response speed and accuracy.

[0110] Reward function: Define a reward function that gives positive rewards when the camera cleanliness improves, and negative rewards when it decreases.

[0111] Status indication: The status includes the camera contamination level, vehicle speed, cleaning device status, etc.

[0112] OTA update: Through OTA (Over-the-Ai r) update technology, the detection model on the vehicle equipment is regularly updated to ensure that it is always the latest and most accurate.

[0113] Update frequency: The model is updated once a month, and the new model file is downloaded via wireless network and installed automatically.

[0114] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A camera pollutant detection system for an automobile, characterized in that: include: Image preprocessing module: collects video streams around the vehicle in real time, where each video stream includes multiple frames of images, and preprocesses the collected images; Self-supervised learning module: receives image data processed by the image preprocessing module, extracts multimodal features from a large amount of unlabeled data through multi-task self-supervised learning, and generates high-dimensional feature representation F SSL , shape is (H,W,64); Attention mechanism module: receives the high-dimensional feature representation F from the self-supervised learning module SSL , using the attention mechanism to dynamically adjust the area of ​​interest and generate a multi-scale feature map F with attention weights AM , shape is (H,W,32); Generate adversarial network module: receives the multi-scale feature map F with attention weights from the attention mechanism module AM , using a generative adversarial network to generate images of specific types and degrees of pollution and generate enhanced feature representations F GAN , shape is (H,W,48); Feature fusion module: The high-dimensional features extracted by the self-supervised learning module are represented as F SSL , the multi-scale feature map F with attention weights generated by the attention mechanism module AM and the enhanced feature representation F generated by the generative adversarial network module GAN Spliced ​​together according to the channel dimension to form a comprehensive feature vector F combined , the shape is (H, W, 144); then F combined Flatten into a one-dimensional vector F flat , with a shape of (N,144), where N equals the height H times the width W; processed by two fully connected layers F flat , generate the final feature representation F final , with a shape of (1,64) as the input of the pollution detection module; Pollution detection module: Use a multi-layer perceptron to represent the final feature F final Perform classification and output the probability value P of the pollution degree. According to the set threshold T, judge whether the camera is polluted and output the decision result that needs to be cleaned. Cleaning control module: according to the pollution degree judgment result provided by the pollution detection module, the corresponding cleaning device is started. When the pollution is light, the jet cleaning device is started, and when the pollution is heavy, the water spray cleaning device is started. At the same time, the cleaning strategy is dynamically adjusted considering the driving status of the vehicle, and the cleaning instruction is output to the cleaning device; Feedback and optimization module: optimize and provide feedback on test results and cleaning effects.

2. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The self-supervised learning module receives the pre-processed image data as input, designs multiple self-supervised tasks through multi-task self-supervised learning, including rotation prediction task, jigsaw task and color channel separation task, and combines these tasks to extract further features; uses the time information of consecutive frames to detect contamination by comparing the changes of adjacent frames; optimizes the feature representation through the contrast loss function, Generate enhanced features F enhanced ; Map the features extracted by self-supervised learning to a high-dimensional space so that they can be effectively integrated with the features of other modules; Generate a high-dimensional feature representation F SSL , the shape is (H,W,64), where H and W are the height and width of the feature map respectively.

3. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The attention mechanism module receives the high-dimensional feature representation F extracted from the supervised learning module. SSL As input, by calculating the attention weight of each pixel, the model can automatically adjust the attention area according to the current image content; Extract multi-scale feature maps and use attention mechanisms of different scales to detect pollutants of different sizes and types; generate multi-scale feature maps F with attention weights AM , the shape is (H,W,32).

4. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The generative adversarial network module receives the multi-scale feature map F with attention weights generated by the attention mechanism module AM As input, generate pollution images of specific types and degrees; combine a small amount of labeled data with a large amount of unlabeled data, and use semi-supervised GANs to improve the generalization ability of the model; combine infrared images and visible light images to generate multimodal pollution images, and enhance the model's pollution detection ability under complex environmental conditions; reduce domain shift under different environmental conditions through domain adaptation technology, and improve the generalization ability of the model; generate pollution images and enhanced feature representation F GAN , the shape is (H,W,48).

5. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The feature fusion module receives the high-dimensional feature representation F extracted from the supervised learning module. SSL , shape is (H, W, 64); the multi-scale feature map F with attention weights generated by the attention mechanism module AM , shape is (H, W, 32); the enhanced feature representation F generated by the generative adversarial network module GAN , with a shape of (H, W, 48); these features are concatenated together according to the channel dimension to form a comprehensive feature vector F combined , the shape is (H, W, 144); through the fully connected layer, the comprehensive feature vector F combined Processing is performed to generate the final feature representation F final , with a shape of (1,64), is used for subsequent pollution detection and classification in the system. The specific steps are as follows:

1. F SSL 、F AM and F GAN Spliced ​​together according to the channel dimension to form a comprehensive feature vector F combined , shape is (H,W,144); 2. F combined Flatten into a one-dimensional vector F flat , shape is (N,144), where N is equal to the height H times the width W; 3. Through the fully connected layer FC1 to F flat Perform dimensionality reduction to generate the intermediate feature vector F mid , the shape of the weight matrix W1 is (144,128), the shape of the bias vector b1 is (128), and the intermediate eigenvector F mid is the weight matrix W1 and the flattened eigenvector F flat The dot product of the bias vector b1 is added, and then the result of the ReLU activation function is: mid =ReLU(W1+F flat +b1); 4. Through the fully connected layer FC2, F mid Further processing generates the final feature representation F final , the shape of the weight matrix W2 is (128,64), the shape of the bias vector b2 is (64), and the final feature representation F final is the weight matrix W2 and the intermediate eigenvector F mid The dot product of the bias vector b2 is added, and then the result of the ReLU activation function is: final =ReLU(W2+F mid +b2).

6. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The pollution detection module receives the final feature representation F final , with a shape of ((1,64)) as input, the results of the self-supervised learning module, the attention mechanism module and the generative adversarial network module are integrated to determine whether the camera is contaminated; the features extracted by each module are fused, and the final classification decision is made using a fully connected layer or a multi-layer perceptron (MLP); the final feature representation F is used final Classify and output the probability value P of the pollution degree; the probability value P of the pollution degree is the output layer weight matrix W out The shape is (64,1), bias b out The shape is (1), and the probability value P of the pollution degree is the output layer weight matrix W out And the final feature representation F final The dot product of plus the bias b out After that, the result of the Sigmoid function is passed; the probability value of the pollution degree is compared with a preset threshold to determine the pollution degree, the preset threshold T = 0.5, and can be adjusted according to actual applications and environmental conditions; if P ≥ T, the camera is considered to be severely polluted; otherwise, the camera is considered to be slightly polluted.

7. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The cleaning control module includes receiving the pollution detection result as input, starting the corresponding cleaning device according to the pollution degree, and starting the jet cleaning device when the pollution degree is less than T, using high-pressure gas to blow away the light pollutants on the surface of the camera; If the pollution degree is greater than or equal to T, the water spray cleaning device is activated to use the jet water flow to clean the heavy pollutants on the surface of the camera; Dynamically adjust the cleaning strategy according to the vehicle's driving status.

8. The camera pollutant detection system for automobiles according to claim 1, characterized in that: The feedback and optimization module receives the pollution detection results and cleaning effects as inputs and performs online optimization; uses a reinforcement learning algorithm to dynamically adjust the cleaning strategy to improve the response speed and accuracy of the system; and regularly updates the detection model on the vehicle-mounted equipment through OTA update technology.

9. A method for detecting pollutants using a camera in a vehicle according to any one of the group requirements 1-8, characterized in that: The following steps are involved: Step 1: Collect the video stream around the vehicle in real time and pre-process the collected images; Step 2: Extract features from a large amount of unlabeled data through multi-task self-supervised learning to generate high-dimensional features; Step 3: Use the attention mechanism to dynamically adjust the attention area and generate a multi-scale feature map with attention weights; Step 4: Generate images of specific types and degrees of pollution using a generative adversarial network; Step 5: The high-dimensional feature representation extracted by the self-supervised learning module, the multi-scale feature map with attention weights generated by the attention mechanism module, and the enhanced feature representation generated by the generative adversarial network module are spliced ​​together according to the channel dimension to form a comprehensive feature vector; the comprehensive feature vector is processed through the fully connected layer to generate the final feature; Step 6: Use a multi-layer perceptron to classify the final features and output the probability value P of the pollution degree; if P ≥ T, the camera is considered to be severely polluted; otherwise, the camera is considered to be slightly polluted; Step 7: Start the corresponding cleaning device according to the degree of pollution, and dynamically adjust the cleaning strategy according to the vehicle driving status; Step 8: Feedback the test results and cleaning effects to the system for online optimization.

Images